Open vocabulary task-oriented grasping method, apparatus, medium, and program product

Abstract

The application provides an open vocabulary task-oriented grasping method and device, medium and program product, comprising inputting a user language instruction and a complex scene image into a preset visual language model; identifying a target task corresponding to the user language instruction through the preset visual language model, matching a target object in the complex scene image associated with the target task, and determining a best grasping component of the target object based on a plurality of functional components of the target object; adopting the best grasping component and the complex scene image to construct a plurality of grasping candidate schemes; and optimizing and solving each grasping candidate scheme to determine a best grasping pose, aiming to solve the problem that in the related art, due to the limited size of the operability memory bank, the existing method has poor effect when dealing with unseen objects and new tasks in an open vocabulary scene.

Description

Technical Field

[0001] This application belongs to the field of embodied intelligence technology, and specifically relates to an open vocabulary task-oriented grasping method, device, medium and program product. Background Technology

[0002] Inferring the manipulability of an object and performing grasping in a task-oriented manner is crucial for a robot to successfully complete its tasks. Manipulability, by combining the functional characteristics of an object, indicates where and how to grasp it, laying the foundation for efficient task-oriented grasping. For example, when performing the task of pouring water, the robot must identify the handle of the cup as a manipulable area to ensure successful operation. This highlights the need for manipulability-based task-oriented grasping, which is similar in principle to how humans interact with objects (such as grasping the handle of a spoon or racket) and can generalize this cognition to new objects.

[0003] Existing task-oriented methods typically rely on training data limited to specific tasks and objects, making it difficult to generalize to new objects and complex scenarios. Some methods employ object manipulation transfer to guide the robot in selecting grasping positions. For example, RAM and Robo-ABC learn object manipulation from training data by constructing an "operability memory," and then transfer manipulation by retrieving the object most similar to the currently tested object. However, these methods rely on retrieving similar objects, which is often very time-consuming; moreover, due to the limited size of the manipulation memory, these methods perform poorly in scenarios with open vocabulary when dealing with unseen objects and new tasks. Summary of the Invention

[0004] The purpose of this application is to provide an open vocabulary task-oriented grasping method, apparatus, medium, and program product, which aims to solve the problem that related technologies are not effective in dealing with unseen objects and new tasks in open vocabulary scenarios due to the limited size of the operable memory.

[0005] According to a first aspect of this application, an open vocabulary task-oriented grasping method is provided, comprising: inputting user language instructions and complex scene images into a preset visual language model; identifying the target task corresponding to the user language instructions through the preset visual language model, matching target objects associated with the target task in the complex scene image, and determining the optimal grasping component of the target object based on multiple functional components of the target object; constructing multiple grasping candidate schemes using the optimal grasping component and the complex scene image; and optimizing each of the grasping candidate schemes to determine the optimal grasping pose.

[0006] In an optional implementation, before the step of inputting user language instructions and complex scene images into a preset visual language model, the method further includes: when receiving user fuzzy language instructions and complex scene images, identifying user intent based on the user fuzzy language instructions and complex scene images, and generating user language instructions.

[0007] In an optional implementation, the step of identifying the target task corresponding to the user language instruction through the preset visual language model, matching the target object associated with the target task in the complex scene image, and determining the optimal grasping component of the target object based on multiple functional components of the target object includes: identifying the target task and task requirements corresponding to the user language instruction through the preset visual language model; determining the object structure associated with the target task and task requirements based on the target task and task requirements; matching objects in the complex scene image that have the same object structure to generate a target object; dividing the target object into various functional components according to a first preset rule and constructing a set of functional components; identifying the operability of each functional component and constructing an operability set; and selecting the functional component with the highest grasping operability from the set of functional components and the operability set based on the target task and the target object, and determining it as the optimal grasping component.

[0008] In an optional implementation, the open vocabulary task-oriented crawling method further includes: dividing the objects in the complex scene image according to a second preset rule and obtaining the bounding box of the target object; converting the image corresponding to the target object within the bounding box into an object mask image; and calculating the operability mask of the best crawling component on the object mask image.

[0009] In an optional implementation, the step of constructing multiple grasping candidate schemes using the optimal grasping component and the complex scene image includes: obtaining the depth image and camera intrinsics corresponding to the complex scene image; inputting the operability mask corresponding to the optimal grasping component, the depth image corresponding to the complex scene image, and the camera intrinsics into a preset grasping pose generation model; converting the depth image into a partial viewpoint cloud using the camera intrinsics; filtering the partial viewpoint cloud using the operability mask corresponding to the optimal grasping component to generate a target point cloud; and constructing multiple grasping candidate schemes based on the target point cloud, where each grasping pose candidate is represented as the rotation matrix, translation vector, minimum width of the gripper, and confidence score of the gripper in the camera coordinate system.

[0010] In an optional implementation, the step of optimizing each of the candidate grasping schemes to determine the optimal grasping pose includes: obtaining the confidence score corresponding to the grasping pose in each candidate grasping scheme; sorting all the candidate grasping schemes from high to low according to the ratio of the confidence score to the distance between the translation vector and the center of the operability mask in the candidate grasping scheme; determining the optimal grasping pose based on the grasping pose in the highest-ranked candidate grasping scheme; and performing the operation of the optimal grasping pose in a preset scenario.

[0011] In an optional implementation, the open vocabulary task-oriented grasping method further includes: constructing an object knowledge base based on the physical properties of preset objects, object tilt angles, and object tilt angle thresholds; constructing a robot grasping performance test set based on each object in the object knowledge base and multiple preset grasping scenarios; and using the robot grasping performance test set to select objects in the complex scene image that have the same structure as each object to determine the target object.

[0012] A second aspect of this application provides a control device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the open vocabulary task-oriented crawling method described in any of the preceding claims.

[0013] A third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the above-described open vocabulary task-oriented crawling method.

[0014] A fourth aspect of this application provides a computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer performs the above-described open vocabulary task-oriented crawling method.

[0015] This application leverages the reasoning capabilities of a pre-defined visual language model to infer the target task and the operability of objects that enable the target task within user language commands and cluttered environments, and generates task-specific grasping poses. By integrating context-based operability reasoning with visual operability localization, it can effectively parse ambiguous user commands, identify task-related objects, and locate component-level operable areas to generate precise grasping poses. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram illustrating the differences between the steps of this application and existing task-oriented crawling methods;

[0018] Figure 2 This is a flowchart of the steps of an open vocabulary task-oriented crawling method provided in Embodiment 1 of this application;

[0019] Figure 3 This is a flowchart of the steps of an open vocabulary task-oriented crawling method provided in Embodiment 2 of this application;

[0020] Figure 4 This is a schematic diagram of the AffordGrasp task-oriented crawling method provided in Embodiment 2 of this application;

[0021] Figure 5 This is a schematic diagram of context-operable reasoning provided in Embodiment 2 of this application;

[0022] Figure 6 This is a simulation diagram of cluttered scene capture provided in Embodiment 2 of this application;

[0023] Figure 7 This is a schematic diagram of capturing cluttered scenes in a real-world experiment provided in Embodiment 2 of this application;

[0024] Figure 8 This is a schematic diagram of a mug in a real-world experiment provided in Embodiment 2 of this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] Please refer to Figure 1 , Figure 1 This diagram illustrates the differences between the steps of this application and existing task-oriented crawling methods.

[0027] In related technologies, open-vocabulary grasping is receiving increasing attention. These methods generate grasping poses by jointly training a visual-language grasping framework or by utilizing the visual localization capabilities of visual-language models (VLMs). However, they only focus on generating grasping poses for the object as a whole, neglecting the task-related manipulability of the object. To achieve more refined grasping, some methods combine open-vocabulary object detection with part segmentation techniques to perform part-level grasping based on user-provided part prompts. These methods typically rely on explicit object names in the prompts, making it difficult to handle ambiguous user commands (e.g., "I'm thirsty"), and lack the ability to interpret task objectives and object manipulability in complex open-vocabulary environments. Therefore, this application designs an open-vocabulary task-oriented grasping method, apparatus, medium, and program product to effectively solve the above problems.

[0028] Figure 2 This is a flowchart illustrating the steps of an open vocabulary task-oriented crawling method provided in Embodiment 1 of this application.

[0029] This application provides an open vocabulary task-oriented crawling method, which may include the following steps:

[0030] Step 101: Input the user's language instructions and complex scene images into the preset visual language model.

[0031] In this embodiment, a complex scene image can refer to an RGB image acquired by a camera in a fixed location. Pre-defined visual language models (VLMs) can refer to a multimodal artificial intelligence system that combines visual perception and natural language processing capabilities. This system can achieve cross-modal tasks such as image description generation, visual question answering, and text-guided image generation by jointly training image and text data. Its core advantage lies in significantly improving the model's generalization ability through large-scale pre-training (especially by utilizing visual-language correlations for unsupervised learning), reducing reliance on manually labeled data, and supporting zero-shot transfer to new tasks.

[0032] For example, the pre-defined visual language model can use the GPT-4o large language model to parse user language commands and complex scene images.

[0033] In this embodiment, the user provides a user language command and obtains a complex scene image corresponding to the user language command. The user language command and the complex scene image are then input into a preset visual language model for easy recognition by the preset visual language model.

[0034] Step 102: Identify the target task corresponding to the user's language command through a preset visual language model, match the target object associated with the target task in the complex scene image, and determine the best grasping component of the target object based on multiple functional components of the target object.

[0035] In this embodiment of the application, the target task may refer to the task that needs to be performed according to the user's language instructions, such as "I want to scoop something".

[0036] In this embodiment, after the preset visual language model receives a user's language instruction, it analyzes the instruction to determine the user's intended meaning, i.e., the target task. By analyzing the user's language instruction, it matches the target object in the complex scene image with the target object associated with the target task. That is, in the complex scene image, it understands the user's intent and finds objects in the complex scene image that closely match the user's intent, thereby improving the intelligent recognition effect.

[0037] After acquiring the target object, the target object is divided into multiple functional components. Among these functional components, a preset visual language model determines the component that is easiest to grasp.

[0038] Step 103: Utilize the best grasping components and complex scene images to construct multiple grasping candidate schemes.

[0039] In this embodiment, after selecting the best grasping component, multiple grasping candidate schemes are constructed using complex scene images, and the path of the grasping candidate schemes facilitates the final grasping.

[0040] Step 104: Optimize each candidate grasping scheme to determine the best grasping pose.

[0041] The embodiments of this application can utilize AnyGrasp, a 6D grasping generation model trained on a large-scale dataset, to determine the optimal grasping pose.

[0042] In the embodiments of this application, the best grasping candidate scheme is selected from multiple grasping candidate schemes, and the best grasping pose is determined, so as to grasp the object.

[0043] Leveraging the reasoning capabilities of a pre-defined visual language model, the system infers the target task and the operability of objects that enable the task within user language commands and cluttered environments, generating task-specific grasping poses. By integrating context-based operability reasoning with visual operability localization, it can effectively parse ambiguous user commands, identify task-related objects, and locate component-level operable areas to generate precise grasping poses.

[0044] Specifically, this application leverages the inherent reasoning capabilities of pre-defined Visual Language Models (VLMs) to establish the association between tasks, objects, and operability in complex scenes. Secondly, based on the operability reasoning results, the target object and its visually operable region are located using VLMs to generate a grasping pose, thus eliminating the need for additional training data. Finally, with a deep understanding of the task and operability, this application's method outperforms existing methods in cluttered scenes, achieving robust and context-aware grasping.

[0045] Please refer to Figures 3-8 , Figure 3 This application provides a flowchart illustrating the steps of an open vocabulary task-oriented crawling method as described in Embodiment 2. The application provides an open vocabulary task-oriented crawling method, which may include the following steps:

[0046] Step 201: When receiving a user's ambiguous language instruction and a complex scene image, identify the user's intent based on the user's ambiguous language instruction and the complex scene image, and generate a user's language instruction.

[0047] It should be noted that user vague language commands refer to language including but not limited to that in traditional knowledge bases. Most of them are based on vague statements made by users, such as "I want to scoop something up" or similar vague language commands.

[0048] In this example, GPT-4o is used to perform context-based operational inference of user fuzzy language commands. At the same time, based on complex scene images, the user's intent for the task requirement is identified and user language commands are generated.

[0049] Step 202: Input the user's language instructions and complex scene images into the preset visual language model.

[0050] This step is the same as step 101, and will not be repeated here.

[0051] Please refer to Figures 4-5 Please refer to Figure 4 , Figure 4 This is a schematic diagram illustrating the task-oriented crawling method generated by AffordGrasp, as provided in Embodiment 2 of this application. Figure 5 This is a schematic diagram of context-operable reasoning provided in Embodiment 2 of this application.

[0052] Step 203: Identify the target task corresponding to the user's language command through a preset visual language model, match the target object associated with the target task in the complex scene image, and determine the best grasping component of the target object based on multiple functional components of the target object.

[0053] In an optional embodiment, step 203 includes the following steps S11-S16:

[0054] S11. Identify the target task and task requirements corresponding to the user's language commands through a preset visual language model;

[0055] S12. Based on the target task and task requirements, determine the object structure associated with the target task and task requirements;

[0056] S13. Match objects with the same structure as the object in the complex scene image and generate the target object;

[0057] S14. Divide the target object into its functional components according to the first preset rule, and construct a set of functional components;

[0058] S15. Identify the operability of each functional component and construct an operability set;

[0059] S16. Based on the target task and target object, select the functional component with the highest grasping operability from the functional component set and operability set, and determine it as the best grasping component.

[0060] In this embodiment, given user language instructions and the RGB image of the current scene The user's language commands and the RGB image of the current scene are input into GPT-4o. Based on the analysis of the user's language commands by GPT-4o, the specific task objectives and task requirements are extracted. According to the given user language instructions and the RGB image of the current scene First, GPT-4o is used to analyze the user's language commands in order to extract the specific target task and task requirements.

[0061] For example, a user inputs a vague language command into GPT-4o: "I want to scoop something up, please help me choose the appropriate tool." GPT-4o recognizes the user's vague language command, obtains the user's language command, and determines the target task as scooping something by analyzing or inferring the user's language command. After analyzing the target task, it learns that scooping something usually requires the use of a tool with a concave structure to collect or transfer materials, thus determining that the tool with the concave structure is the associated object.

[0062] In this embodiment, after analyzing the target task and task requirements based on GPT-4o, the RGB image of the current scene is used. Determine the most suitable object structure (concave structure) for the target task (scooping something) and the task requirements, that is, to match the instruction "I want to scoop something, please help me choose the appropriate tool". In the current scene's RGB image... In the process, based on the determined object structure associated with the target task and task requirements, the system matches the object to be manipulated and generates the target object, such as the RGB image of the current scene. The "wooden spoon" in the text.

[0063] For example, based on the target task and task requirements RGB image of the current scene Identify the objects most relevant to the task.

[0064] In this embodiment, the selected functional components are decomposed into multiple component sets.

[0065] In this example, the operability set corresponding to each component is inferred based on the operability of the functional components.

[0066] In this embodiment, based on the target task and task requirements For compatibility, select the best component p * and its operability a * The entire reasoning process can be represented as:

[0067]

[0068] In an optional embodiment, the open vocabulary task-oriented crawling method further includes the following steps S21-S23:

[0069] S21. Divide the objects in the complex scene image according to the second preset rule, and obtain the bounding box of the target object;

[0070] S22. Convert the image corresponding to the target object within the bounding box into an object mask image;

[0071] S23. Calculate the operability mask of the best grasping part on the object mask image.

[0072] It should be noted that this application utilizes the open-vocabulary component segmentation method VLPart for visual operability localization, enabling it to handle open-vocabulary scenarios. This is based on the RGB image of the current scene. and operability reasoning The input is the bounding box of the object, and the output is the operability mask on the object parts. This process consists of the following two steps:

[0073] 1) Use VLPart to locate the bounding box of an object. And generate object mask image

[0074]

[0075]

[0076] 2) Based on the mask image, utilize the optimal component p * and its operability a * In the object mask image Predicting operability component mask

[0077]

[0078] The two steps described above effectively eliminate interference from other objects in a cluttered scene.

[0079] It is worth mentioning that the first preset rule refers to the rule of dividing the target object according to its functional parts, while the second preset rule refers to the rule of dividing all objects in the image according to the different objects in the complex scene image.

[0080] Step 204: Utilize the best grasping components and complex scene images to construct multiple grasping candidate schemes.

[0081] In an optional embodiment, step 204 includes the following steps S31-S35:

[0082] S31. Obtain the depth image and camera intrinsic parameters corresponding to the complex scene image;

[0083] S32. Input the operability mask corresponding to the best grasping component, the depth image corresponding to the complex scene image, and the camera intrinsic parameters into the preset grasping pose generation model.

[0084] S33. Use camera intrinsics to convert the depth image into a partial viewpoint point cloud;

[0085] S34. Use the operability mask corresponding to the best grasping component to filter part of the viewpoint point cloud and generate the target point cloud;

[0086] S35. Construct multiple grasping candidate schemes based on the target point cloud. Each grasping pose candidate is represented as the rotation matrix, translation vector, minimum width of the gripper, and confidence score of the gripper in the camera coordinate system.

[0087] It should be noted that AnyGrasp, which is used in this application, can generate grasping poses from partial viewpoint point clouds.

[0088] A rotation matrix is ​​a 3×3 orthogonal matrix that describes the rotational relationship between the world coordinate system and the camera coordinate system. Its core function is to transform points in the world coordinate system to the camera coordinate system.

[0089] Confidence score refers to the quantitative indicator that measures the reliability of a model or system's prediction results. It is usually expressed as a score from 0 to 1 (or 0% to 100%), with a higher score indicating stronger confidence in the results. In artificial intelligence and data analysis, it is widely used in scenarios such as question-answering systems, object detection, and document parsing. For example, in question-answering systems (such as Microsoft QnA), confidence reflects the relevance between the user's query and the answer; a high score (e.g., >0.7) means that the answer and question are highly matched. In object detection (such as YOLO), confidence combines the probability of object presence and location accuracy; for example, 0.95 indicates that the model is confident that the target exists within the predicted bounding box and is accurately located. In document processing, it evaluates the reliability of field extraction or table structuring; a low score may indicate data anomalies requiring manual review.

[0090] A translation vector is a vector that describes the direction and distance of translational movement. Its direction is the same as the direction of the object's movement, and its length is equal to the distance moved. For example, after translating point P(x,y,z) by vector T=(a,b,c), the new coordinates are P′(x+a,y+b,z+c). It preserves the shape, size, and orientation of the figure, only changing its position, and is widely used in physics (describing object displacement) and computer graphics (coordinate system transformation).

[0091] An operability masking center refers to a system or mechanism used to manage, configure, and coordinate masking-related operations. Its core function is to achieve precise control over data, network, or hardware resources through binary bitmasking technology.

[0092] This application utilizes AnyGrasp to generate 6D grasping poses. First, the depth image is converted into a partial viewpoint point cloud using camera intrinsics. Then, a visual operability localization module generates an operability mask to filter the point cloud, generating the target point cloud, thereby achieving task-oriented grasping generation. This strategy constrains the grasping generation process and effectively avoids interference from surrounding objects.

[0093] This application utilizes AnyGrasp to process the point cloud of object parts corresponding to grasping operability and generates grasping candidates. Each grasping pose g is represented as:

[0094]

[0095] in, For rotation matrix, It is a translation vector. This represents the minimum width of the gripper. A set of candidate gripping poses has been generated. Then, select the optimal grasping pose g according to the following criteria. * :

[0096]

[0097] Here, score(g) and t(g) represent the confidence score and translation vector of the grasping pose g, respectively. By selecting a grasping pose that is close to the center of the operability mask c = [xyz] and has a high confidence score, the grasping pose is aligned with the geometric center of the object's operability region to improve the stability of the grasping.

[0098] Step 205: Optimize each candidate grasping scheme to determine the best grasping pose.

[0099] In an optional embodiment, step 205 includes the following steps S41-S44:

[0100] S41. Obtain the confidence score corresponding to the grasping pose in each grasping candidate scheme;

[0101] S42. Sort all crawling candidate schemes from high to low according to the ratio of confidence score to the distance between the translation vector and the center of the operability mask in the crawling candidate scheme;

[0102] S43. Determine the optimal grasping pose based on the grasping pose among the highest-ranked grasping candidate schemes;

[0103] S44. Perform the operation of optimal grabbing pose in the preset scene.

[0104] It should be noted that sorting all crawling candidate solutions from high to low means that they are arranged in descending order according to their confidence scores, and the crawling pose of the crawling candidate solution with the highest ranking is taken as the optimal crawling pose.

[0105] Preset scenes can refer to a set, chaotic scene.

[0106] In this embodiment, each grasping candidate scheme has a corresponding grasping pose with a confidence score. After calculating all confidence scores and translation vectors through steps S31-S35, all grasping candidate schemes are sorted in descending order of the distance ratio of the operability mask center. From all the grasping candidate schemes, the grasping pose involved in the grasping candidate scheme with the highest ranking is selected as the optimal grasping pose. The optimal grasping pose is used to grasp the specified task target and the object to be grasped in the preset scene.

[0107] In an optional embodiment, the open vocabulary task-oriented crawling method further includes the following steps S51-S53:

[0108] S51. Construct an object knowledge base based on the preset physical properties of the object, the object tilt angle, and the object tilt angle threshold.

[0109] S52. Based on each object in the object knowledge base and multiple preset grasping scenarios, construct a robot grasping performance test set;

[0110] S53. Select objects with the same structure as each object in complex scene images using the robot grasping performance test set to determine the target object.

[0111] In this embodiment, a multimodal object knowledge base is constructed, which quantifies and integrates the physical properties (such as density, material, and inertial parameters), dynamic structures (such as the range of motion of articulated joints), and tilt angle thresholds of objects. A structured knowledge graph (such as ArtiKG) is formed through 3D modeling, mechanical testing, and semantic association, providing the robot with accurate object characteristic references and dynamic reasoning basis. Based on this knowledge base and preset diverse grasping scenarios, such as tilted object stability testing, material friction interference simulation, and dynamic path planning for articulated structures, a test set (i.e., the robot's grasping performance test set) is generated, incorporating complex conditions such as lighting changes and stacking occlusion. By quantifying indicators such as grasping success rate and response time, the robustness of the algorithm is verified and the grasping strategy is optimized. Finally, in complex scenarios, by combining visual feature extraction (such as point cloud matching and texture analysis), physical property verification (such as real-time tilt angle monitoring), and semantic association reasoning (such as functional area localization), hash indexing accelerates matching and dynamic priority sorting, the target object is quickly identified and a safe grasping path is planned, reducing the false detection rate and improving real-time performance. This solves problems such as object tipping risk and multi-object interference in complex scene environments.

[0112] In one optional embodiment, an experimental result is provided, and the specific experiment is as follows:

[0113] Implementation Details: All experiments were conducted on a workstation equipped with a 24GB GeForce RTX 4090 GPU. The simulation environment was built using PyBullet and included a UR5 robotic arm with a ROBOTIQ-85 gripper and an Intel RealSense L515 camera. Raw images were resized to 224×224 pixels for subsequent segmentation. The real-world experimental platform included a UR5 robotic arm with an RS-485 gripper and an Intel RealSense L515 camera with eye-to-hand calibration. The captured RGB-D images had a resolution of 1280×720. In the simulation experiments, GraspNet (grasping network) was used to generate the grasping pose; while in the real-world experiments, AnyGrasp (arbitrary grasping) was used to generate the grasping pose.

[0114] Benchmark methods: This application compares the proposed method with the following methods: (1) GraspNet and AnyGrasp, both of which are 6D grasping methods trained on large-scale real datasets; (2) RAM, which generates 2D operability graphs and upscales them to 3D for manipulation; and (3) ThinkGrasp, a visual-language grasping system that uses GPT-4o to perform reasoning in cluttered scenes.

[0115] Evaluation metrics: Grasp Success Rate (GSR) is used to evaluate the performance of simulation and real-world experiments. It is calculated as the percentage of successful grabs out of the total number of grabs.

[0116] Table 1 Simulation results of different methods in single object grasping tasks.

[0117]

[0118] Table 2 Simulation results of different methods in cluttered scene grasping tasks.

[0119]

[0120] In simulation experiments, the proposed method was compared with GraspNet and ThinkGrasp, and GraspNet was used to generate the grasping pose. The experiments included two tasks: single object grasping and cluttered scene grasping. In each test case, all methods were run 50 times under the same scene settings. In the single object grasping task, the object was placed alone on the table. The results in Table 1 show that the proposed method achieves an average grasp success rate (GSR) of 0.85, which is better than GraspNet and ThinkGrasp. For simple objects (such as spoons, hammers, and bowls), the proposed method performs comparable to ThinkGrasp; while for challenging objects such as scissors and wine glasses, the proposed method achieves GSRs of 0.62 and 0.66, respectively, which are improvements of 0.48 and 0.22 over GraspNet, and improvements of 0.14 and 0.20 over ThinkGrasp. This shows that the proposed method can adapt to objects with complex geometries.

[0121] In cluttered scene grasping tasks, objects are randomly placed, and the graspable positions of the target objects are marked with a red asterisk. Figure 6 The quantitative results in Table 2 show that the method of this application has an average crawling success rate of 0.77, which is significantly better than ThinkGrasp's 0.54.

[0122] The method provided in this application performs excellently when grasping cups, spoons, hammers, and screwdrivers. For example, for screwdrivers, ThinkGrasp's GSR is only 0.16, while the method in this application achieves 0.76. ThinkGrasp also has difficulties handling wine glasses because it prioritizes grasping the most salient objects, leading to grasping failures even when the target object is visible.

[0123] Table 3 Comparison results of different methods in real-world experiments

[0124]

[0125] In the experiments, we focused on task-oriented grasping in cluttered scenes and evaluated the proposed method on a variety of objects. Table 3 shows that our method achieved the highest average grasp success rate (GSR) of 0.83, outperforming RAM, AnyGrasp, and ThinkGrasp. For simple objects (such as bottles), all methods exhibited high grasp success rates. However, RAM performed poorly when handling complex geometries due to its limited operability memory; while ThinkGrasp showed competitiveness on specific objects (such as kettles and screwdrivers). Our method performed excellently across a wider range of objects, achieving grasp success rates of 0.80, 0.90, and 0.90 for cups, shovels, and pots, respectively, demonstrating its robustness to objects of different shapes and functions.

[0126] Figure 7 This section presents a visualization of the results of maneuverability region localization and task-oriented grasping pose generation. Relevant objects and their maneuverability masks are labeled with cyan bounding boxes and masks in the RGB image. The grasping pose is generated on the point cloud corresponding to the maneuverability mask and visualized on the object point cloud. (See attached image.) Figure 7 As shown, the grasping poses are densely distributed on the object parts that are graspable and operable, as identified by GPT-4o inference.

[0127] Extensive simulations and real-world experiments highlight the crucial role of VLMs and manipulability localization in grasping tasks, demonstrating their task understanding and manipulable part recognition capabilities. Furthermore, VLMs enable structured task understanding by parsing ambiguous user commands into specific tasks, enabling target object and manipulable part recognition, and generating grasping poses that meet task requirements. These advantages effectively improve the grasping success rate of AffordGrasp.

[0128] For example, to demonstrate the advantages of the proposed method, this application provides a case study of a mug, such as... Figure 8As shown. The first row displays the candidate grasping poses, and the second row depicts the optimal grasping poses chosen by different methods. ThinkGrasp, due to its 3×3 grid strategy (dividing the image into numbered regions, 1: top left, 9: bottom right), tends to select grasping poses near the center of the cropped image. However, this method often fails when dealing with objects with curved edges (such as cups) because center-based grasping poses lack stability due to insufficient contact points or improper force distribution. Similarly, AnyGrasp generates random grasping poses without operable reasoning, making task-oriented grasping unreliable. Figure 8 (b) and (c)). In contrast, this application combines operability reasoning with fine-grained grasping generation, enabling the generation of accurate and stable grasping poses. This highlights the importance of operability reasoning and demonstrates the effectiveness of this application in practical applications.

[0129] A third aspect of this application provides a control device, the control device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the computer program to implement the open vocabulary task-oriented crawling method of any of the above embodiments.

[0130] The fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements an open vocabulary task-oriented crawling method.

[0131] The fifth aspect of this application provides a computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer performs an open vocabulary task-oriented crawling method.

[0132] It is understood that the circuit structures, names, and parameters described in the above embodiments are merely examples. Those skilled in the art can also make readily conceived combinations and adjustments to the structural features of the above embodiments according to their needs, and the concept of this application should not be limited to the specific details of the above examples.

[0133] Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An open vocabulary task-oriented extraction method, characterized in that, The open vocabulary task-oriented crawling method includes: Input user language commands and complex scene images into a preset visual language model; The system identifies the target task corresponding to the user's language command through the preset visual language model, matches the target object associated with the target task in the complex scene image, and determines the optimal grasping component of the target object based on multiple functional components of the target object. Using the optimal grasping component and the complex scene image, multiple grasping candidate schemes are constructed; The optimal grasping pose is determined by optimizing each of the candidate grasping schemes. The steps of identifying the target task corresponding to the user's language command through the preset visual language model, matching the target object associated with the target task in the complex scene image, and determining the optimal grasping component of the target object based on multiple functional components of the target object include: The preset visual language model is used to identify the target task and task requirements corresponding to the user's language instructions. Based on the target task and task requirements, determine the object structure associated with the target task and task requirements; Match objects in the complex scene image that have the same structure as the object to generate a target object; The target object is divided into various functional components according to a first preset rule, and a set of functional components is constructed. The first preset rule is a rule for dividing the target object according to its functional parts. Identify the operability of each of the aforementioned functional components and construct an operability set; Based on the target task and the target object, the functional component with the highest grasping operability is selected from the functional component set and the operability set, and determined as the optimal grasping component. The open vocabulary task-oriented crawling method also includes: An object knowledge base is constructed based on the preset physical properties of the object, the object tilt angle, and the object tilt angle threshold. Based on the objects in the object knowledge base and multiple preset grasping scenarios, a robot grasping performance test set is constructed. The target object is determined by selecting objects with the same structure as each object in the complex scene image using the robot grasping performance test set.

2. The open vocabulary task-oriented crawling method according to claim 1, characterized in that, Before the step of inputting user language instructions and complex scene images into a preset visual language model, the method further includes: When a user's ambiguous language instruction and a complex scene image are received, the user's intention is identified based on the ambiguous language instruction and the complex scene image, and a user language instruction is generated.

3. The open vocabulary task-oriented crawling method according to claim 1, characterized in that, The open vocabulary task-oriented crawling method also includes: The complex scene image is divided into objects according to the second preset rule, and the bounding box of the target object is obtained. Convert the image corresponding to the target object within the bounding box into an object mask image; Calculate the operability mask of the optimal grasping component on the object mask image.

4. The open vocabulary task-oriented crawling method according to claim 3, characterized in that, The step of constructing multiple grasping candidate schemes using the optimal grasping component and the complex scene image includes: Obtain the depth image and camera intrinsic parameters corresponding to the complex scene image; The operability mask corresponding to the optimal grasping component, the depth image corresponding to the complex scene image, and the camera intrinsic parameters are input into the preset grasping pose generation model. The depth image is converted into a partial viewpoint point cloud using the camera intrinsic parameters. The partial viewpoint point cloud is filtered using the operability mask corresponding to the optimal grasping component to generate the target point cloud; Based on the target point cloud, multiple grasping candidate schemes are constructed. Each grasping pose candidate is represented by the rotation matrix, translation vector, minimum width of the gripper, and confidence score of the gripper in the camera coordinate system.

5. The open vocabulary task-oriented crawling method according to claim 4, characterized in that, The step of optimizing each of the candidate grasping schemes to determine the optimal grasping pose includes: Obtain the confidence score corresponding to the grasping pose in each of the grasping candidate schemes; All crawling candidate schemes are sorted from high to low according to the ratio of the confidence score to the distance between the translation vector and the center of the operability mask in the crawling candidate scheme; The optimal grasping pose is determined based on the grasping pose among the highest-ranked grasping candidate schemes. Perform the operation of the optimal grasping pose in the preset scenario.

6. A control device, characterized in that, The control device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the open vocabulary task-oriented crawling method according to any one of claims 1-5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it implements the open vocabulary task-oriented crawling method as described in any one of claims 1-5.

8. A computer program product, characterized in that, The computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, wherein when the program instructions are executed by a computer, the computer performs the open vocabulary task-oriented crawling method as described in any one of claims 1-5.

Images