Articulated object robot grabbing method based on visual language action model

Abstract

The invention discloses a hinged object robot grabbing method based on a visual language action model. A hinged object is put in a meta-universe environment for rendering, a geometric center point of the hinged object is used as a core reference point, six instance representation methods are adopted to mark the hinged object, and a sample database is established; generating six training tasks based on six instance representation methods; constructing a visual language action model based on LLaMA2 and SPHINX-X, and performing supervised training on the visual language action model according to a training task and a sample database to obtain an optimal visual language action model; constructing a joint optimizer, and selecting an optimal result from output results of the visual language action model by decomposing a hinged object manipulation sequence planning task; and S3, using the optimal visual language action model obtained in the step S3 and a joint optimizer to determine the hinged object robot grabbing method.

Description

Technical Field

[0001] The present invention relates to the field of metaverse and artificial intelligence technology, and is particularly applicable to a robot grasping method for articulated objects based on a visual language action model. Background Art

[0002] With the development of metaverse services, the industrial metaverse, built through digital twins, virtual-real coexistence, and virtual-real collaboration, demonstrates tremendous potential for the collaborative development and control of digital products. Services such as equipment operation and maintenance and product testing can now be performed on virtual platforms, saving significant costs. Furthermore, the digital advantages of the metaverse can be combined with the practical needs of real-world robotic operations, transferring motion planning capabilities honed in the metaverse to real-world applications, further developing robotic skills.

[0003] Articulated objects are present in large numbers in daily life and industrial applications, such as doors, refrigerators, and drawers in daily life, but manipulating articulated objects in the industrial metaverse is still a complex movement.

[0004] Currently, methods for manipulating articulated objects in virtual worlds are Figure 1 As shown in (a), the kinematic structure of the articulated object, the posture of the articulated parts, and the joint parameters can be obtained through the visualization model to express the articulation operation; Figure 1 As shown in (b), the prompt template can be used to generate articulated operations through behavioral learning and large language model learning; Figure 1 As shown in (c), a large multimodal language model can be used to automatically generate articulated operations by identifying and learning from multiple sources such as images, text, and videos. While the three aforementioned methods are all capable of generating articulated operations, they all rely heavily on manually annotated datasets and, in practical applications, exhibit poor generalization to objects of unseen categories in the dataset, lacking category-independent capabilities.

[0005] Furthermore, current methods for generating manipulation of articulated objects often employ a single-instance mechanism to describe these objects, making them difficult to handle complex articulated objects with multiple operable parts. Furthermore, given the diversity of real-life users and their varying command preferences, current methods for generating manipulation of articulated objects are unable to identify the interdependencies and correspondences between different reasoning tasks within a defined workflow, enabling automated decision-making and continuous optimization.

[0006] Therefore, a better and more efficient fine-grained representation method for articulated objects is needed, which can accurately characterize the features of the operable parts of each articulated object and can make automatic decisions and continuously optimize the manipulation method of articulated objects. Summary of the Invention

[0007] The purpose of the present invention is to provide a robot grasping method for articulated objects based on a visual language action model, which is used to solve the problems of not being able to accurately characterize the features of the operable parts of each articulated object, not being able to adapt to user instruction preferences, and automatically making decisions according to a certain workflow to obtain a robot grasping method for articulated objects.

[0008] To achieve the above object, the present invention adopts the following technical solutions: The method for grasping an articulated object by a robot based on a visual language motion model of the present invention comprises the following steps: S1, place the articulated object in the metaverse environment for rendering, use the geometric center point of the articulated object as the core reference point, use six instance representations to mark the articulated object, and establish a sample database; S2, generates six training tasks based on six instance representations; S3, building a visual language action model based on LLaMA2 and SPHINX-X, and performing supervised training on the visual language action model according to the training task and sample database to obtain the optimal visual language action model; S4, constructing a joint optimizer to decompose the articulated object manipulation sequence planning task and select the best result from the output of the visual language action model; S5, using the optimal visual language action model and joint optimizer obtained in step S3 to determine the robot grasping method for articulated objects.

[0009] Furthermore, the six instance representations use object category, bounding box, rotation axis, grasping direction, contact point, and motion path to represent an articulated object; the object category is represented by the name of the articulated object and the joint type; the bounding box is represented by eight vertices; the rotation axis is represented by two points; the grasping direction is represented by a vector; and the moving path is represented by a set of multiple points.

[0010] Furthermore, the six training tasks are respectively used to obtain information of six instances of articulated objects, including object category recognition detection for identifying the category and joint type of the articulated object; bounding box detection for obtaining the spatial position of the articulated object in the metaspace; axis detection for obtaining the axis position of the articulated object; contact point detection for identifying the contact point for operating the articulated object; grasping direction detection for identifying the grasping direction for operating the articulated object; and motion path detection for converting operation instructions into specific three-dimensional motion waypoints based on the object category, bounding box, axis, grasping direction, and contact point.

[0011] Furthermore, the visual language action model includes a hybrid visual encoder, a projection layer and an LLaMA2 model; the hybrid visual encoder integrates CLIP-VIT, CLIP-ConvNeXt, DINOv2-VIT and Q-Former visual encoders, and simultaneously understands the image features of articulated objects; the projection layer maps the visual markers of the visual encoder to a visual language alignment marker; the LLaMA2 model is used to identify the task labels corresponding to the operation instructions, so that the visual language action model outputs results according to the task labels.

[0012] Furthermore, the joint optimizer includes an advisor module and a strategist module; the advisor module plans and issues operation instructions for the articulated object according to the manipulation sequence of object category recognition detection, bounding box detection, rotation axis detection, contact point detection, grasping direction detection, and motion path detection; the strategist module selects the best result from multiple output results of the visual language action model.

[0013] Furthermore, the operation instructions issued by the advisor module will call the optimal visual language action model obtained in step S3, identify the task label, and obtain multiple possible output results.

[0014] Furthermore, the best result selected by the strategist module in the previous operation sequence will serve as a constraint condition for the operation instructions issued by the advisor module in the next operation sequence.

[0015] Furthermore, the optimal result is determined for the object category of the articulated object based on the similarity between the image of the articulated object and the candidate object category; for the bounding box of the articulated object, a unique bounding box is determined based on the determined object category; for the rotation axis of the articulated object, the optimal result is determined based on the distance between the candidate rotation axis of the articulated object and the center point of the bounding box; for the contact point of the articulated object, the optimal result is determined based on the proximity between the candidate contact point of the articulated object and the contact surface; for the grasping direction of the articulated object, the optimal result is determined based on the cosine similarity between the candidate grasping direction of the articulated object and the rotation axis; and the motion path of the articulated object is uniquely determined by the optimal results of the object category, bounding box, rotation axis, contact point, and grasping direction.

[0016] Furthermore, it also has a feedback update strategy, which updates the original label set L in the training task with the appropriate results output by the visual language action model according to the four evaluation methods of the strategist to obtain a new label set L'. If L' is different from L, the new label set L' is used to train the projection layer of the visual language action model.

[0017] The advantage of the present invention is that it uses six-instance representation to mark articulated objects, achieving a refined description of multiple operable parts of the articulated object. Based on this refined description, it is possible to identify multiple different articulated components on an object and generate operation methods.

[0018] The joint optimizer constructed in this paper decomposes the task of planning sequences for articulated object manipulation. This automatically decomposes the task, allowing the VLA model to gradually acquire feature data for the articulated object. This improves the accuracy of feature data acquisition and makes the entire reasoning process more logically relevant. This paper also provides adaptive update capabilities, encompassing the full integration of automatic dataset construction and articulated manipulation, resulting in a trained visual language action model with enhanced generalization capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Schematic diagram of the flow of existing methods for generating articulated object manipulation.

[0020] Figure 2 A schematic diagram of a model of an articulated object robot grasping method based on a visual language action model of the present invention is constructed.

[0021] Figure 3 Labeling articulated objects for six instance representations. Figure 3 a is a labeled diagram of six instance representations of storage furniture drawers; Figure 3 b is the six instance representation labeling diagram of the trash can; Figure 3 c is the labeled graph of the six instance representations of the table; Figure 3 d is the labeling diagram of the six instance representations of the bottle.

[0022] Figure 4 This is a diagram of the VLA model architecture constructed by the present invention. Figure 4 a is a schematic diagram of the input and output of six training tasks; Figure 4 b Architecture diagram of the VLA model constructed by the present invention, and its input and output for the object category recognition and detection task.

[0023] Figure 5 Diagram of the joint optimizer and VLA model architecture constructed for the present invention.

[0024] Figure 6 This is a comparative experimental result diagram of Example 3 of the present invention.

[0025] Figure 7 This is a diagram of the test process of Example 4 of the present invention.

[0026] Figure 8 This is a test result diagram of Example 4 of the present invention. DETAILED DESCRIPTION

[0027] The following is a clear and complete description of the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts are within the scope of protection of the present invention.

[0028] First, the professional terms in this application are explained as follows: Articulated objects: Articulated objects are composed of multiple rigid components connected by joints, each of which can rotate or translate around the joints. This type of object is commonly used in furniture, doors, windows, and other devices. Compared to ordinary rigid objects, which have only six degrees of freedom (DoF), articulated objects have a higher degree of freedom and are more difficult for robots to understand and interact with.

[0029] MLLM stands for Multimodal Large Model. MLLM is a large language model capable of processing and understanding multimodal data. Compared to traditional language models (such as GPT or BERT), which only process text, MLLM can combine multiple modalities (such as text, images, video, audio, and even 3D data) for unified processing and reasoning.

[0030] The Vision-Language-Action (VLA) model, short for Vision-Language-Action (VLA), is a multimodal AI model that integrates visual perception, language understanding, and action generation. It can process images, text instructions, and operational tasks in complex scenes, forming a closed loop from perception to decision-making and execution. For example, the VLA model can understand user-provided natural language instructions (such as "grab the red cup on the table"), analyze visual information (such as the location and features of objects in the scene), and generate corresponding action plans (such as the grasping path of a robotic arm). This model is widely used in fields such as robotic control and industrial automation, enabling natural interaction and efficient task execution through a unified multimodal processing framework.

[0031] Example 1 The method for grasping an articulated object by a robot based on a visual language action model of the present invention specifically comprises the following steps: Step 1: Use multi-instance multi-label geometric center representation to represent articulated objects.

[0032] Existing techniques use a single-instance mechanism to describe articulated objects. In practice, this mechanism often encounters limitations when dealing with complex articulated objects. In particular, when an articulated object has a complex structure and contains multiple operable parts, a single-instance, single-label representation cannot fully capture all operable parts of the articulated object.

[0033] like Figure 3As shown, the articulated object "table" contains two instances: "drawer" and "door." Traditionally, the entire "table" is treated as a single instance entity and assigned a single label (e.g., "table"). This approach to representing articulated objects ignores the fact that the "drawer" and "door" within the table can also interact with each other as independent articulated components.

[0034] The present invention's method for robotic grasping of articulated objects based on a visual language action model uses a multi-instance, multi-label geometric center representation to represent articulated objects. The object is rendered in a metaverse environment, its geometric center point is used as the core reference point, and the six-instance representation is used to label the articulated object. The six-instance representation represents the articulated object using object category, bounding box, rotation axis, grasping direction, contact points, and movement path.

[0035] The object category mainly includes the name of the articulated object and the joint type, such as "drawer, sliding", "door, rotating", etc. The bounding box is defined by eight vertices, denoted as The grasping direction is determined by the vector Definition, the contact point consists of a single point Definition; the movement path is composed of the set Definition: The axis of rotation is defined by two points, denoted as .

[0036] Step 2: Generate six training tasks based on the six instance representations.

[0037] The six instances are converted into specific training tasks to obtain six instance data of specific articulated objects.

[0038] There are six types of training missions as follows: 1. Object category recognition. Object category recognition is used to guide the VLA model to identify specific object categories and joint types, which is an important task in the field of machine vision. In this paper, the visual language embedding model ViT-L / 14 is used to map images and corresponding object category labels into a shared embedding space. Specifically, the ViT-L / 14 model uses two different encoders (image encoder and text encoder) to transform the input image into a shared embedding space. and candidate text Converted to a high-dimensional vector representation, the expression is shown in formula (1): (1) in, and They are the image and text encoders of the ViT-L / 14 model, and are the image and text feature vectors in the shared embedding space.

[0039] Then use and The cosine similarity between them is used as the object category recognition task label The specific expression is shown in formula (2): (2) 2. Bounding box detection. The purpose of bounding box detection is to guide the VLA model to identify and confirm the spatial position of the articulated object in the metaspace. In the camera coordinate system, It is represented by 8 vertices, denoted as, , Figure 4 In (a), there are eight vertices outlined by yellow lines.

[0040] To calculate , first determine the geometric center point of the articulated object in the camera coordinate system and the length, width, and height of half the minimum enclosing rectangle that encloses the articulated object Then use the center point and The value of is added and subtracted to calculate 8 points The expression is shown in formula (3): (3) In addition, some articulated objects contain both sliding and rotating joint axes. For example, Figure 3 The "mouth" of the "bottle" in d can be pressed and rotated. For this type of special articulated object, in order to distinguish the different action capabilities of the same part, it is necessary to manually Assign labels and compare them to object category recognition task labels The two joint axes in are associated.

[0041] 3. Rotation axis detection. Rotation axis detection is to guide the VLA model to identify the joint axis. This invention specifies the starting point and end point. To determine the shaft position.

[0042] For sliding type shaft, the shaft The starting point and end point of and For the rotary shaft, the shaft The starting point, end point and center point and The expression is as shown in formula (4): (4) Among them, Prismatic represents a sliding shaft, and Revolute represents a rotating shaft.

[0043] 4. Contact point detection. Contact point detection is designed to guide the VLA model to identify the contact points of articulated objects, which is crucial for stable manipulation and efficient control. Figure 4 As shown in (a), in three-dimensional Six feasible aspects There are six possible touchpoints , at a specific contact surface There is only one point on As a point of contact.

[0044] For a sliding articulated object, first use the formula (3) calculate The six sides , as shown in formula (5): (5) From formula (5), we can see that each face are associated with 4 vertices. For each face , the corresponding geometric center The calculation method is shown in formula (6): (6) Next, define each face Direction of movement and the joint axis vector . Depend on and the 3D center point The difference is calculated as shown in formula (7). From the starting point in formula (4) and end point Definition, as shown in formula (8): (7) (8) Finally, calculate the contact surface and contact points ,because It is the outermost force-bearing surface, so Direction of movement Should be with The most consistent. So the calculation and The dot product value of is calculated and the surface with the largest dot product value is selected as the contact surface, as shown below: (9) (10) For rotating objects, the present invention introduces sampling weights ,in Based on the sampling point With shaft The distance is determined by For formula (3) Perform weighted sampling to determine : (11) 5. Grasping direction detection. Grasping direction detection is designed to guide the VLA model to identify the direction of the gripper. , thereby improving the stability of operation.

[0045] For sliding articulated objects, the grasping direction needs to be maintained Parallel to the direction of the joint axis.

[0046] For rotating articulated objects, The calculation of is more complicated. Because the grasping direction of the rotating articulated object is different from that of the sliding articulated object. Should For a rotating articulated object, first use the contact force in formula (11) and the axis of rotation in formula (4) Constructed by cross product , to express the vertical relationship, the expression is shown in formula (12): (12) 6. Motion planning. Motion planning aims to guide the VLA model to convert user instructions and geometric features into specific 3D motion waypoints. When constructing motion planning, the VLA model can generate a set of motion waypoints. Each waypoint contains the position of the robot's end effector.

[0047] For sliding joints, according to the contact point and the axis vector , waypoints can be calculated , as shown in formula (13): (13) in, and Represent the movement speed and time step respectively.

[0048] For rotational joints, you also need to use the axis vector and rotation angle , calculate the rotation matrix according to the Rodriguez rotation formula ,Record Finally, when the contact point When rotating by the rotation matrix, the corresponding waypoints are generated It can be calculated as follows: (14) Step 3: Design and build the VLA model architecture.

[0049] In the present invention, the VLA model is developed and constructed based on SPHINX-X and LLaMA2. Figure 4 As shown in (b), unlike the Mixture of Visual Experts (MOV) used in SPHINX-X (a multimodal large language model (MLLM) series developed based on SPHINX, which aims to improve model performance by expanding data and parameter size), the present invention uses a Mixed Four Vision Encoder. The Mixed Four Vision Encoder integrates the CLIP-VIT (a multimodal model that combines the CLIP model with the Vision Transformer (ViT) architecture, which has demonstrated strong performance in image classification, image and text retrieval, and other fields) and CLIP-ConvNeXt (a multimodal model that combines the CLIP model with the ConvNeXt architecture, which has been widely used in image classification, image and text retrieval, and other fields) visual encoders and DINOv2-VIT (a self-supervised visual model based on the Vision Transformer (ViT) architecture launched by Meta AI) to extract local semantic features, while using Q-Former (a neural network architecture in multimodal large models) to summarize global visual features. In order to ensure the robustness of the four hybrid visual encoders, the final constructed VLA model (vision-language-action model) should understand both the global and local features of the input image.

[0050] The present invention separates the visual image into four sub-images and a low-resolution image. The five images are simultaneously input into the hybrid visual encoder. Then, a projection layer is designed to map the five visual labels to a visual language alignment label in the projection network to complete the Figure 4 The spatial alignment of visual and textual features in (b) realizes the spatial alignment of visual and language features, which not only enhances the semantic understanding ability of the VLA model, but also improves the accuracy and efficiency of the fusion of visual and textual features.

[0051] The training tasks are then input into the LLaMA2 model (an open source large language model launched by Meta), and the total label set and sample database of the training tasks constructed in step 2 are used to perform supervised training on each training task to obtain the optimal visual language action model.

[0052] Step 4: Build a joint optimizer to decompose the articulated object manipulation sequence planning task and select the best result from the output of the vision-language action model.

[0053] In order to ensure that the VLA model can generate accurate 3D motion waypoints for articulated object manipulation, this paper proposes a hierarchical kinematic perception facilitation method called Joint Optimizer (JO). Since some inference results are interdependent in the hierarchical prompting process, such as Figure 5 As shown in Figure 1, to capture the bounding box of an articulated object, it is necessary to obtain the correct result from the object category identification task label OCI. In order to better guide the VLA model to provide accurate motion waypoints, JO decomposes the manipulation sequence planning and evaluates the candidate responses of the VLA model according to the manipulation sequence planning.

[0054] like Figure 5 As shown in Figure 1, the core of JO consists of two components: the Advisor and the Strategist. In JO, the Advisor follows a six-instance representation instruction template and plans and issues manipulation commands for articulated objects according to a manipulation sequence consisting of object category recognition and detection, bounding box detection, rotation axis detection, contact point detection, grasping direction detection, and motion path detection. The manipulation commands issued by the Advisor module invoke the optimal visual language action model obtained in step S3 to identify the task label and obtain multiple possible output results.

[0055] The strategist module selects the best result from multiple possible output results of the vision-language-action model (VLA).

[0056] like Figure 5 As shown in FIG, in order to enhance the adaptability of the VLA model, the present invention also adds an adaptive feedback strategy. For example, for the "bottle" object, the present invention uses a basic template, such as "How do I use this bottle?" Figure 5 It is easy to observe from the OCI results in that there are two candidate results, such as “mouth. revolute.” (bottle, rotation type) and “mouth. prismatic.” (bottle, sliding type).” However, “mouth. prismatic.” is the appropriate answer because it is the only articulated target for pressing the bottle.

[0057] Assume that Advisor has a candidate result. The design purpose of Strategist is to score all possible results and select the best result according to the four hierarchical kinematic perception promotion processes. In this way, when there are multiple candidate results, such as when there are multiple object category recognition task labels, , multiple shafts , multiple touchpoints and multiple grasping directions If Strategist recognizes task labels from multiple object categories If the best candidate is selected from the above, the subsequent bounding box BBOX, rotation axis, contact point, grasping direction and motion waypoint results are unique, such as Figure 5 The blue arrows connect mouth, prismatic, BBOX, 、 , G0. That is to say, the best result selected by the strategist module in the previous operation sequence will serve as the constraint condition for the operation instructions issued by the advisor module in the next operation sequence.

[0058] exist Figure 5 In the Strategist for object category recognition OCI, the goal is to get the candidate OCI results The best result is selected from the above. The similarity between the image and each candidate OCI result is obtained by selecting the most similar OCI result according to formula (1) as follows: (15) The Strategist for pivot detection aims to evaluate multiple pivots. and Center point The degree of closeness between them. For multiple axes , calculate their rotation axis vectors according to formula 8 And select the optimal joint axis detection result according to formula (16) : (16) in express The algorithm evaluates multiple joint axes. and Center point The distance between them.

[0059] The Strategist for touchpoint detection aims to evaluate multiple touchpoints. The contact surface in formula (9) The degree of proximity between them and the optimal contact point detection result are selected and contact surface The distance and the optimal contact point detection result are selected according to formula (17) : (17) in, Represents the contact surface direction of movement, represent The algorithm uses distance to evaluate multiple contact points. and contact surface The degree of closeness between them.

[0060] The Strategist for grasp direction detection aims to evaluate multiple grasp directions. and shaft The alignment between them is calculated and the optimal grasping direction is selected according to formula (18) : (18) The algorithm calculates the cosine similarity between each grasping direction result and the rotation axis. For a specific rotation axis type, formula (18) will select the appropriate grasping direction. For sliding objects, since the grasping direction is parallel to the joint axis, the grasping direction with the highest similarity score will be selected. For rotated objects, since the grasping direction is perpendicular to the joint axis, the grasping direction with the lowest similarity score is selected. .

[0061] Finally, to enhance the VLA model's adaptability in handling diverse scenarios, this paper also provides a feedback update strategy. When the VLA model is asked, "How do I use this bottle?", the appropriate OCI answer should be "mouth.prismatic," not "mouth.revolute." Therefore, the original OCI label set used in the training task needs to be updated to ensure the VLA model produces appropriate responses.

[0062] Specifically, the feedback update strategy of the present invention utilizes the scores of appropriate answers Update the original label set in the training task , get a new label set : (19) in, Refers to the four scoring formulas for strategists. Represents the updated tag set. If and Different, we will use the new label set The projection layer in the VLA model is trained to enhance the adaptability of the VLA model in detecting instances with different labels.

[0063] In step 5, the optimal visual-language-action model obtained in step S3 and the joint optimizer are used to determine the robot's grasping method for articulated objects. Specifically, based on the object image and user input, the joint optimizer's advisor module plans and issues manipulation instructions for the articulated object according to a manipulation sequence consisting of object category recognition and detection, bounding box detection, rotation axis detection, contact point detection, grasping direction detection, and motion path detection. The optimal visual-language-action model identifies the task label and obtains multiple possible outputs. The joint optimizer's strategist module then selects the optimal output from the visual-language-action model to determine the robot's grasping method for articulated objects.

[0064] Example 2 The present invention uses the training tasks generated in step 2 to train the VLA model designed in step 3. The training tasks use the default Visual Question Answering (VQA) dataset format of the SPHINX framework. In order to evaluate the category-independent ability of the present invention, 10 object categories are selected from the generated training tasks as "seen categories", and another 20 object categories are selected for training the VLA model, which are called "unseen categories". In addition, the present invention uses the SPHINX-v2-1k model obtained from the official resource library as the pre-training basis. Figure 4 As shown in (b), the hybrid visual encoder remains frozen throughout the training process, and only the projection layer and LLaMA2-13B are fine-tuned to maintain the integrity of the pre-trained features.

[0065] Example 3 In order to evaluate the performance of the present invention, the present invention adopted the evaluation method described in ManipLLM in the comparative experiment, and used the Franka Panda robot gripper and the Sapien simulator to perform evaluation in the metaverse environment.

[0066] 1. Comparative Analysis in the Metaverse. The operation success rate is a metric used to measure the performance of our method across 20 previously seen categories and 10 previously unseen categories. We compared our method with six baseline methods: Where2Act, UMPNet, FlowBot3D, Implicit3D, ManipLLM, and A3VLM.

[0067] Where2Act uses a neural network to map pixels to six basic actions, processing point cloud data by predicting actionable information for each pixel to manipulate articulated objects.

[0068] UMPNet is similar to Where2Act in that it ensures that the end-effector is oriented perpendicular to the object surface when predicting the contact force point.

[0069] Flowbot3D predicts potential movement directions by representing the three-dimensional articulation flow and deploys an analytical motion planner to determine the articulation strategy.

[0070] Implicit3D employs a transporter network to identify contact force points on articulated objects, helping to guide the pose of the end-effector when performing manipulation tasks by leveraging hybrid 3D representations, cross-attention, and implicit reconstruction mechanisms.

[0071] ManipLLM utilizes a fine-tuned vision-language model combined with object category understanding, affordance reasoning, and object-centric pose prediction to simulate manipulation.

[0072] A3VLM focuses on the articulated structure and motion tolerance of articulated objects, and uses visual language models to detect the bounding box, axis and joint type of objects to realize the manipulation process.

[0073] from Figure 6 As can be seen from the data, the present invention outperforms other baseline methods on 18 of the 30 object categories. For example, in the "door" category, the present invention achieved a success rate of 98%, 17% higher than the 82% of the second-ranked A3VLM. Furthermore, in the "success rate of seen categories," the present invention achieved an AVG (average success rate) of 83%, 10% higher than the second-ranked A3VLM. In the "success rate of unseen categories," the present invention achieved an AVG (average success rate) of 76%, 1.3% higher than the second-ranked A3VLM, demonstrating the present invention's strong zero-error capability.

[0074] Example 4 To verify the operational performance of the present invention in real-world applications, the present invention generates manipulations of articulated objects in a metaverse environment and conducts reasoning experiments in a real-world environment.

[0075] The present invention first initializes the Teledyne DALSA TM461-E2 depth camera and calibrates the intrinsic and extrinsic parameters of the depth camera to capture images of articulated objects. Then, in order to eliminate the influence of the background difference between the camera and the reference image, Segment Anything is used to extract image semantics and replace the image background. The VLA model then gives the motion waypoints of the manipulator that opens the captured articulated object. The experiment uses the inverse kinematics of the end effector based on the cubic spline trajectory interpolation mechanism to calculate the motion waypoints of the manipulator. Among them, the CyberTech Pincer 2F120S manipulator and the Faurecia FR5 6-DOF manipulator were used. like Figure 8 As shown in Figure 1, 10 real-world experiments were conducted involving five different types of articulated objects: microwave ovens, kettles, pots, lids, and drawers. The success rate results are shown in Figure 1. Figure 8 As shown in Figure 2, the present invention achieved a 90% success rate in the kettle and pot subcategory and an 80% success rate in the microwave, lid, and drawer subcategory. Analysis of these experimental results demonstrates that the manipulation knowledge developed in the real-world environment can be transferred from the metaverse environment and applied to a new category of robotic arms, demonstrating that this method maintains category-independence when transferred from the metaverse environment to the real-world environment.

Claims

1. A robot grasping method for articulated objects based on a visual language action model, characterized in that: The following steps are involved: S1, place the articulated object in the metaverse environment for rendering, use the geometric center point of the articulated object as the core reference point, use six instance representations to mark the articulated object, and establish a sample database; S2, generates six training tasks based on six instance representations; S3, building a visual language action model based on LLaMA2 and SPHINX-X, and performing supervised training on the visual language action model according to the training task and sample database to obtain the optimal visual language action model; S4, constructing a joint optimizer to decompose the articulated object manipulation sequence planning task and select the best result from the output of the visual language action model; S5, using the optimal visual language action model and joint optimizer obtained in step S3 to determine the robot grasping method for articulated objects.

2. The method for grasping an articulated object by a robot based on a visual language motion model according to claim 1, characterized in that: The six instance representations use object category, bounding box, rotation axis, grasping direction, contact point, and motion path to represent an articulated object; the object category is represented by the name of the articulated object and the joint type; the bounding box is represented by eight vertices; the rotation axis is represented by two points; the grasping direction is represented by a vector; and the moving path is represented by a set of multiple points.

3. The method for grasping an articulated object by a robot based on a visual language motion model according to claim 1, characterized in that: The six training tasks are used to obtain information about six instances of articulated objects, including object category recognition detection for identifying the category and joint type of the articulated object; bounding box detection for obtaining the spatial position of the articulated object in the meta-space. ; Axis detection is used to obtain the axis position of the articulated object; contact point detection is used to identify the contact points of the articulated object; grasping direction detection is used to identify the grasping direction of the articulated object; motion path detection is used to convert operation instructions into specific three-dimensional motion waypoints based on object category, bounding box, axis, grasping direction, and contact points.

4. The method for grasping an articulated object by a robot based on a visual language motion model according to claim 1, wherein: The visual-language-action model includes a hybrid visual encoder, a projection layer, and an LLaMA2 model; the hybrid visual encoder integrates CLIP-VIT, CLIP-ConvNeXt, DINOv2-VIT, and Q-Former visual encoders, while understanding the image features of articulated objects; the projection layer maps the visual markers of the visual encoder to a visual-language alignment marker; the LLaMA2 model is used to identify the task labels corresponding to the operation instructions, so that the visual-language-action model outputs results based on the task labels.

5. The method for grasping an articulated object by a robot based on a visual language motion model according to claim 1, characterized in that: The joint optimizer includes an advisor module and a strategist module; the advisor module plans and issues operation instructions for articulated objects according to the manipulation sequence of object category recognition detection, bounding box detection, rotation axis detection, contact point detection, grasping direction detection, and motion path detection; the strategist module selects the best result from multiple output results of the visual language action model.

6. The method for grasping an articulated object by a robot based on a visual language motion model according to claim 5, characterized in that: The operation instructions issued by the advisor module will call the optimal visual language action model obtained in step S3, identify the task label, and obtain multiple possible output results.

7. The method for grasping an articulated object by a robot based on a visual language motion model according to claim 5, characterized in that: The best result selected by the strategist module in the previous operation sequence will serve as a constraint condition for the operation instructions issued by the advisor module in the next operation sequence.

8. The method for grasping an articulated object by a robot based on a visual language motion model according to claim 5, characterized in that: The optimal result is determined as follows: for the object category of the articulated object, the optimal result is determined based on the similarity between the image of the articulated object and the candidate object category; for the bounding box of the articulated object, a unique bounding box is determined based on the determined object category; for the rotation axis of the articulated object, the optimal result is determined based on the distance between the candidate rotation axis of the articulated object and the center point of the bounding box; for the contact point of the articulated object, the optimal result is determined based on the proximity between the candidate contact point of the articulated object and the contact surface; for the grasping direction of the articulated object, the optimal result is determined based on the cosine similarity between the candidate grasping direction of the articulated object and the rotation axis; the motion path of the articulated object is uniquely determined by the optimal results of the object category, bounding box, rotation axis, contact point, and grasping direction.

9. The method for grasping an articulated object by a robot based on a visual language motion model according to claim 1, characterized in that: It also has a feedback update strategy, which updates the original label set L in the training task with the appropriate results output by the visual language action model according to the four evaluation methods of the strategist to obtain a new label set L'. If L' is different from L, the new label set L' is used to train the projection layer of the visual language action model.

Images