Robotic grasping method for articulated objects based on visual language action model

Abstract

The application discloses a kind of articulated object robot grabbing methods based on visual language action model, articulated object is placed in meta-universe environment rendering, using the geometric center point of articulated object as core reference point, six instance representation method is used to mark articulated object, and sample database is established;Based on six instance representation method, six kinds of training tasks are generated;Based on LLaMA2, SPHINX-X constructs visual language action model, according to training task and sample database, the visual language action model is supervised training respectively, and the optimal visual language action model is obtained;Joint optimizer is constructed, the articulated object manipulation sequence planning task is decomposed, and the best result is selected from the output result of the visual language action model;The articulated object robot grabbing method is determined using the optimal visual language action model and joint optimizer obtained by S3 step.

Description

Technical Field

[0001] This invention relates to the fields of metaverse and artificial intelligence technology, and is particularly applicable to a method for grasping articulated objects by a robot based on a visual language action model. Background Technology

[0002] With the development of metaverse services, the industrial metaverse, constructed through digital twins, virtual-real coexistence, and virtual-real collaboration, has demonstrated enormous potential in the collaborative development and control of digital products. Equipment maintenance, product testing, and other services can all be conducted on virtual platforms, saving significant costs. Furthermore, the digital advantages of the metaverse can be combined with the actual needs of robot operation in the real world, allowing motion planning capabilities honed in the metaverse to be transferred to real-world applications, thus better developing robot skills.

[0003] Hinged objects are ubiquitous in daily life and industrial applications, such as doors, refrigerators, and drawers in everyday life, but manipulating hinged objects in the industrial metaverse is still a complex motion.

[0004] Currently, methods for manipulating articulated objects in virtual worlds, such as Figure 1 As shown in (a), the motion structure of the articulated object, the posture of the articulated components, and the joint parameters can be obtained through a visualization model to express the articulation operation; such as Figure 1 As shown in (b), articulated operations can be generated using prompt templates through behavioral learning and large language model learning; such as Figure 1 As shown in (c), articulation operations can be automatically generated by recognizing and learning from multiple sources such as images, text, and videos using a multimodal large language model. All three methods described above can generate articulation operations, but they all heavily rely on manually labeled datasets and exhibit poor generalization to objects of unseen categories in the dataset during practical applications, lacking category-independent capabilities.

[0005] Meanwhile, current manipulation methods for generating articulated objects mostly employ a single-instance mechanism to describe articulated objects, making it difficult to handle complex articulated objects containing multiple operable parts. Furthermore, given the diversity of real-life users and their varying command preferences, current manipulation methods for generating articulated objects cannot determine the interdependencies and correspondences between different inference tasks based on a given workflow, nor can they make automatic decisions and continuously optimize.

[0006] Therefore, a better and more efficient fine-grained representation method for articulated objects is needed, capable of accurately depicting the characteristics of each operable part of the articulated object, and a manipulation method for articulated objects that can make automatic decisions and continuously optimize. Summary of the Invention

[0007] The purpose of this invention is to provide a method for grasping articulated objects by a robot based on a visual language action model, which solves the problems of not being able to accurately depict the features of the operable parts of each articulated object, not being able to adapt to user command preferences, and automatically making decisions to obtain a method for grasping articulated objects based on a certain workflow.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] The articulated object grasping method based on a visual language action model of the present invention includes the following steps: The articulated object grasping method based on a visual language action model is characterized by including the following steps:

[0010] S1, place the hinged object in the metaverse environment for rendering, use the geometric center point of the hinged object as the core reference point, use six instance representations to mark the hinged object, and establish a sample database.

[0011] S2 generates six training tasks based on six instance representations;

[0012] S3. Construct a visual language action model based on LLaMA2 and SPHINX-X. Perform supervised training on the visual language action model according to the training task and sample database to obtain the optimal visual language action model.

[0013] S4. Construct a joint optimizer to select the best result from the output of the visual language action model by decomposing the articulated object manipulation sequence planning task.

[0014] S5 uses the optimal visual language action model and joint optimizer obtained in step S3 to determine the robot's grasping method for articulated objects.

[0015] Furthermore, the six instance representations use object category, bounding box, pivot, gripping direction, contact point, and motion path to represent the articulated object; the object category is represented by the name of the articulated object and the joint type; the bounding box is represented by eight vertices; the pivot is represented by two points; the gripping direction is represented by a vector; and the motion path is represented by a set of multiple points.

[0016] Furthermore, the six training tasks are used to acquire information about six instances of the articulated object, including object category recognition and detection to identify the category and joint type of the articulated object; bounding box detection to obtain the spatial position of the articulated object in the metaspace; rotation axis detection to obtain the rotation axis position of the articulated object; contact point detection to identify the contact point of the articulated object; grasping direction detection to identify the grasping direction of the articulated object; and motion path detection to convert the operation command into a specific three-dimensional motion waypoint based on the object category, bounding box, rotation axis, grasping direction, and contact point.

[0017] Furthermore, the visual language action model includes a hybrid visual encoder, a projection layer, and an LLaMA2 model; the hybrid visual encoder integrates CLIP-VIT, CLIP-ConvNeXt, DINOv2-VIT, and Q-Former visual encoders, while simultaneously understanding the image features of the articulated object; the projection layer maps the visual tags of the visual encoder onto a visual language alignment tag; the LLaMA2 model is used to identify the task label corresponding to the operation instruction, enabling the visual language action model to output results based on the task label.

[0018] Furthermore, the joint optimizer includes an advisor module and a strategist module; the advisor module issues operation instructions to the articulated object according to the manipulation sequence planning of object category recognition and detection, bounding box detection, pivot detection, contact point detection, grasping direction detection, and motion path detection; the strategist module selects the best result from multiple output results of the visual language action model.

[0019] Furthermore, the operation instructions issued by the advisor module will invoke the optimal visual language action model obtained in step S3, identify task labels, and obtain multiple possible output results.

[0020] Furthermore, the optimal result selected by the strategist module in the previous manipulation sequence will serve as a constraint on the operation instructions issued by the advisor module in the next manipulation sequence.

[0021] Furthermore, the optimal result is determined as follows: for the object category of the articulated object, the optimal result is determined based on the similarity between the image of the articulated object and the candidate object category; for the bounding box of the articulated object, a unique bounding box is determined based on the determined object category; for the axis of rotation of the articulated object, the optimal result is determined based on the distance between the candidate axis of rotation of the articulated object and the center point of the bounding box; for the contact point of the articulated object, the optimal result is determined based on the proximity between the candidate contact point of the articulated object and the contact surface; for the gripping direction of the articulated object, the optimal result is determined based on the cosine similarity between the candidate gripping direction of the articulated object and the axis of rotation; the motion path of the articulated object is uniquely determined by the optimal results of the object category, bounding box, axis of rotation, contact point, and gripping direction.

[0022] Furthermore, it also has a feedback update strategy. According to the four evaluation methods of strategists, the appropriate results output by the visual language action model are used to update the original label set L in the training task to obtain a new label set L'. If L' is different from L, the new label set L' is used to train the projection layer of the visual language action model.

[0023] The advantage of this invention lies in its use of a six-instance representation method to label hinged objects, achieving a refined depiction of multiple operable parts of the hinged object. Based on this refined depiction, it is possible to identify multiple different hinged components on an object and generate operation methods.

[0024] The joint optimizer constructed in this invention can decompose the articulated object manipulation sequence planning task. By automatically decomposing the task, the VLA model gradually acquires the feature data of the articulated object, improving the accuracy of feature data acquisition and making the entire inference process more logically relevant. This invention also provides adaptive update capabilities, covering the entire process of automatic dataset construction and articulated operation integration, resulting in a more powerful generalization ability for the trained visual language action model. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating an existing method for generating and manipulating articulated objects.

[0026] Figure 2 A schematic diagram illustrating the articulated object grasping method of the robot based on a visual language action model, as presented in this invention.

[0027] Figure 3 Label the hinged objects for six instance representations. Figure 3 (a) A notation diagram of six instances of storage furniture drawers; Figure 3 (b) is a notation diagram of six instances of a trash can; Figure 3 (c) A notation diagram of six instances of the table; Figure 3 (d) is a notation diagram of six instances of the bottle.

[0028] Figure 4 This is a diagram of the VLA model architecture constructed for this invention. Figure 4 (a) is a schematic diagram of the input and output of the six training tasks; Figure 4 (b) The VLA model architecture diagram constructed in this invention, and the input and output for the object category recognition and detection task.

[0029] Figure 5 The diagram shows the architecture of the joint optimizer and VLA model constructed for this invention.

[0030] Figure 6 This is a comparative experimental result diagram of Example 3 of the present invention.

[0031] Figure 7 This is a diagram of the experimental process in Embodiment 4 of the present invention.

[0032] Figure 8 The figure shows the test results of Example 4 of the present invention. Detailed Implementation

[0033] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0034] The technical terms used in this application are explained below:

[0035] Articulated objects: Articulated objects are composed of multiple rigid components connected by joints, and each rigid component can rotate or translate around the joints. This type is commonly used in the connection of furniture, doors, windows, and other devices. Compared to ordinary rigid objects with only 6 degrees of freedom (DoF), articulated objects have a higher degree of freedom and are more difficult for robots to understand and interact with.

[0036] MLLM stands for Multimodal Large Model. MLLM is a large language model capable of processing and understanding multimodal data. Compared to traditional language models (such as GPT or BERT) that can only process text, MLLM can combine multiple modalities (such as text, images, video, audio, and even 3D data) for unified processing and inference.

[0037] VLA (Vision-Language-Action) model is a multimodal artificial intelligence model that integrates visual perception, language understanding, and action generation capabilities. It can handle images, text commands, and operational tasks in complex scenes, forming a closed loop from perception to decision-making to execution. For example, a VLA model can understand natural language commands provided by the user (such as "grab the red cup on the table"), analyze visual information (such as the position and features of objects in the scene), and generate corresponding operational plans (such as the grasping path of a robotic arm). This model is widely used in fields such as robot control and industrial automation, achieving natural interaction and efficient task execution through a unified multimodal processing framework.

[0038] Example 1

[0039] The articulated object grasping method of the robot based on visual language action model described in this invention specifically includes the following steps:

[0040] Step 1: Represent the hinged object using the multi-instance, multi-label geometric center representation method.

[0041] Existing technologies use a single-instance mechanism to describe articulated objects. In practice, it has been found that the single-instance mechanism often encounters limitations when dealing with complex articulated objects. In particular, when the articulated object has a complex structure containing multiple operable parts, the single-instance, single-label representation cannot fully capture all operable parts of the articulated object.

[0042] like Figure 3 As shown, the hinged object "table" contains two instances: "drawer" and "door". In traditional representations, the entire "table" is treated as a single instance and assigned a single label (such as "table"). This method of representing hinged objects ignores the fact that the "drawer" and "door" within the table can also be hinged components that interact with each other.

[0043] The articulated object robot grasping method based on a visual language action model described in this invention uses a multi-instance, multi-label geometric center representation to represent articulated objects. The articulated object is rendered in a metaverse environment, using its geometric center point as the core reference point, and is labeled with a six-instance representation. This six-instance representation uses object category, bounding box, axis of rotation, grasping direction, contact point, and movement path to represent the articulated object.

[0044] Object categories primarily include the name of the hinged object and its joint type, such as "drawer, sliding type" or "door, rotating type." The bounding box is defined by eight vertices, denoted as... The grabbing direction is determined by a vector. By definition, the contact point consists of a single point. Definition; the movement path consists of a set Definition; the axis of rotation is defined by two points, denoted as .

[0045] Step 2: Generate six training tasks based on the six instance representations.

[0046] The six instances are transformed into specific training tasks to obtain six instance data of a specific articulated object.

[0047] There are six training tasks as follows:

[0048] 1. Object Category Recognition. Object category recognition is used to guide VLA models to identify specific object categories and joint types, which is an important task in the field of machine vision. In this invention, the ViT-L / 14 visual language embedding model is used to map images and corresponding object category labels to a shared embedding space. Specifically, the ViT-L / 14 model employs two different encoders (an image encoder and a text encoder) to process the input image... and candidate text Converted to a high-dimensional vector representation, the expression is shown in formula (1):

[0049] (1)

[0050] in, and These are the image and text encoders for the ViT-L / 14 model. and These are image and text feature vectors that share an embedding space.

[0051] Then adopt and Cosine similarity between them serves as a label for object category recognition tasks. The basis is shown in formula (2):

[0052] (2)

[0053] 2. Bounding box detection. The purpose of bounding box detection is to guide the VLA model to identify and confirm the spatial position of the hinged object in the metaspace. In the camera coordinate system, Represented by 8 vertices, denoted as . , Figure 4 (a) shows the 8 vertices enclosed by the yellow lines.

[0054] In order to calculate First, determine the geometric center point of the hinged object in the camera coordinate system. and the length, width, and height dimensions of half of the smallest enclosing rectangle surrounding the hinged object. Then use the center point and Perform addition and subtraction operations on the values ​​to calculate the result. 8 points The expression is shown in formula (3):

[0055] (3)

[0056] In addition, some articulated objects contain both sliding and rotating joint axes. For example, Figure 3 In (d), the "mouth" of the "bottle" can be both pressed and rotated. For this type of special articulated object, in order to distinguish the different movement capabilities of the same part, it is necessary to manually assign different... Assign labels and associate them with object category recognition task labels. The two joint axes are linked together.

[0057] 3. Joint axis detection. Joint axis detection aims to guide the VLA model in identifying joint axes. This invention specifies a start point and an end point. To determine the position of the pivot.

[0058] For sliding type shafts, the shaft The starting and ending points are the same as the center point in formula (3). and The value is related to this. For rotary shafts, the shaft... The starting point, the ending point, and the center point and The value of is related, and the expression is shown in formula (4):

[0059] (4)

[0060] Prismatic represents a sliding hinge, and Revolute represents a rotating hinge.

[0061] 4. Contact Point Detection. Contact point detection aims to guide the VLA model in identifying the contact points of articulated objects, which is crucial for stable manipulation and efficient control. For example... Figure 4 As shown in (a), in three dimensions Six feasible aspects There are six feasible contact points. At a specific contact surface There is only one point. As a contact point.

[0062] For sliding hinged objects, first use formula (3) calculate The six sides As shown in formula (5):

[0063] (5)

[0064] As can be seen from formula (5), each face Each face is associated with four vertices. The corresponding geometric center The calculation method is shown in formula (6):

[0065] (6)

[0066] Next, define each face. direction of movement and joint axis vector . Depend on With the three-dimensional center point The difference is calculated as shown in formula (7). Rotation axis vector From the starting point in formula (4) and the end point The definition is as shown in formula (8):

[0067] (7)

[0068] (8)

[0069] Finally, calculate the contact surface. and contact points ,because It is the outermost surface subjected to force, so direction of movement Should with Most consistent. Therefore, calculate. and The dot product value is calculated, and the surface with the largest dot product value is selected as the contact surface, as shown below:

[0070] (9)

[0071] (10)

[0072] For rotating objects, this invention introduces sampling weights. ,in It is based on the sampling points With the shaft The distance is determined, that is, through For formula (3) Weighted sampling is used to determine :

[0073] (11)

[0074] 5. Grasping Direction Detection. Grasping direction detection aims to guide the VLA model in recognizing the direction of the gripper. This improves the stability of the operation.

[0075] For sliding hinged objects, the gripping direction needs to be maintained. Parallel to the direction of the joint axis.

[0076] For rotary hinged objects The calculations are more complex. This is because, unlike sliding articulated objects, the grasping direction of rotary articulated objects... Should be with Perpendicular. For rotationally hinged objects, first utilize the contact force in formula (11). And the axis of rotation in formula (4) Constructed through cross product To represent the vertical relationship, the expression is shown in formula (12):

[0077] (12)

[0078] 6. Motion Planning. Motion planning aims to guide the VLA model in translating user commands and geometric features into specific 3D waypoints. When constructing the motion plan, the VLA model can generate a set of waypoints. Each waypoint contains the location of the robot's end effector.

[0079] For sliding joints, based on the contact point and rotation axis vector Waypoints can be calculated. Specifically, as shown in formula (13):

[0080] (13)

[0081] in, and These represent the speed of motion and the time step, respectively.

[0082] For rotary joints, rotation axis vectors are also required. and rotation angle Calculate the rotation matrix according to Rodriguez's rotation formula. ,Record Finally, at the contact point When rotating using a rotation matrix, the corresponding waypoints are generated. It can be calculated as follows:

[0083] (14)

[0084] Step 3: Design and build the VLA model architecture.

[0085] In this invention, the VLA model is developed and constructed based on SPHINX-X and LLaMA2. For example... Figure 4As shown in (b), unlike the Mixture of Visual Experts (MOV) used in SPHINX-X (a series of multimodal large language models (MLLMs) developed based on SPHINX, designed to improve model performance by expanding data and parameter scale), this invention employs Mixed Four Vision Encoders. The Mixed Four Vision Encoders integrate CLIP-VIT (a multimodal model combining the CLIP model with the Vision Transformer (ViT) architecture, exhibiting powerful performance in image classification, image and text retrieval, etc.), CLIP-ConvNeXt (a multimodal model combining the CLIP model with the ConvNeXt architecture, widely used in image classification, image and text retrieval, etc.), and DINOv2-VIT (a self-supervised visual model based on the Vision Transformer (ViT) architecture launched by Meta AI) to extract local semantic features, while utilizing Q-Former (a neural network architecture in multimodal large models) to summarize global visual features. To ensure the robustness of the four hybrid visual encoders, the final VLA model (Vision-Language-Action Model) should be able to understand both the global and local features of the input image.

[0086] This invention separates a visual image into four sub-images and one low-resolution image. All five images are simultaneously input into a hybrid visual encoder. Then, a projection layer is designed to map the five visual tags onto a single visual language alignment tag within the projection network, thus completing the process. Figure 4 (b) Spatial alignment of visual and textual features is achieved, which not only enhances the semantic understanding ability of the VLA model, but also improves the accuracy and efficiency of visual and textual feature fusion.

[0087] The training tasks are then input into the LLaMA2 model (an open-source large language model launched by Meta), and supervised training is performed on each training task using the total label set and sample database of the training tasks constructed in step two to obtain the optimal visual language action model.

[0088] Step 4: Construct a joint optimizer to select the best result from the output of the visual language action model by decomposing the articulated object manipulation sequence planning task.

[0089] To ensure that the VLA model can generate accurate 3D waypoints for articulated object manipulation, this invention proposes a hierarchical kinematics awareness facilitation method called Joint Optimizer (JO). Because some inference results are interdependent during the hierarchical cues process, such as... Figure 5 As shown, to capture the bounding box of an articulated object, the correct results need to be obtained from the Object Class Recognition (OCI) task labels. To better guide the VLA model in providing accurate motion waypoints, JO will decompose the manipulation sequence planning and evaluate the candidate responses of the VLA model according to the manipulation sequence planning.

[0090] like Figure 5 As shown, the core of JO consists of two parts: Advisor and Strategist. In JO, the Advisor follows a six-instance representation instruction template, issuing manipulation instructions to the articulated object according to the manipulation sequence planning of object category recognition and detection, bounding box detection, axis detection, contact point detection, grasping direction detection, and motion path detection. The manipulation instructions issued by the Advisor module will call the optimal visual language action model obtained in step S3, identify task labels, and obtain multiple possible output results.

[0091] The strategist module selects the best result from multiple possible outputs of the Visual Language Action Model (VLA).

[0092] like Figure 5 As shown, to enhance the adaptability of the VLA model, this invention also adds an adaptive feedback strategy. For example, for the "bottle" object, this invention uses a basic template, such as "How do I use this bottle?", in... Figure 5 In the OCI results, it's easy to observe two candidate results: "mouth. revolute." and "mouth.prismatic." However, "mouth.prismatic." is the appropriate answer because it is the only articulated target for pressing the bottle.

[0093] Suppose the Advisor has candidate outcomes. The Strategist is designed to score all possible outcomes and select the best outcome according to four hierarchical kinematic perception facilitation processes. This is useful when multiple candidate outcomes exist, such as in a task with multiple object category labels. Multiple rotating shafts Multiple contact points and multiple grab directions When Strategyist identifies task labels from multiple object categories If the best candidate is selected, then the results for the subsequent bounding box, pivot, contact point, gripping direction, and waypoint will be unique, such as... Figure 5 The blue text with arrows connecting the mouth, prismatic, and BBOX is mentioned. , G0. In other words, the best result selected by the strategist module in the previous manipulation sequence will serve as a constraint on the operation instructions issued by the advisor module in the next manipulation sequence.

[0094] exist Figure 5 In this context, the Strategyist approach for object category recognition (OCI) aims to analyze candidate OCI results. The best result is selected from the similarities between the image and each candidate OCI result. The most similar OCI result is selected according to formula (1) as follows:

[0095] (15)

[0096] The Strategist tool for hinge detection aims to evaluate multiple hinges. and center point The degree of proximity between them. For multiple shafts. Calculate their rotation axis vectors according to Formula 8. And select the optimal joint shaft detection result according to formula (16). :

[0097] (16)

[0098] in express One point in the algorithm. The algorithm evaluates multiple joint axes. and center point The distance between them.

[0099] The Strategyist tool for contact point detection aims to evaluate multiple contact points. Contact surface in formula (9) The degree of proximity between them, and the selection of the optimal contact point detection result. and Contact surface The distance is determined, and the optimal contact point detection result is selected according to formula (17). :

[0100] (17)

[0101] in, Representative contact surface The direction of movement, represent A point on the surface. The algorithm uses distance to evaluate multiple contact points. and contact surface The degree of closeness between them.

[0102] The Strategyist algorithm for gripping orientation detection aims to evaluate multiple gripping orientations. and pivot The alignment between them is determined, and the optimal grabbing direction is selected according to formula (18). :

[0103] (18)

[0104] The algorithm calculates the cosine similarity between each gripping direction result and the axis of rotation. For a specific axis of rotation type, formula (18) selects the appropriate gripping direction. For sliding objects, since the gripping direction is parallel to the joint axis, the gripping direction with the highest similarity score is selected. For rotating objects, since the grasping direction is perpendicular to the joint axis, the grasping direction with the lowest similarity score is selected. .

[0105] Finally, to enhance the adaptability of the VLA model in handling responses to different scenarios, this invention also protects a feedback update strategy. When asking the VLA model "How do I use this bottle?", the appropriate answer from OCI should be "mouth.prismatic." rather than "mouth.revolute." Therefore, the original label set of OCI in the training task needs to be updated to ensure that the VLA model makes an appropriate answer.

[0106] Specifically, the feedback update strategy of this invention utilizes the score of the appropriate answer. Update the original label set in the training task A new tag set is obtained. :

[0107] (19)

[0108] in, These refer to four scoring formulas for strategists. This represents the updated tag set. If found... and Unlike others, we will adjust according to the new tag set. The projection layer in the VLA model was trained to enhance the VLA model's adaptability in detecting instances with different labels.

[0109] Step 5: Use the optimal visual-language-action model and joint optimizer obtained in step S3 to determine the articulated object grasping method for the robot. Specifically, based on the object image and user input, the advisor module of the joint optimizer plans and issues manipulation commands to the articulated object according to the manipulation sequence planning, which includes object category recognition and detection, bounding box detection, axis detection, contact point detection, grasping direction detection, and motion path detection. The optimal visual-language-action model identifies the task label and obtains multiple possible output results. Then, the strategist module of the joint optimizer selects the best result from the visual-language-action model output to obtain the articulated object grasping method for the robot.

[0110] Example 2

[0111] This invention utilizes the training task generated in step two to train the VLA model designed in step three. The training task adopts the default Visual Question Answering (VQA) dataset format of the SPHINX framework. To evaluate the category-independent capability of this invention, 10 object categories are selected from the generated training task as "seen categories," and another 20 object categories are selected for training the VLA model, referred to as "unseen categories." Furthermore, this invention uses the SPHINX-v2-1k model obtained from the official resource repository as the pre-training foundation. Figure 4 As shown in (b), the hybrid visual encoder remains frozen throughout the training process, with only the projection layer and LLaMA2-13B being fine-tuned to maintain the integrity of the pre-trained features.

[0112] Example 3

[0113] To evaluate the performance of this invention, the evaluation method described in ManipLLM was used in the comparative experiment, and the Franka Panda robot gripper and Sapien simulator were used for evaluation in a metaverse environment.

[0114] 1. Comparative Analysis in the Metaverse. Operational success rate is a metric for evaluating the performance of this invention across 20 "seen categories" and 10 "unseen categories." This invention was compared with six baseline methods: Where2Act, UMPNet, FlowBot3D, Implicit3D, ManipLLM, and A3VLM.

[0115] Where2Act uses a neural network to map pixels to six basic actions, processing point cloud data by predicting the operable information for each pixel, thereby manipulating articulated objects.

[0116] Similar to Where2Act, UMPNet ensures that the direction of the end effector is perpendicular to the object surface when predicting the contact force point.

[0117] Flowbot3D predicts potential movement directions by representing 3D articulated flows and deploys an analytical motion planner to determine articulation strategies.

[0118] Implicit3D uses a transmitter network to identify contact force points on articulated objects. By leveraging hybrid 3D representations, cross-attention, and implicit reconstruction mechanisms, it helps guide the posture of the end effector when performing manipulation tasks.

[0119] ManipLLM uses a fine-tuned visual language model, combining object category understanding, force reasoning, and object-centric pose prediction to simulate manipulation.

[0120] A3VLM focuses on the articulation structure and motion tolerance of articulated objects, using a visual language model to detect the bounding box, axis, and joint type of the object, thereby enabling the manipulation process.

[0121] from Figure 6 The data shows that, out of 30 object categories, this invention outperforms other baseline methods in 18 categories. For example, in the "gate" category, the success rate of this invention is 98%, 17% higher than the second-ranked A3VLM's 82%. Furthermore, in the "success rate of seen categories" category, the AVG (average success rate) of this invention is 83%, 10% higher than the second-ranked A3VLM. In the "success rate of unseen categories" category, the AVG (average success rate) of this invention is 76%, 1.3% higher than the second-ranked A3VLM, indicating that this invention has a strong zero-error capability.

[0122] Example 4

[0123] To verify the operational performance of the invention in real-world applications, the invention generates manipulation of articulated objects in a metaverse environment and conducts reasoning experiments in a real-world environment.

[0124] This invention first initializes the Teledyne DALSA TM461-E2 depth camera, calibrating its intrinsic and extrinsic parameters to capture images of the articulated object. Then, to eliminate the influence of background differences between the camera and the reference image, Segment Anything is used to extract image semantics and replace the image background. Next, a VLA model is used to determine the waypoints of the manipulator that opens the captured articulated object. The experiment utilizes end effector inverse kinematics based on cubic spline trajectory interpolation to calculate the manipulator's waypoints. Specifically, a CyberTech Pincer 2F120S manipulator and a Faurecia FR5 6-DOF manipulator were used.

[0125] like Figure 8 As shown, 10 real-world experiments were conducted, involving five different types of hinged objects: microwave ovens, kettles, pots, lids, and drawers. The success rate results are as follows: Figure 8 As shown, the success rate of this invention is 90% in the kettle and pot category and 80% in the microwave oven, lid, and drawer category. Analysis of the experimental results demonstrates that the manipulation knowledge of this invention in the real-world environment can be transferred from the metaverse environment and applied to new categories of robotic arms, proving that the method maintains category independence when transferring from the metaverse environment to the real-world environment.

Claims

1. A method for grasping articulated objects by a robot based on a visual language action model, characterized in that, Includes the following steps: S1, place the hinged object in the metaverse environment for rendering, use the geometric center point of the hinged object as the core reference point, use six instance representations to mark the hinged object, and establish a sample database. S2 generates six training tasks based on six instance representations; S3. Construct a visual language action model based on LLaMA2 and SPHINX-X. Perform supervised training on the visual language action model according to the training task and sample database to obtain the optimal visual language action model. S4. Construct a joint optimizer to select the best result from the output of the visual language action model by decomposing the articulated object manipulation sequence planning task. The joint optimizer includes an advisor module and a strategist module; the advisor module issues operation instructions to the articulated object according to the manipulation sequence planning of object category recognition and detection, bounding box detection, pivot detection, contact point detection, grasping direction detection, and motion path detection; the strategist module selects the best result from multiple output results of the visual language action model. The operation command issued by the advisor module will call the optimal visual language action model obtained in step S3, identify task labels, and obtain multiple possible output results. The best result selected by the strategist module in the previous operation sequence will serve as a constraint on the operation instructions issued by the advisor module in the next operation sequence. S5 uses the optimal visual language action model and joint optimizer obtained in step S3 to determine the robot's grasping method for articulated objects.

2. The articulated object grasping method of a robot based on a visual language action model according to claim 1, characterized in that: The six instance representations use object category, bounding box, pivot, gripping direction, contact point, and motion path to represent articulated objects; the object category is represented by the name of the articulated object and the joint type; the bounding box is represented by eight vertices; the pivot is represented by two points; the gripping direction is represented by a vector; and the motion path is represented by a set of multiple points.

3. The articulated object grasping method of a robot based on a visual language action model according to claim 1, characterized in that: The six training tasks are used to acquire information about six instances of articulated objects, including: object category recognition and detection to identify the category and joint type of the articulated object; bounding box detection to obtain the spatial position of the articulated object in the metaspace; rotation axis detection to obtain the rotation axis position of the articulated object; contact point detection to identify the contact point of the articulated object; grasping direction detection to identify the grasping direction of the articulated object; and motion path detection to convert the operation command into a specific three-dimensional motion waypoint based on the object category, bounding box, rotation axis, grasping direction, and contact point.

4. The method for grasping articulated objects by a robot based on a visual language action model according to claim 1, characterized in that: The visual language action model includes a hybrid visual encoder, a projection layer, and an LLaMA2 model. The hybrid visual encoder integrates CLIP-VIT, CLIP-ConvNeXt, DINOv2-VIT, and Q-Former visual encoders, and simultaneously understands the image features of the articulated object. The projection layer maps the visual tags of the visual encoder to a visual language alignment tag. The LLaMA2 model is used to identify the task label corresponding to the operation instruction, so that the visual language action model outputs the result according to the task label.

5. The articulated object grasping method of a robot based on a visual language action model according to claim 1, characterized in that: The optimal result is determined as follows: for the object category of the articulated object, the optimal result is determined based on the similarity between the image of the articulated object and the candidate object category; for the bounding box of the articulated object, a unique bounding box is determined based on the determined object category; for the axis of rotation of the articulated object, the optimal result is determined based on the distance between the candidate axis of rotation of the articulated object and the center point of the bounding box; for the contact point of the articulated object, the optimal result is determined based on the proximity between the candidate contact point of the articulated object and the contact surface; for the gripping direction of the articulated object, the optimal result is determined based on the cosine similarity between the candidate gripping direction of the articulated object and the axis of rotation; the motion path of the articulated object is uniquely determined by the optimal results of the object category, bounding box, axis of rotation, contact point, and gripping direction.

6. The articulated object grasping method of a robot based on a visual language action model according to claim 1, characterized in that: It also has a feedback update strategy. According to the four evaluation methods of strategists, the appropriate results output by the visual language action model are used to update the original label set L in the training task to obtain a new label set L'. If L' is different from L, the new label set L' is used to train the projection layer of the visual language action model.

Images