A method for constructing a VLA large model for articulated object manipulation

Abstract

The application discloses a construction method of a VLA large model for articulated object manipulation, and belongs to the technical field of embodied intelligence. The method is as follows: an action affordance of an articulated object is described by using a geometric center representation method; an instruction following a feature set of the articulated object is constructed; one instruction is decomposed into six different sub-task feature sets; a VLA model architecture is designed; and a geometric thinking planner is constructed. The application describes the articulated object by using the geometric center representation method, develops the geometric thinking planner, promotes the understanding of the environment by the VLA large model, enhances the adaptability of the VLA large model to complex tasks, enables the VLA large model to process diversified scenes and requirements, and enables the VLA large model to fully understand the behaviors and reactions of objects in different situations, so that the VLA large model can make more reasonable decisions when planning a motion path, and the precision and reliability of path planning are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to a method for constructing a VLA large model for articulated object manipulation, belonging to the field of embodied intelligence. BACKGROUND

[0002] An articulated object is an object composed of multiple rigid components connected by joints, each rigid component can rotate or translate around the joint. This form is often used in the connection of furniture, doors and windows, etc. Compared with ordinary rigid objects with only 6 degrees of freedom (DoF), articulated objects have higher degrees of freedom and are more difficult for robots to understand and interact with.

[0003] A multimodal large language model (MLLM) is a large language model capable of processing and understanding multi-modal data. Compared with traditional language models (such as GPT or BERT) that can only process text, MLLM can combine multiple modalities (such as text, images, videos, audio, and even 3D data) for unified processing and reasoning.

[0004] VLA large model, full name visual-language-action large model, is a multi-modal artificial intelligence model that integrates visual perception, language understanding and action generation capabilities. It can process images, text instructions and operation tasks in complex scenarios, forming a closed loop from perception to decision-making to execution. For example, the VLA model can understand natural language instructions provided by the user (such as "grab the red cup on the table"), analyze visual information (such as the position and characteristics of objects in the scene), and generate corresponding operation plans (such as the grabbing path of the robotic arm). This model is widely used in robot control, industrial automation and other fields, and through a unified multi-modal processing framework, it realizes the ability of natural interaction and efficient task execution.

[0005] The Action Affordance of an articulated object refers to the operation possibilities and interaction methods provided by the articulated object through hinges, sliding rails and other motion constraints. Such objects, due to the presence of movable components, not only have the functional characteristics of static objects, but also contain dynamic characteristics that can be operated. For example, the hinges of a door imply that it can be opened by pushing or pulling, and the sliding rails of a drawer indicate that it can be pulled out or pushed in. In robot control and task planning, studying the Action Affordance of articulated objects helps the model understand their motion laws, constraint conditions and applicable actions, thereby generating reasonable manipulation strategies, such as opening doors, closing drawers and other complex interaction behaviors.

[0006] Flexible manipulation of articulated objects is of great significance for the development of intelligent multipurpose robots. These objects differ significantly in their movement characteristics, so specialized manipulation techniques need to be designed for different types of objects. In order to achieve stable task execution in a changing real-world environment, researchers have proposed various methods for general object manipulation, mainly including imitation learning through demonstration and reinforcement learning in simulation environments. Imitation learning usually relies on human demonstration data to enable robots to learn manipulation skills through imitation, which is particularly effective when dealing with complex or difficult-to-model tasks. However, imitation learning often requires a large amount of diverse data so that the robot can adapt well to new environments or different objects. On the other hand, reinforcement learning enables robots to explore and optimize strategies in simulation environments through extensive trial and error to improve manipulation capabilities. For example, Figure 1 (a) shows the data requirements in this process, these techniques rely on large robot interaction datasets to help robots effectively learn and adapt to the needs of different tasks, and collecting these datasets in real life is often time-consuming and laborious.

[0007] Recently, there has been widespread interest in using large language models (LLMs) for robotic applications, which have expanded beyond natural language processing in recent years and into the field of robot control, particularly for robotic arm control. LLMs have strong reasoning capabilities and can use prompt templates to decompose user instructions, helping robots more efficiently understand and execute complex tasks. In articulated object manipulation, large language models can be used as high-level task planning and instruction parsing tools. There are methods that use LLMs to understand natural language instructions, combined with fixed manipulation API and scene information to generate robotic arm operation plans. LLMs mainly focus on processing text data, but they are not usually good at processing visual information, which limits their performance in tasks that require the combination of visual input (such as images or videos) to make decisions. For example, Figure 1 (b) shows that in current methods, LLMs need to plan robotic arm movements based on URDF (Unified Robot Description Format) information, although this algorithm is effective in the IsaacGym virtual environment, it lacks generalization ability and is difficult to adapt to real-world environments. This is because real-world scenarios do not always provide fixed URDF information.

[0008] In contrast to LLMs, MLLMs can handle data from multiple modalities, including images, text, and videos. This ability makes them perform better in complex environments, especially in tasks that require the integration of visual information and language instructions. For example, Figure 1(c) shows that current methods use MLLM for robot applications, using MLLM for object force-bearing recognition and motion planning, and can automatically execute user instructions to complete manipulation tasks in the real world. However, in the real world, it is very expensive to collect such a large amount of robot interaction data. In addition, these methods focus on obtaining contact point information of objects, and do not have complete mechanical arm operation planning. If you want to deploy a robot to control the mechanical arm to complete a task in reality, you need to manually implement the motion planning algorithm.

[0009] In addition, some methods train VLA models based on the Open-X dataset. Compared with MLLM models, VLA large models can directly plan physical operations according to visual and language understanding. As shown in Figure 1 (d) shows that the model maps the joint torque of the mechanical arm to a continuous interval, obtaining N discrete integers ∈ [0...255] for expressing N-degree-of-freedom robot actions, and exhibits strong generalization ability in real-world environments. VLA models can flexibly cope with diverse operation tasks and perform particularly well in complex and changeable work scenarios. However, such methods often rely on large amounts of image-action data collection, which not only increases the difficulty of data preparation but also significantly increases the training cost. For example, the OpenVLA study used a cluster of 64 A100 GPUs for 14 days of training, consuming a total of 21,500 A100 GPU hours. In addition, many large VLA models are Category-Related. The use of these models is limited to the dataset they were trained on, meaning that the same type of VLA model can only be used for the same type of mechanical body, limiting their generalization ability.

[0010] To solve the foregoing problems, the present application designs a new method for training VLA models for operating articulated objects, as shown in Figure 1(e) as shown. The present application includes an end-to-end VLA large model that can solve the articulated object manipulation problem using multi-modal data. It is based on the geometric center and designed specifically for manipulating articulated objects, and can accurately plan the motion path of the robot end-effector in the current scene. Compared with traditional methods, one of the significant advantages of the present application is that it realizes the integration of the whole process from perception to motion control. It can generate end-effector motion paths and easily control the robot in real-world environments through Spline instructions to successfully complete various operation tasks. In addition, the present application can understand the Action Affordance of articulated objects without relying on expensive robot interaction data, which means it can identify and execute possible actions in the environment. More importantly, the method is Category-Agnostic, which means that different robots can use the same learning model to complete manipulation tasks, thereby improving the versatility and applicability of the VLA model. SUMMARY

[0011] To solve the problems in the background art, the present application provides a construction method of a VLA large model for articulated object manipulation.

[0012] To achieve the above-mentioned object, the present application adopts the following technical solution: a construction method of a VLA large model for articulated object manipulation, the method comprising the following steps:

[0013] S1: using the geometric center representation method to describe the Action Affordance of the articulated object:

[0014] A={B,D,F,M,X} (1)

[0015] In formula (1):

[0016] B represents the three-dimensional bounding box of the articulated object;

[0017] D represents the direction of the end-effector of the mechanical gripper;

[0018] F represents the optimal force point;

[0019] M represents a sequence of movable path planning points;

[0020] X represents the pivot position;

[0021] S2: construct instructions following the articulated object feature set, the feature set is consistent with the format of the MLLM fine-tuning data set, the input is image and text instructions, and the output is formatted text, i.e. the Action Affordance of the articulated object; decompose one instruction into six different sub-task feature sets;

[0022] S201: Object category recognition: guide the VLA model to output the name of the specific object to be operated according to the given picture and user instructions;

[0023] S202: Border detection: aims to guide the VLA model to accurately identify and label the specific position of the target object in three-dimensional space;

[0024] S203: Axis detection: aims to guide the VLA model to accurately identify the rotation axis type and movement direction of the target object;

[0025] S204: Force point detection: guide the VLA model to accurately identify the best force point of the target object in order to perform efficient mechanical operation and control;

[0026] S204 said best force point P optimal The calculation process of the best force point of the target of the sliding joint type is as follows:

[0027] S20401: Define the six faces of the object three-dimensional bounding box:

[0028] faces{F1,F2,...,F6},F i ∈R 4×3 (2)

[0029] In formula (2):

[0030] R represents a real number, R 4×3 represents a two-dimensional real number vector with a size of 4x3;

[0031] S20402: Calculate the geometric center point c bbox of the three-dimensional bounding box and the geometric center c i of each face:

[0032]

[0033] In formulas (3) and (4):

[0034] p i represents a three-dimensional vertex in the three-dimensional bounding box, and each three-dimensional bounding box is determined by 8 three-dimensional vertices p i

[0035] p ij represents a three-dimensional vertex in the face F i , and each face F i is determined by 4 three-dimensional vertices p ij ;

[0036] S20403: Through the geometric center point c bbox of the three-dimensional bounding box and the geometric center point c​i The difference between the two calculates the moving direction vector n of each face i :

[0037]

[0038] S20404: Calculate the unit vector d in the direction of motion based on any two points d1 and d2 on the axis of rotation unit :

[0039]

[0040] S20405: Calculate the moving direction vector n of each face i The unit vector d with the direction of motion unit dot_product i :

[0041] dot_product i =n i ·d unit (7)

[0042] S20406: dot product i The closer it is to 1, the closer it is to the unit vector d in the direction of motion. unit The more consistent, therefore, the face F with the largest dot product k The exposed surface:

[0043]

[0044] S20407: Calculate the face F with the largest dot product k The geometric center, that is, the optimal force point P optimal :

[0045]

[0046] In formula (9):

[0047] p kj Represents the face F with the largest dot product k The three-dimensional vertices in each face with the largest dot product F k There are 4 three-dimensional vertices p kj Sure.

[0048] The optimal force point of the target of the rotation joint type in the optimal force point in S204 is located far away from the rotation axis due to its rotational freedom. The calculation process is as follows:

[0049] S20401: assign different weights to different points on the target object according to their distance from the rotation axis;

[0050] S20402: Calculate the optimal force point P optimal :

[0051]

[0052] In formula (10):

[0053] w i is the weight of the i-th point;

[0054] S205: Clamping jaw direction detection: the guide VLA model can accurately identify the direction of the clamping jaw, thereby improving the precision and efficiency of the robot operation;

[0055] For a target of sliding joint type, since the clamping jaw operates along the axial thrust of the target, the direction of the clamping jaw is the opposite direction of the rotation axis direction;

[0056] For a target of rotation joint type, the direction of the clamping jaw is determined by the cross product of two vectors on the force surface of the target object;

[0057] S206: Movement path planning: the guide VLA model converts abstract text instructions into specific mechanical movements.

[0058] S20601: According to the results of the frame detection, the rotation axis detection, the force point detection, and the clamping jaw direction detection, a movement planning sequence composed of multiple points is generated:

[0059] {(xi,yi,zi)}(11)

[0060] In formula (11):

[0061] i = 1, …, 4, representing the 1st to 4th points in the movement planning sequence, each point contains the x, y, z coordinates of the robot end in three-dimensional space, which defines the movement path of the robot end;

[0062] S20602: Calculate the unit vector v of the rotation axis direction according to any two points d1, d2 in the rotation axis direction :

[0063]

[0064] In formula (12):

[0065] d1 = (x1, y1, z1);

[0066] d2 = (x2, y2, z2);

[0067] S20603: Movement path planning;

[0068] For a target of sliding joint type, calculate the sliding path pt :

[0069] p t = P optimal + t · speed · v direction , t = 0, 1, 2, 3 (13)

[0070] In equation (13):

[0071] speed = 0.1 means the speed of movement;

[0072] For the target of the type of revolute joint,

[0073] S2060301: Calculate the rotation angle per second angle_per_second:

[0074] angle_per_second = speed x 5 (14)

[0075] In equation (14):

[0076] speed = 0.1 means the speed of movement;

[0077] S2060302: Calculate the corresponding rotation angle θ at each time point t:

[0078] θ = angle_per_second x t, t = 0, 1,..., 3 (15)

[0079] S2060303: Calculate the rotation matrix R(v direction , θ) by equation (15):

[0080]

[0081] In equation (16):

[0082] u x , u y , u z all represent the components of the unit vector v direction of the rotation axis direction;

[0083] S2060304: Rotate the best force point P optimal and record the new best force point P' position:

[0084] P' = (P optimal - d1) · R(v direction , θ) + d1 (17).

[0085] S3: Design VLA model architecture;

[0086] S301: The input high-resolution image is spatially divided into multiple sub-images, and the high-resolution image is down-sampled into a low-resolution image;

[0087] S302: After multiple images are simultaneously input into the hybrid visual encoder, they are encoded and concatenated into a long sequence, and then input into the LLaMA2-13B model; wherein the first group of encodings is responsible for global semantics, and the remaining four groups of encodings record fine-grained local features;

[0088] S303: Design a projection layer to achieve spatial alignment of visual and language features.

[0089] S4: Train the VLA model;

[0090] Train the VLA model in S3 using the feature set generated in S2;

[0091] S5: Build a geometric thinking planner.

[0092] S501: Build a geometric thinking planner including Advisor and Strategist;

[0093] S502: Advisor makes VLA model list all results, then Strategist scores each result according to the scoring algorithm formulated, and finally filters out the optimal result for subsequent reasoning tasks.

[0094] S502: The Strategist scores and filters the answers to the object category identification includes the following steps:

[0095] S50201: Compare the language-image pre-training model to convert the input image and text candidate into high-dimensional vector representation through two independent encoders:

[0096] I=f image (x image ),T=f text (x text ) (18)

[0097] In equation (18):

[0098] x image represents the input image;

[0099] x text represents the input text candidate;

[0100] f image represents the image encoder of the CLIP model;

[0101] f text represents the text encoder of the CLIP model;

[0102] I represents an image feature vector in a shared embedding space;

[0103] T represents a text feature vector in a shared embedding space;

[0104] S50202: Calculate the similarity between the image embedding and the text embedding, ensuring that the similarity value is in the range [-1, 1], and generate a score for class prediction:

[0105]

[0106] In equation (19):

[0107] I represents the dot product of the image embedding vector for object class recognition calculation;

[0108] T represents the dot product of the text embedding vector for object class recognition calculation;

[0109] ||I|| represents the Euclidean norm of the dot product of the image embedding vector for object class recognition calculation;

[0110] ||T|| represents the Euclidean norm of the dot product of the text embedding vector for object class recognition calculation;

[0111] S50203: Select the result with the highest similarity as the best object class recognition answer:

[0112] Best Prediction = argmax(Similarity(I, T)) (20).

[0113] The Strategist of S502 scores and filters the answers of the stress point position detection as follows:

[0114] S50201: Define a plane by any three points P1, P2, P3, and calculate the normal vector of the plane

[0115]

[0116] In equation (21):

[0117]

[0118] S50202: Determine the equation of the plane according to the components A, B, C of the normal vector :

[0119] Ax + By + Cz + D = 0 (22)

[0120] S50203: Calculate D according to any point P1(x0, y0, z0) on the plane:

[0121] D=-(A·x0+B·y0+C·z0) (23)

[0122] S50204: Calculate the distance from the target point P(x,y,x) to the surface:

[0123]

[0124] S50205: Calculate each point P in the given point set in turn i The distances to the six faces of the 3D bounding box;

[0125] S50206: Calculate the total score S of all points i :

[0126]

[0127] In formula (25):

[0128] d i,j Represents point P i To any face of the 3D bounding box F j The distance, F j is F1, F2, ..., F6;

[0129] S50207: Total score S for all points i Sort and select the point with the lowest score as the optimal point:

[0130]

[0131] The Strategist scoring and screening of the answers to the pivot test in S502 includes the following steps:

[0132] S50201: Given a target axis vector axis and a grasping direction vector g i , calculate the cosine similarity between the two:

[0133]

[0134] In formula (27):

[0135] In formula (27):

[0136] axis represents the three-dimensional vector of the target axis;

[0137] g i A three-dimensional vector representing the grasping direction;

[0138] ||axis|| represents the Euclidean norm of the three-dimensional vector of the target axis;

[0139] ||g i|| represents the Euclidean norm of the three-dimensional vector of the grasping direction;

[0140] S50202: Select the grasping direction according to the type of joint:

[0141] For the target of the sliding joint type, select the grasping direction most aligned with the target direction;

[0142] For the target of the rotating joint type, select the direction with the lowest alignment with the target direction.

[0143] Compared with the prior art, the present application has the beneficial effects that:

[0144] Firstly, the present application describes the articulated object using the geometric center representation method, which can capture the Action Affordance of the articulated object in a fine-grained manner. Secondly, after constructing the VLA large model, a geometric thinking planner is developed, which not only promotes the understanding of the environment by the VLA large model, but also enhances its adaptability to complex tasks, enabling it to handle diverse scenarios and demands. The core of this planner is an Advisor-Strategist evaluation mechanism, through which the VLA model can fully understand the behavior and reaction of objects in different situations, making more reasonable decisions when planning the motion path, improving the accuracy and reliability of path planning. BRIEF DESCRIPTION OF DRAWINGS

[0145] Figure 1 is a background technology schematic diagram;

[0146] Figure 2 is a schematic diagram of the present application;

[0147] Figure 3 is a schematic diagram of the geometric center representation method;

[0148] Figure 4 is a schematic diagram of the instruction following articulated object feature set;

[0149] Figure 5 is a schematic diagram of the VLA model architecture;

[0150] Figure 6 is a schematic diagram of the geometric thinking planner;

[0151] Figure 7 is a schematic diagram of the results of Example 1;

[0152] Figure 8 is a schematic diagram of the results of Example 2;

[0153] Figure 9 is a schematic diagram of the results of Example 3. DETAILED DESCRIPTION

[0154] The technical solutions in the present application will be described clearly and completely in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work belong to the protection scope of the present application.

[0155] A method for constructing a VLA large model for articulated object manipulation, the input is the URDF file of the articulated object, and the output is a VLA large model with generalization ability, which can infer the end moving path of the robot arm according to visual information and user instructions. The method comprises the following steps:

[0156] S1: Many methods represent the articulated object around the robot arm, which will lose a lot of physical information of the articulated object, such as the type of rotation axis and the direction of movement. As shown in Figure 3 The Action Affordance of the articulated object is described using the geometric center representation method:

[0157] A={B,D,F,M,X} (1)

[0158] In formula (1):

[0159] B represents the three-dimensional bounding box (BBOX) of the articulated object, which is defined by eight vertices (xi, yi, zi) i=0…7;

[0160] D represents the direction (Direction) of the end effector of the mechanical gripper, which is defined by two vertices (xi, yi, zi) i=0, 1;

[0161] F represents the best force point (Force Point), which is defined by one vertex (xi, yi, zi);

[0162] M represents the sequence of movable path planning points (Move Points), which is defined by n vertices (xi, yi, zi) i=0,…,n;

[0163] X represents the rotation axis position (Axis), which is defined by two vertices (xi, yi, zi) i=0, 1;

[0164] S2: In the present invention, it is necessary to generate a class-agnostic VLA large model. Given that training a VLA model from scratch would require a huge amount of data and computing resources, the present invention chooses a different path to develop the VLA model. The strategy adopted is to fine-tune an existing MLLM. To achieve this goal, it is necessary to construct instructions following the hinge object feature set, which is consistent with the format of the MLLM fine-tuning dataset, with the input being an image and a text instruction, and the output being a formatted text, i.e. the Action Affordance of the hinge object; the specific method of constructing such instruction tracking dataset will be discussed in this section. As discussed in Step 1, the Action Affordance of the hinge object is represented by A = {B, D, F, M, X}. In actual operation, it may not be satisfactory to submit all the information to the MLLM at once and let it give the correct answer, because the MLLM will ignore some important information. Therefore, as shown in Figure 4 , in order to make the VLA large model obtained by fine-tuning fully understand the Action Affordance information of the hinge object, the present invention divides a instruction into six different sub-task feature sets.

[0165] S201: Object Class Identification (OCI): As shown in Figure 4 (a), it aims to guide the VLA model to output the name of the specific object to be operated according to the given picture and user instruction; OCI is an important task in the field of machine vision, which involves enabling machines to recognize and classify different objects in images. In order to effectively manipulate objects, the VLA model must first identify the type of object it is facing, because objects of the same type usually have similar geometric features. This recognition ability is crucial for robot manipulation tasks, as it directly affects the grasping strategy and execution of actions. In the dataset of the present invention, a total of 62 common object types are collected, which cover multiple fields from daily necessities to office supplies.

[0166] S202: Position Detection: As shown in Figure 4 (b), it aims to guide the VLA model to accurately identify and label the specific position of the target object in three-dimensional space; for this purpose, a three-dimensional bounding box is used to label the target object to ensure the accuracy of spatial positioning. The three-dimensional bounding box (BBOX) is represented by a set of eight vertex coordinates {(xi, yi, zi)}, i = 1, …, 8, each coordinate corresponds to a corner of the three-dimensional bounding box. In this representation, (xi, yi) is the two-dimensional projection position of vertex i on the image plane, i.e. the pixel coordinates in the image. While zi represents the depth information to distinguish the relative spatial position of the object. The depth value zi is normalized using the maximum and minimum depth values, scaling its range to the standardized interval (0, 1).

[0167] S203: Axis Detection: As shown in Figure 4 (c), aiming to guide the VLA model to accurately identify the rotation axis type and its movement direction of the target object; in three-dimensional space, the rotation axis of the object is the core feature of its movement, and through the fine-tuning training of the model, it can accurately judge the type and direction of the axis. In this task, the rotation axis type is represented by revolute (revolute joint) and prismatic (prismatic joint). In order to define the direction of the rotation axis, the coordinates of two points {(xi, yi, zi)}, i = 1, 2 are used, that is, the two points on the axis determine the rotation axis direction vector.

[0168] S204: Force Point Detection: As shown in Figure 4 (d), guiding the VLA model to accurately identify the best force point of the target object in order to perform efficient mechanical operation and control; when constructing the sub-task feature set, the position of the best force point is crucial for the stability and force distribution of the object.

[0169] The best force point P optimal of the target of the prismatic joint type, that is, the geometric center of the exposed surface of the target object, is calculated as follows:

[0170] S20401: Define the six faces faces of the three-dimensional bounding box of the object:

[0171] faces{F1, F2,..., F6}, F i ∈R 4×3 (2)

[0172] In formula (2):

[0173] R represents a real number, R 4×3 represents a two-dimensional real number vector with a size of 4x3;

[0174] S20402: Calculate the geometric center point c bbox of the three-dimensional bounding box and the geometric center c i of each face:

[0175]

[0176]

[0177] In formula (3), (4):

[0178] p i represents a three-dimensional vertex in the three-dimensional bounding box, and each three-dimensional bounding box has a total of 8 three-dimensional vertices p idetermine;

[0179] p ij represent three-dimensional vertices in each face F i Each face F i is composed of 4 three-dimensional vertices p ij determine;

[0180] S20403: Calculate the moving direction vector n bbox of each face by the difference between the geometric center point c i of the three-dimensional bounding box and the geometric center point c i of each face:

[0181]

[0182] S20404: According to any two points d1, d2 in the rotation axis, calculate the unit vector d unit of the moving direction:

[0183]

[0184] S20405: Calculate the dot product dot_product i of the moving direction vector n unit of each face and the unit vector d i of the moving direction:

[0185] dot_product i = n i · d unit (7)

[0186] S20406: The closer the dot product dot_product i is to 1, the more consistent it is with the unit vector d unit of the moving direction, so the face F k with the largest dot product is the exposed face:

[0187]

[0188] S20407: Calculate the geometric center of the face F k with the largest dot product, which is the best force point P optimal :

[0189]

[0190] In equation (9):

[0191] p kj represent three-dimensional vertices in each face F k with the largest dot product, each face F k with the largest dot product is composed of 4 three-dimensional vertices p kjDetermination.

[0192] The optimal force point of the target of the revolute joint type in the optimal force point S204 helps to improve the effect of the moment and enhance the control efficiency because it has a rotational degree of freedom and the optimal force point is away from the axis of rotation. In constructing this sub-task feature set, in order to optimize the selection of the force point of the revolute type target, a weight sampling method is adopted. The calculation process is as follows:

[0193] S20401: According to the distance of different points on the target object from the axis of rotation, different points are given different weights; in the calculation process, points far from the axis of rotation are given a weight of 4, considering that these points have a greater impact on the object during force application; points close to the axis of rotation are given a lower weight of 1, considering that they have a smaller contribution to the moment. Through this weighting method, the influence of different points on the calculation of the optimal force point can be effectively adjusted to ensure that the calculation result is more reasonable.

[0194] S20402: Calculate the optimal force point P optimal :

[0195]

[0196] In formula (10):

[0197] w i is the weight of the i-th point;

[0198] S205: Gripper direction detection (Gripper Detection): as shown in Figure 4 (e), the VLA model can accurately identify the direction of the gripper, thereby improving the precision and efficiency of the robot arm operation; in actual articulated object manipulation experiments, the direction of the gripper of the robot arm directly affects the way the object is gripped and forced, and thus determines the effect and stability of the operation. By identifying the correct direction of the gripper and combining the end point information, the robot can calculate the torque of each joint through inverse kinematics algorithm to complete precise object manipulation. For different types of target objects, the definition of the direction of the gripper is also different.

[0199] For targets of the sliding joint type, since the gripper operates along the axial force of the target, the direction of the gripper is the opposite direction of the axis of rotation;

[0200] For targets of the revolute joint type, the direction of the gripper is determined by the cross product of two vectors on the force surface of the target object. The advantage of this method is that it can ensure that the force applied by the gripper acts precisely on the force surface of the object, thereby promoting the rotation of the object and maximizing the stability and success rate of task execution.

[0201] S206: Move planning: As shown in (f), the VLA model converts abstract text instructions into concrete mechanical movements. Figure 4 (f) shows that the VLA model converts abstract text instructions into concrete mechanical movements.

[0202] S20601: Generate a motion planning sequence composed of multiple points based on the results of border detection, axis detection, force point detection, and jaw direction detection:

[0203] {(xi,yi,zi)}(11)

[0204] In equation (11):

[0205] i = 1, …, 4, representing the first point to the fourth point in the motion planning sequence, each point containing the x, y, z coordinates of the robot arm end in three-dimensional space, which accurately defines the movement path of the robot arm end;

[0206] S20602: Calculate the unit vector v of the axis direction based on any two points d1, d2 in the axis direction :

[0207]

[0208] In equation (12):

[0209] d1 = (x1, y1, z1);

[0210] d2 = (x2, y2, z2);

[0211] S20603: Move path planning;

[0212] For sliding joint type targets, calculate the sliding path p t :

[0213] p t = P optimal +t·speed·v direction ,t=0,1,2,3 (13)

[0214] In equation (13):

[0215] speed = 0.1 represents the speed of movement;

[0216] For rotating joint type targets,

[0217] S2060301: Calculate the rotation angle per second angle_per_second:

[0218] angle_per_second = speed × 5 (14)

[0219] In formula (14):

[0220] speed=0.1 represents the speed of movement;

[0221] S2060302: Calculate the corresponding rotation angle θ at each time point t:

[0222] θ=angle_per_second×t,t=0,1,...,3 (15)

[0223] S2060303: Calculate the rotation matrix R(v direction ,θ) by formula (15):

[0224]

[0225] In formula (16):

[0226] u x ,u y ,u t all represent the components of the unit vector v direction of the rotation axis direction;

[0227] S2060304: Rotate the best force point P optimal and record the new best force point P' position:

[0228] P'=(P optimal -d1)·R(v direction ,θ)+d1 (17)

[0229] Through such a sequence, the mechanical arm can accurately perform each action according to the predetermined path, thereby completing a complex task.

[0230] According to the above method, the present application generates a total of six fine-tuning data sets, totaling 1,249,654 data. Subsequently, the data is screened to eliminate abnormal point positions and duplicate data. After filtering, the total data amount is 310,548.

[0231] S3: Design VLA model architecture;

[0232] In the present application, the VLA model is developed based on SPHINX-X. As Figure 5As shown, unlike the Mixture Of Visual Experts (MOV) used in SPHINX-X, the present application adopts Mixed Four Vision Encoders. The Mixed Four Vision Encoders integrate CLIP-VIT and CLIP-ConvNeXt vision encoders and DINOv2-VIT for extracting local semantic features, while utilizing Q-Former to summarize global visual features.

[0233] In order to overcome the challenge of the Mixed Four Vision Encoders in low resolution limitation and enable the VLA model to fully understand the global and local features of the input picture, the following strategies are adopted:

[0234] S301: The input high-resolution image is spatially divided into multiple sub-images, and the high-resolution image is down-sampled into a low-resolution image;

[0235] S302: After inputting multiple images into the Mixed Four Vision Encoders, encoding and concatenating them into a long sequence, then inputting them into the LLaMA2-13B model; among them, the first group of encodings is responsible for global semantics, and the remaining four groups of encodings record fine-grained local features; this design enables the VLA model to adaptively explore finer-grained visual semantics while maintaining encoding efficiency.

[0236] S303: Design a projection layer to realize spatial alignment of visual and language features, which not only enhances the semantic understanding ability of the model, but also improves the accuracy of feature fusion.

[0237] S4: Train the VLA model;

[0238] Train the VLA model in S3 using the feature set generated in S2;

[0239] The instruction following the articulated object feature set format in S2 is the Visual Question Answering (VQA) data set provided by the SPHINX framework by default. After modifying the configuration information, the VLA model can be fine-tuned using the fine-tuning method provided by the SPHINX framework. The present application uses the SPHINX-v2-1k model obtained from the official resource library as the pre-training basis. As shown in Figure 5 As shown, during the entire fine-tuning stage, the Mixed Four Vision Encoders are always in a frozen state, and only the projection layer and LLaMA2-13B are fine-tuned to maintain the integrity of the pre-training features.

[0240] S5: Construct a geometric thinking planner.

[0241] S501: Since the instructions designed in step two follow the hinge object feature set with continuity, that is, the subsequent reasoning task depends on the results of the previous reasoning task. During the inference process using the VLA model, the inference effect of the VLA model on the previous task will significantly affect the inference quality of the subsequent task. In order to improve the accuracy of the VLA model in inferring the geometric center information of the hinge object, and based on these information, accurate three-dimensional manipulation waypoints are provided, such as Figure 6 As shown in the figure, the present application constructs a geometric thinking planner including Advisor (instruction template) and Strategist (scoring algorithm);

[0242] S502: In order to improve the accuracy of the inferred geometric center information as much as possible, Advisor needs to list all the results it thinks possible, and then Strategist scores each result according to the scoring algorithm developed, and finally filters out the best result for the subsequent reasoning task.

[0243] The Strategist scores and filters the answers to the object category identification according to the following steps:

[0244] S50201: In order to score and filter the results of OCI, the present application develops a scoring algorithm based on the contrastive language-image pre-training model (CLIP). The visual-semantic embedding model ViT-L / 14 is adopted to map the picture and OCIPossible Results into a shared embedding space. Specifically, the contrastive language-image pre-training model converts the input image and text candidate into high-dimensional vector representation through two independent encoders (an image encoder and a text encoder):

[0245] I=f image (x image ),T=f text (x text ) (18)

[0246] In formula (18):

[0247] x image represents the input image;

[0248] x text represents the input text candidate;

[0249] f image represents the image encoder of the CLIP model;

[0250] f text represents the text encoder of the CLIP model;

[0251] I represents the image feature vector in the shared embedding space;

[0252] T represents a text feature vector in a shared embedding space;

[0253] S50202: Calculate the similarity between the image embedding and the text embedding, ensure that the similarity value is in the range [-1, 1], and generate a score for category prediction:

[0254] In this process, cosine similarity is used as an indicator to measure similarity.

[0255]

[0256] In equation (19):

[0257] I represents the dot product of the image embedding vector of the object category recognition calculation;

[0258] T represents the dot product of the text embedding vector of the object category recognition calculation;

[0259] ||I|| represents the Euclidean norm of the dot product of the image embedding vector of the object category recognition calculation;

[0260] ||T|| represents the Euclidean norm of the dot product of the text embedding vector of the object category recognition calculation;

[0261] S50203: Select the result with the highest similarity as the best object category recognition answer:

[0262] Best Prediction = argmax(Similarity(I, T)) (20).

[0263] This method ensures the accuracy and objectivity of the scoring process, and finally returns the text description that best matches the input image.

[0264] The Strategist of S502 scores and filters the answers of the stress point detection as follows:

[0265] This Strategist aims to select a BBOX best stress point from several points given by the Advisor. By calculating the distance of each point to each face of the BBOX, the point closest to the BBOX is selected.

[0266] S50201: Define a plane by any three points P1, P2, P3, and calculate the normal vector of the plane

[0267]

[0268] In equation (21):

[0269]

[0270] S50202: Determine the equation of the plane according to the components A, B, C of the normal vector

[0271] Ax+By+Cz+D=0 (22)

[0272] S50203: Calculate D according to any point P1(x0, y0, z0) on the plane

[0273] D=-(A·x0+B·y0+C·z0) (23)

[0274] S50204: Calculate the distance of the target point P(x, y, x) to the plane

[0275]

[0276] S50205: Calculate the distance of each point P in the given point set to the six faces of the three-dimensional bounding box in turn i

[0277] S50206: Calculate the total score S of all points (the cumulative value of all face distances) i

[0278]

[0279] In formula (25):

[0280] d i,j represents the distance of point P i to any face F j of the three-dimensional bounding box, F j is F1, F2,..., F6

[0281] S50207: The score reflects the closeness of the point to the bounding box as a whole, the lower the score, the higher the contact degree of the point to the bounding box. Finally, sort the total score S of all points i , and select the point with the lowest score as the optimal point:

[0282]

[0283] The Strategist described in S502 scores and filters the answers of the axis detection, including the following steps:

[0284] The purpose of this Strategist is to evaluate the alignment between a set of gripper directions and the target axis, and select the optimal grasping direction according to the alignment.

[0285] ​​​S50201: Calculate the cosine value of each grasp direction vector and the target axis vector, which reflects the alignment degree between the two directions. Specifically, given the target axis vector axis and a certain grasp direction vector g i , calculate the cosine value similarity of the two:

[0286]

[0287] In formula (27):

[0288] axis represents the three-dimensional vector of the target axis;

[0289] g i represents the three-dimensional vector of the grasp direction;

[0290] ||axis|| represents the Euclidean norm of the three-dimensional vector of the target axis;

[0291] ||g i || represents the Euclidean norm of the three-dimensional vector of the grasp direction;

[0292] S50202: When calculating, invalid vectors (such as vectors with zero length) will be filtered out, and the cosine value and its corresponding grasp direction index will be stored. According to the different types of joints, the grasp direction is selected:

[0293] For sliding joint type targets, select the grasp direction that is most aligned with the target direction;

[0294] For rotating joint type targets, select the direction that is least aligned with the target direction.

[0295] Example 1

[0296] In order to measure the performance of the present application, the articulated object in PartNet-Mobility and the Franka Panda Robot gripper are placed in the Sapien simulator, and then the Franka Panda Robot gripper is set according to the best force point position and gripper direction inferred by the VLA model.

[0297] Evaluation criteria. This experiment aims to test the performance of the model interacting with the environment. In each task, an object will be selected from the PartNet-Mobility dataset. Then, guide the gripper to interact with the articulated object. The base of the object remains stationary, and the gripper must identify the movable part and master the correct driving direction.

[0298] Comparative analysis. The success rates of the pinch operation in 18 training classes (used in the training phase) and 9 test classes (not used in the training phase) were compared. The method proposed in the present invention was compared with the following six baseline methods:

[0299] (1) Where2Act: This method processes point cloud data by scoring each point to assess its importance, and selects the point with the highest score as the contact point. At the same time, it also predicts the possible direction of the end effector, and selects the best direction to determine the contact pose.

[0300] (2) UMPNet: Similar to Where2Act, UMPNet ensures that the direction of the end effector is perpendicular to the object surface based on the prediction of the contact point.

[0301] (3) Flowbot3D: This method identifies the direction of motion in the point cloud, called "flow", and selects the point with the largest flow to determine the interaction point, and the direction of the flow determines the orientation of the end effector.

[0302] (4) Implicit3D: This method identifies key points on three-dimensional articulated objects by applying Transporter network, thereby formulating strategies for subsequent manipulation tasks. These key points help to plan the pose of the end effector.

[0303] (5) ManipLLM: ManipLLM uses a vision-language model to combine RGBD images and language prompts to predict the contact point and forward direction of the suction manipulator.

[0304] (6) A3VLM: This method focuses on the object and uses a vision-language model to detect the object's three-dimensional bounding box, axis, joint type, and link name to guide the manipulation process.

[0305] Results. Figure 7 The performance data of the present invention and the six control groups were compared. The analysis table results showed that in most object categories, the performance of the present invention exceeded that of all other control groups. This performance improvement is mainly due to two key factors: one is the accurate identification of the object geometric center information by VLA, and the other is the accurate decision-making of Advisor-Strategist in the geometric thinking planner. These methods support the high efficiency of the present invention in handling complex articulated objects.

[0306] Example 2

[0307] A task decomposition strategy was implemented in S2 with the goal of enabling the VLA model to better understand and absorb key information. Specifically, the user instruction was subdivided into six unique subtasks, collectively referred to as T6. By fine-tuning the VLA model separately for these subdivided tasks, the performance of the model could be optimized. To validate the effectiveness of this task decomposition approach, we removed the object class identification (OCI), point location detection, and gripper detection subtasks from T6, respectively. After removing these subtasks, the VLA model was used to reason about the movement path planning for articulated objects, with the relevant results shown in Figure 8

[0308] The analysis results show that after removing the OCI subtask, the VLA model encountered serious problems in identifying BBOX, which in turn led to errors in identifying all relevant Action Affordances. When the Gripper Detection subtask was removed, the VLA model made directional errors in identifying the movement path. And when the Force Point Detection subtask was removed, the VLA model made errors in identifying the starting point of the movement path, which in turn caused the entire path to deviate. These performance declines highlight the importance of reasonable division of subtasks, which is crucial for ensuring that the model can accurately understand and process complex information.

[0309] The results of Example 2 are shown in Figure 8 (a) T6 shows the results of reasoning after fine-tuning the VLA model using the complete feature set. (b) T6-OCI, (c) T6-Gripper Detection, and (d) T6-Force Point Detection show the results obtained by fine-tuning the VLA model after removing the OCI, Gripper Detection, and Force Point Detection subtasks, respectively.

[0310] Example 3

[0311] The development process of the geometric thinking planner is described in detail in S5. This approach aims to improve the accuracy of the VLA's understanding of the geometric center information of articulated objects through the Advisor and Strategist, and to provide accurate three-dimensional manipulation waypoints based on this information. The goal of this section is to validate the effectiveness of the geometric thinking planner. We removed the geometric thinking planner and instead used the chain-of-thought prompting method to reason. This method involves incorporating the answers obtained in the previous tasks into the question of the subsequent task, and using this question to ask the VLA model.

[0312] ​In the design of the experiment, the results of the geometric thinking planner (GTP) were used as a control group, and four experimental groups were set up to compare the effects of using COT to replace GTP at different reasoning stages: the GTP-w / o OCI group only replaces GTP with OCI in the OCI stage, the GTP-w / o Point group only replaces GTP with OCI in the point detection stage, the GTP-w / o Direction group only replaces GTP with OCI in the direction detection stage, and the COT group replaces GTP in the OCI, point detection, and direction detection stages. Such experimental design aims to evaluate the performance impact of replacing GTP with COT at each stage.

[0313] Results, Figure 9 The performance data of using COT in each stage is compared with the control group using GTP. The success rate of the reasoning results of the geometric thinking planner and COT is evaluated in the seen category and the unseen category. The weighted average of the seen category and the unseen category is used as the standard to measure performance, with the weight being the number of articulated objects of each type. The analysis results show that when using the geometric thinking planner, the performance of VLA reasoning is significantly better than using COT only. This performance advantage can be attributed to the Strategist in the geometric thinking planner, which can re-examine and select the optimal solution, thereby reducing the likelihood of decision errors and improving the accuracy of planning. This indicates that the geometric thinking planner has a significant advantage in handling complex geometric information and providing accurate three-dimensional maneuver waypoints.

[0314] It is obvious to those skilled in the art that the present application is not limited to the details of the above-described exemplary embodiments, but can be implemented in other forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered in all aspects as exemplary and non-limiting, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended to encompass all changes falling within the meaning and scope of equivalents of the claims. Any reference signs in the claims should not be considered as limiting the claims to which they relate.

[0315] In addition, it should be understood that although the present specification is described in terms of embodiments, not every embodiment contains only one independent technical solution, and the description of the specification is only for the sake of clarity, and those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that those skilled in the art can understand.

Claims

1. A method for constructing a large VLA model for manipulation of articulated objects, characterized by: The method comprises the following steps: S1: Use geometric center representation to describe the action affordance of articulated objects: A={B,D,F,M,X} (1) In formula (1): B represents the 3D bounding box of the articulated object; D represents the direction of the end effector of the robotic gripper; F represents the best force point; M represents a movable path planning point sequence; X represents the position of the shaft; S2: Construct instructions that follow the feature set of articulated objects. The feature set format is consistent with the MLLM fine-tuning dataset. The input is an image and text instruction, and the output is formatted text, namely the action affordance of the articulated object. Decompose an instruction into six different subtask feature sets. S3: Design VLA model architecture; S4: train the VLA model; Use the feature set generated in S2 to train the VLA model in S3; S5: Constructing a geometric thinking planner; The S5 comprises the following steps: S501: Build a geometric thinking planner including Advisor and Strategist; S502: The Advisor asks the VLA model to list all results. The Strategist then scores each result according to the specified scoring algorithm and ultimately selects the best result for subsequent reasoning tasks. The Strategist scoring and screening of the answers to the object category recognition in S502 includes the following steps: S50201: The contrastive language-image pre-training model converts the input image and text candidates into high-dimensional vector representations through two independent encoders: I=f image (x image ),T=f text (x text ) (18) In formula (18): x image represents the input image; x text Represents text candidates for input; f image The image encoder representing the CLIP model; f text Represents the text encoder of the CLIP model; I represents the image feature vector in the shared embedding space; T represents the text feature vector in the shared embedding space; S50202: Calculate the similarity between the image embedding and the text embedding, ensure that the similarity value is in the range [-1, 1], and generate a score for the category prediction: In formula (19): I represents the dot product of the image embedding vector calculated for object category recognition; T represents the dot product of the text embedding vector calculated for object category recognition; ||I|| represents the Euclidean norm of the dot product of the image embedding vector calculated for object category recognition; ||T|| represents the Euclidean norm of the dot product of the text embedding vector calculated for object category recognition; S50203: Select the result with the highest similarity as the best object category recognition answer: Best Prediction=argmax(Similarity(I,T)) (20).

2. The method for constructing a large VLA model for manipulation of articulated objects according to claim 1, characterized in that: The subtask feature set of S2 includes: S201: Object category recognition: guide the VLA model to output the name of the specific object to be operated based on the given image and user instructions; S202: Bounding box detection: aims to guide the VLA model to accurately identify and mark the specific location of the target object in three-dimensional space; S203: Rotation axis detection: aims to guide the VLA model to accurately identify the rotation axis type and movement direction of the target object; S204: Force point detection: guides the VLA model to accurately identify the optimal force point of the target object to enable efficient mechanical operation and control; S205: Gripper direction detection: This guides the VLA model to accurately identify the direction of the gripper, thereby improving the accuracy and efficiency of the robot arm operation; For the sliding joint type target, since the gripper operates along the axial thrust of the target, the direction of the gripper is opposite to the direction of the rotation axis; For targets of the revolute joint type, the orientation of the gripper is determined by the cross product of two vectors on the force-bearing surface of the target object; S206: Mobile path planning: guiding the VLA model to convert abstract text instructions into specific mechanical movements.

3. The method for constructing a large VLA model for manipulation of articulated objects according to claim 2, characterized in that: S204 The optimal force point P optimal The optimal force point for a target with a sliding joint type, that is, the geometric center of the exposed surface of the target object, is calculated as follows: S20401: Six faces defined by the object's 3D bounding box: faces{F1,F2,...,F6},F i ∈R 4×3 (2) In formula (2): R represents a real number, R 4×3 Represents a two-dimensional real vector of size 4×3; S20402: Calculate the geometric center point c of the 3D bounding box bbox and the geometric center c of each face i : In formulas (3) and (4): p i Represents the three-dimensional vertices in the three-dimensional bounding box. Each three-dimensional bounding box consists of a total of 8 three-dimensional vertices p i Sure; p ij Indicates surface F i The three-dimensional vertices in each face F i There are 4 three-dimensional vertices p ij Sure; S20403: Through the geometric center point c of the 3D bounding box bbox and the geometric center point c of each face i The difference between the two calculates the moving direction vector n of each face i : S20404: Calculate the unit vector d in the direction of motion based on any two points d1 and d2 on the axis of rotation unit : S20405: Calculate the moving direction vector n of each face i The unit vector d with the direction of motion unit dot_product i : dot_product i =n i ·d unit (7) S20406: dot product i The closer it is to 1, the closer it is to the unit vector d in the direction of motion. unit The more consistent, therefore, the face F with the largest dot product k This is the exposed surface: S20407: Calculate the face F with the largest dot product k The geometric center, that is, the optimal force point P optimal : In formula (9): p kj Represents the face F with the largest dot product k The three-dimensional vertices in each face with the largest dot product F k There are 4 three-dimensional vertices p kj Sure.

4. The method for constructing a large VLA model for manipulation of articulated objects according to claim 2, characterized in that: The optimal force point of the target of the rotation joint type in the optimal force point in S204 is located far away from the rotation axis due to its rotational freedom. The calculation process is as follows: S20401: assign different weights to different points on the target object according to their distance from the rotation axis; S20402: Calculate the optimal force point P optimal : In formula (10): w i is the weight of the i-th point.

5. The method for constructing a large VLA model for manipulation of articulated objects according to claim 3 or 4, characterized in that: The S206 includes the following steps: S20601: Generates a motion planning sequence consisting of multiple points based on the results of frame detection, axis detection, force point detection, and gripper direction detection: {(xi,yi,zi)}(11)In formula (11): i = 1, ..., 4, representing the first to fourth points in the motion planning sequence. Each point contains the x, y, and z coordinates of the end of the manipulator in three-dimensional space, which define the movement path of the end of the manipulator. S20602: Calculate the unit vector v in the direction of the axis of rotation based on any two points d1 and d2 on the axis of rotation direction : In formula (12): d1=(x1,y1,z1); d2=(x2,y2,z2); S20603: Mobile path planning; For sliding joint type targets, calculate the sliding path p t : p t =P optimal +t·speed·v direction ,t=0,1,2,3 (13) In formula (13): speed=0.1 indicates the speed of movement; For revolute joint type targets, S2060301: Calculate the rotation angle angle_per_second per second: angle_per_second=speed×5 (14) In formula (14): speed=0.1 indicates the speed of movement; S2060302: Calculate the corresponding rotation angle θ at each time point t: θ=angle_per_second×t,t=0,1,...,3 (15) S2060303: Calculate the rotation matrix R(v) by formula (15) direction ,θ): In formula (16): u x ,u y ,u z Both represent the unit vector v in the direction of the axis of rotation direction The weight; S2060304: The best force point P optimal Rotate and record the new optimal force point P' position: P'=(P optimal -d1)·R(v direction ,θ)+d1 (17).

6. The method for constructing a large VLA model for manipulation of articulated objects according to claim 5, characterized in that: The S3 comprises the following steps: S301: spatially dividing an input high-resolution image into multiple sub-images, and downsampling the high-resolution image into a low-resolution image; S302: Multiple images are simultaneously input into the hybrid visual encoder, encoded and concatenated into a long sequence, and then input into the LLaMA2-13B model. The first set of codes is responsible for global semantics, and the remaining four sets of codes record fine-grained local features. S303: Design a projection layer to achieve spatial alignment of visual and language features.

7. The method for constructing a large VLA model for manipulation of articulated objects according to claim 6, characterized in that: The Strategist scoring and screening of the answers to the stress point detection in S502 includes the following steps: S50201: Define a plane by any three points P1, P2, and P3 and calculate the plane's normal vector In formula (21): S50202: According to the normal vector The components A, B, and C determine the equation of the plane: Ax+By+Cz+D=0 (22) S50203: Calculate D based on any point P1 (x0, y0, z0) on the surface: D=-(A·x0+B·y0+C·z0) (23) S50204: Calculate the distance from the target point P(x,y,x) to the surface: S50205: Calculate each point P in the given point set in turn i The distances to the six faces of the 3D bounding box; S50206: Calculate the total score S of all points i : In formula (25): d i,j Represents point P i To any face of the 3D bounding box F j The distance, F j is F1, F2, ..., F6; S50207: Total score S for all points i Sort and select the point with the lowest score as the optimal point:

8. The method for constructing a large VLA model for manipulation of articulated objects according to claim 7, characterized in that: The Strategist scoring and screening of the answers to the pivot test in S502 includes the following steps: S50201: Given a target axis vector axis and a grasping direction vector g i , calculate the cosine similarity between the two: In formula (27): In formula (27): axis represents the three-dimensional vector of the target axis; g i A three-dimensional vector representing the grasping direction; ||axis|| represents the Euclidean norm of the three-dimensional vector of the target axis; ||g i ||Euclidean norm of the three-dimensional vector representing the grasping direction; S50202: Select the grasping direction according to the joint type: For sliding joint type targets, the grasping direction that is most aligned with the target direction is selected; For revolute joint type targets, chooses the orientation that is least aligned with the target orientation.

Images