Construction method of VLA large model for hinged object manipulation

Abstract

The invention discloses a construction method of a VLA large model for hinged object control, and belongs to the technical field of intelligent technology. The method comprises the following steps: describing the Action Affordance of a hinged object by using a geometric center representation method; constructing an instruction for following the feature set of the hinged object; decomposing an instruction into six different sub-task feature sets; designing and training a VLA model architecture; and constructing a geometric thinking planner. According to the method, a geometric center representation method is used for describing a hinged object, a geometric thinking planner is developed, the understanding of a VLA large model to the environment is promoted, the adaptability of the VLA large model to complex tasks is enhanced, the VLA large model can process diversified scenes and requirements, the VLA model can fully understand behaviors and responses of the object under different conditions, and the modeling efficiency of the VLA large model is improved. Therefore, a more reasonable decision is made when the motion path is planned, and the precision and reliability of path planning are improved.

Description

Technical Field

[0001] The invention relates to a method for constructing a VLA large model for articulated object manipulation, and belongs to the technical field of embodied intelligence. Background Art

[0002] An articulated object is an object composed of multiple rigid components connected by joints, and each rigid component can rotate or translate around the joint. This form is often used in the connection of furniture, doors and windows. Compared with ordinary rigid objects with only 6 degrees of freedom (DoF), articulated objects have higher degrees of freedom and are more difficult for robots to understand and interact with.

[0003] Multimodal Large Model (MLLM) is a large language model that can process and understand multimodal data. Compared with traditional language models (such as GPT or BERT) that can only process text, MLLM can combine multiple modalities (such as text, images, video, audio, and even 3D data) for unified processing and reasoning.

[0004] The VLA model, whose full name is the Vision-Language-Action model, is a multimodal artificial intelligence model that integrates the capabilities of visual perception, language understanding, and action generation. It can process images, text instructions, and operation tasks in complex scenes, forming a closed loop from perception to decision-making to execution. For example, the VLA model can understand the natural language instructions provided by the user (such as "grab the red cup on the table"), analyze visual information (such as the position and features of objects in the scene), and generate corresponding operation plans (such as the grasping path of the robotic arm). This model is widely used in fields such as robot control and industrial automation, and achieves the ability to interact naturally and efficiently execute tasks through a unified multimodal processing framework.

[0005] The action affordance of an articulated object refers to the operational possibilities and interaction methods provided by the articulated object through motion constraints such as hinges and slides. Due to the existence of movable parts, this type of object not only has the functional characteristics of a static object, but also contains operable dynamic characteristics. For example, the hinge of a door implies that it can be opened by pushing or pulling, and the slide of a drawer indicates that it can be pulled out or pushed in. In robot manipulation and task planning, studying the action affordance of articulated objects helps the model understand its motion laws, constraints, and applicable actions, thereby generating reasonable manipulation strategies, such as complex interactive behaviors such as opening doors and closing drawers.

[0006] Flexible manipulation of articulated objects is of great significance to the development of intelligent multi-purpose robots. These objects vary significantly in their motion characteristics, so manipulation techniques need to be designed specifically for different types of objects. In order to achieve stable execution of tasks in a changing real-world environment, researchers have proposed a variety of methods for general object manipulation, mainly including imitation learning through demonstration and reinforcement learning in a simulated environment. Imitation learning usually uses data from human demonstrations to allow robots to master manipulation skills through imitation, which is particularly effective when dealing with complex or difficult-to-model tasks. However, imitation learning often requires a large amount of diverse data so that the robot can show good adaptability when encountering new environments or different objects. On the other hand, reinforcement learning allows robots to autonomously explore and optimize strategies in a simulated environment, improving manipulation capabilities through a lot of trial and error. Such as Figure 1 (a) shows the data requirements in this process. These technologies rely on large robot interaction datasets to help robots effectively learn and adapt to the requirements of different tasks. In real life, collecting these datasets is often time-consuming and laborious.

[0007] Recently, the use of large language models (LLMs) for robotic applications has attracted widespread attention. In recent years, LLMs are no longer limited to the field of natural language processing, but have gradually expanded to the field of robotic control, especially robotic arm control. LLMs have powerful reasoning capabilities and can use prompt templates to decompose user instructions, helping robots understand and perform complex tasks more efficiently. In articulated object manipulation, large language models can be used as high-level task planning and instruction parsing tools. There are methods to understand natural language instructions through LLMs, and combine them with written fixed manipulation APIs and scene information to generate operation plans for robotic arms. LLMs mainly focus on processing text data, but they are generally not good at processing visual information, which limits their performance in tasks that require combining visual inputs (such as images or videos) to make decisions. For example Figure 1 As shown in (b), in the current method, LLM needs to plan the motion of the robot arm based on URDF (Uniform Robot Description Format) information. Although this algorithm is effective in the IsaacGym virtual environment, it lacks generalization ability and is difficult to adapt to the real environment. This is because the real world scene does not always provide fixed URDF information.

[0008] Compared with LLM, MLLM can process data from multiple modalities, including images, text, and video. This ability enables them to perform better in complex environments, especially in tasks that require the integration of visual information and language instructions. Figure 1As shown in (c), many current methods use MLLM in robotic applications. Using MLLM for object bearing force identification and motion planning can automatically execute user instructions to complete manipulation tasks in the real world. However, in the real world, collecting such a large amount of robot interaction data is very expensive. In addition, these methods focus on obtaining the contact point information of the object and do not have a complete robot arm operation plan. If you want to deploy it in reality to control the robot arm to complete the task, you need to manually implement the motion planning algorithm.

[0009] In addition, some methods have trained the VLA model based on the Open-X dataset. Compared with the MLLM model, the VLA model can directly plan the physical operations to be performed based on the understanding of vision and language. Figure 1 As shown in (d), the model maps the joint torque of the robot arm to a continuous interval, and obtains N discrete integers ∈ [0...255], which are used to express the robot action with N degrees of freedom, showing strong generalization ability in real environments. The VLA model can flexibly cope with a variety of operation tasks and performs particularly well in complex and varied work scenarios. However, such methods often rely on a large amount of image-action data collection, which not only increases the difficulty of data preparation, but also significantly increases the training cost. Taking the study of OpenVLA as an example, the authors used a cluster of 64 A100 GPUs for 14 days of training, consuming a total of 21,500 A100 GPU hours. In addition, many large VLA models are category-related. The scope of use of these models is limited by the datasets they are trained on, which means that the same type of VLA model can only be applied to the same type of mechanical body, resulting in their limited generalization ability.

[0010] In order to solve the above problems, the present invention designs a new method specifically for training a VLA model for manipulating articulated objects, such as Figure 1(e). The present invention includes an end-to-end VLA large model that can solve the problem of manipulating articulated objects using multimodal data. It is based on the geometric center, is designed specifically for manipulating articulated objects, and can accurately plan the motion path of the end of the robotic arm in the current scene. Compared with traditional methods, a significant advantage of the present invention is that it realizes the full process integration from perception to motion control. It can generate the end motion path and easily control the robotic arm in a real environment through spline interpolation (Spline) instructions to successfully complete various operation tasks. In addition, the present invention can understand the Action Affordance of articulated objects without relying on expensive robot interaction data, which means that it can identify and execute possible actions in the environment. More importantly, the method is category-agnostic, which means that different robotic arms can use the same learning model to complete manipulation tasks, thereby improving the versatility and applicability of the VLA model. Summary of the invention

[0011] In order to solve the problems existing in the background technology, the present invention provides a method for constructing a VLA large model for manipulation of articulated objects.

[0012] To achieve the above object, the present invention adopts the following technical solution: a method for constructing a VLA large model for articulated object manipulation, the method comprising the following steps:

[0013] S1: Using geometric center representation to describe the action affordance of articulated objects:

[0014] A={B,D,F,M,X} (1)

[0015] In formula (1):

[0016] B represents the 3D bounding box of the articulated object;

[0017] D represents the direction of the end effector of the mechanical gripper;

[0018] F represents the best force point;

[0019] M represents a movable path planning point sequence;

[0020] X represents the position of the rotating shaft;

[0021] S2: Construct instructions that follow the feature set of articulated objects. The feature set is in the same format as the MLLM fine-tuning dataset. The input is image and text instructions, and the output is formatted text, i.e., the Action Affordance of the articulated object. Decompose an instruction into six different subtask feature sets.

[0022] S201: Object category recognition: guide the VLA model to output the name of the specific object to be operated according to the given picture and user instructions;

[0023] S202: Bounding box detection: aims to guide the VLA model to accurately identify and mark the specific location of the target object in three-dimensional space;

[0024] S203: Rotation axis detection: aims to guide the VLA model to accurately identify the rotation axis type and movement direction of the target object;

[0025] S204: Force point detection: guide the VLA model to accurately identify the best force point of the target object to perform efficient mechanical operation and control;

[0026] S204 The optimal force point P optimal The calculation process of the optimal force point of the target of the sliding joint type, that is, the geometric center of the exposed surface of the target object, is as follows:

[0027] S20401: Six faces defined by the object's 3D bounding box:

[0028] faces{F1,F2,...,F6},F i ∈R 4×3 (2)

[0029] In formula (2):

[0030] R represents a real number, R 4×3 Represents a two-dimensional real vector of size 4×3;

[0031] S20402: Calculate the geometric center point c of the 3D bounding box bbox and the geometric center c of each face i :

[0032]

[0033] In formula (3) and (4):

[0034] p i Represents the three-dimensional vertices in the three-dimensional bounding box. Each three-dimensional bounding box consists of a total of 8 three-dimensional vertices p i Sure;

[0035] p ij Indicates face F i The three-dimensional vertices in each face F i There are 4 three-dimensional vertices p ij Sure;

[0036] S20403: Through the geometric center point c of the 3D bounding box bbox and the geometric center point c of each facei The difference between the moving direction vector n of each face is calculated i :

[0037]

[0038] S20404: Calculate the unit vector d of the direction of motion based on any two points d1 and d2 on the axis of rotation unit :

[0039]

[0040] S20405: Calculate the moving direction vector n of each face i The unit vector d with the direction of motion unit dot_product i :

[0041] dot_product i =n i ·d unit (7)

[0042] S20406: dot product i The closer it is to 1, the closer it is to the unit vector d in the direction of motion. unit The more consistent, therefore, the face F with the largest dot product k The exposed surface:

[0043]

[0044] S20407: Calculate the face F with the largest dot product k The geometric center, that is, the best force point P optimal :

[0045]

[0046] In formula (9):

[0047] p kj The face F with the largest dot product k The three-dimensional vertices in each face F with the largest dot product k There are 4 three-dimensional vertices p kj Sure.

[0048] The best force point of the target of the rotation joint type in the best force point in S204 is at a position far away from the rotation axis because it has rotational freedom. The calculation process is as follows:

[0049] S20401: assigning different weights to different points on the target object according to their distance from the rotation axis;

[0050] S20402: Calculate the best force point P optimal :

[0051]

[0052] In formula (10):

[0053] w i is the weight of the i-th point;

[0054] S205: Gripper direction detection: guiding the VLA model to accurately identify the direction of the gripper, thereby improving the accuracy and efficiency of the robot arm operation;

[0055] For the sliding joint type target, since the gripper operates along the axial thrust of the target, the direction of the gripper is opposite to the direction of the rotation axis;

[0056] For targets of the revolute joint type, the orientation of the gripper is determined by the cross product of two vectors on the force-bearing surface of the target object;

[0057] S206: Mobile path planning: guiding the VLA model to convert abstract text instructions into specific mechanical movements.

[0058] S20601: Generate a motion planning sequence consisting of multiple points based on the results of frame detection, shaft detection, force point detection, and gripper direction detection:

[0059] {(xi,yi,zi)}(11)

[0060] In formula (11):

[0061] i=1,…,4, represents the 1st to 4th points in the motion planning sequence, each of which contains the x, y, and z coordinates of the end of the robot in three-dimensional space, and the coordinates define the movement path of the end of the robot;

[0062] S20602: Calculate the unit vector v in the direction of the axis of rotation based on any two points d1 and d2 on the axis of rotation direction :

[0063]

[0064] In formula (12):

[0065] d1=(x1,y1,z1);

[0066] d2=(x2,y2,z2);

[0067] S20603: moving path planning;

[0068] For sliding joint type targets, calculate the sliding path pt :

[0069] p t =P optimal +t·speed·v direction ,t=0,1,2,3 (13)

[0070] In formula (13):

[0071] speed=0.1 indicates the speed of movement;

[0072] For revolute joint type targets,

[0073] S2060301: Calculate the rotation angle angle_per_second per second:

[0074] angle_per_second=speed×5 (14)

[0075] In formula (14):

[0076] speed=0.1 indicates the speed of movement;

[0077] S2060302: Calculate the corresponding rotation angle θ at each time point t:

[0078] θ=angle_per_second×t,t=0,1,...,3 (15)

[0079] S2060303: Calculate the rotation matrix R(v) by formula (15) direction ,θ):

[0080]

[0081] In formula (16):

[0082] u x ,u y ,u z Both represent the unit vector v in the direction of the rotation axis. direction The weight of

[0083] S2060304: The best force point P optimal Rotate and record the new best force point P' position:

[0084] P'=(P optimal -d1)·R(v direction ,θ)+d1 (17).

[0085] S3: Design VLA model architecture;

[0086] S301: spatially dividing an input high-resolution image into a plurality of sub-images, and downsampling the high-resolution image into a low-resolution image;

[0087] S302: Multiple images are simultaneously input into the hybrid visual encoder, encoded and concatenated into a long sequence, and then input into the LLaMA2-13B model; the first group of codes is responsible for global semantics, and the remaining four groups of codes record fine-grained local features;

[0088] S303: Design a projection layer to achieve spatial alignment of visual and language features.

[0089] S4: train the VLA model;

[0090] Use the feature set generated in S2 to train the VLA model in S3;

[0091] S5: Construct a geometric thinking planner.

[0092] S501: Build a geometric thinking planner including Advisor and Strategist;

[0093] S502: Advisor lets the VLA model list all the results, and then the Strategist scores each result according to the specified scoring algorithm, and finally selects the best result for subsequent reasoning tasks.

[0094] S502 The Strategist scores and screens the answers to the object category recognition, including the following steps:

[0095] S50201: The contrastive language-image pre-training model converts the input image and text candidates into high-dimensional vector representations through two independent encoders:

[0096] I=f image (x image ),T=f text (x text ) (18)

[0097] In formula (18):

[0098] x image represents the input image;

[0099] x text Represents text candidates for input;

[0100] f image An image encoder representing the CLIP model;

[0101] f text Represents the text encoder of the CLIP model;

[0102] I represents the image feature vector in the shared embedding space;

[0103] T represents the text feature vector in the shared embedding space;

[0104] S50202: Calculate the similarity between the image embedding and the text embedding, ensure that the similarity value is in the range [-1,1], and generate a score for the category prediction:

[0105]

[0106] In formula (19):

[0107] I represents the dot product of the image embedding vector calculated for object category recognition;

[0108] T represents the dot product of the text embedding vector calculated for object category recognition;

[0109] ||I|| represents the Euclidean norm of the dot product of the image embedding vector calculated for object category recognition;

[0110] ||T|| represents the Euclidean norm of the dot product of the text embedding vector calculated for object category recognition;

[0111] S50203: Select the result with the highest similarity as the best object category recognition answer:

[0112] Best Prediction=argmax(Similarity(I,T)) (20).

[0113] S502 The Strategist scores and screens the answers to the stress point detection, including the following steps:

[0114] S50201: Define a plane by any three points P1, P2, P3 and calculate the normal vector of the plane

[0115]

[0116] In formula (21):

[0117]

[0118] S50202: According to the normal vector The components A, B, and C determine the equation of the plane:

[0119] Ax+By+Cz+D=0 (22)

[0120] S50203: Calculate D based on any point P1 (x0, y0, z0) on the surface:

[0121] D=-(A·x0+B·y0+C·z0) (23)

[0122] S50204: Calculate the distance from the target point P(x,y,x) to the surface:

[0123]

[0124] S50205: Calculate each point P in the given point set in turn i The distances to the six faces of the 3D bounding box;

[0125] S50206: Calculate the total score S of all points i :

[0126]

[0127] In formula (25):

[0128] d i,j Indicates point P i To any face of the 3D bounding box F j The distance, F j is F1, F2, ..., F6;

[0129] S50207: Total score S for all points i Sort and select the point with the lowest score as the optimal point:

[0130]

[0131] S502 The Strategist scores and screens the answers to the pivot detection, including the following steps:

[0132] S50201: Given a target axis vector axis and a grasping direction vector g i , calculate the cosine similarity between the two:

[0133]

[0134] In formula (27):

[0135] In formula (27):

[0136] axis represents the three-dimensional vector of the target axis;

[0137] g i A three-dimensional vector representing the grasping direction;

[0138] ||axis|| represents the Euclidean norm of the three-dimensional vector of the target axis;

[0139] ||g i||Euclidean norm of the three-dimensional vector representing the grasping direction;

[0140] S50202: Select the grasping direction according to the different joint types:

[0141] For sliding joint type targets, select the grasping direction that is most aligned with the target direction;

[0142] For revolute joint type targets, choose the orientation that is least aligned with the target orientation.

[0143] Compared with the prior art, the present invention has the following beneficial effects:

[0144] The present invention first uses the geometric center representation method to describe articulated objects, which can capture the action affordance of articulated objects in a fine-grained manner. Secondly, after constructing the VLA large model, a geometric thinking planner was developed. This planner not only promotes the VLA large model's understanding of the environment, but also enhances its adaptability to complex tasks, enabling it to handle diverse scenarios and needs. The core of this planner is an Advisor-Strategist evaluation mechanism. Through this evaluation mechanism, the VLA model can fully understand the behavior and reaction of objects in different situations, so as to make more reasonable decisions when planning motion paths and improve the accuracy and reliability of path planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0145] Figure 1 is a schematic diagram of background technology;

[0146] Figure 2 is a schematic diagram of the present invention;

[0147] Figure 3 It is a schematic diagram of the geometric center representation;

[0148] Figure 4 It is a schematic diagram of the feature set of the instruction-following articulated object;

[0149] Figure 5 This is a schematic diagram of the VLA model architecture;

[0150] Figure 6 It is a schematic diagram of the geometric thinking planner;

[0151] Figure 7 It is a schematic diagram of the results of Example 1;

[0152] Figure 8 It is a schematic diagram of the results of Example 2;

[0153] Fig. 9 It is a schematic diagram of the results of Example 3. DETAILED DESCRIPTION

[0154] The technical solution of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0155] A method for constructing a VLA large model for articulated object manipulation, the input is a URDF file of the articulated object, and the output is a VLA large model with generalization ability, which can infer the end movement path of the robot arm based on visual information and user instructions. The method comprises the following steps:

[0156] S1: Many methods use the robot as the center to represent articulated objects. This representation method loses a lot of physical information of the articulated objects, such as the type of rotation axis and the direction of movement. Figure 3 As shown, the geometric center representation is used to describe the Action Affordance of an articulated object:

[0157] A={B,D,F,M,X} (1)

[0158] In formula (1):

[0159] B represents the 3D bounding box (BBOX) of the articulated object, defined by eight vertices (xi, yi, zi) i = 0…7;

[0160] D represents the direction of the end effector of the robot gripper, which is defined by two vertices (xi, yi, zi) i = 0, 1;

[0161] F represents the optimal force point (Force Point), which is defined by a vertex (xi,yi,zi);

[0162] M represents a movable path planning point sequence (Move Points), which is defined by n vertices (xi,yi,zi) i=0,…,n;

[0163] X represents the axis position (Axis), which is defined by two vertices (xi,yi,zi) i=0,1;

[0164] S2: In the present invention, it is necessary to generate a large category-independent VLA model. In view of the fact that training a VLA model from scratch requires huge data and computing resources, the present invention chooses a different path to develop a VLA model. The strategy adopted is to fine-tune the existing MLLM. In order to achieve this goal, it is necessary to construct instructions that follow the feature set of articulated objects. The feature set is consistent with the format of the MLLM fine-tuning dataset. The input is image and text instructions, and the output is formatted text, that is, the Action Affordance of the articulated object; this section will discuss the specific method of constructing such an instruction tracking dataset. As previously discussed in step one, A={B,D,F,M,X} is used to represent the Action Affordance of the articulated object. In actual operation, submitting all the information to the MLLM at once and asking it to give the correct answer may not be satisfactory, because the MLLM will ignore some important information. Therefore, if Figure 4 As shown, in order to enable the fine-tuned VLA large model to fully understand the Action Affordance information of the articulated object, the present invention decomposes an instruction into six different sub-task feature sets.

[0165] S201: Object Category Identification (OCI): Figure 4 As shown in (a), the VLA model is guided to output the name of the specific object to be operated according to the given picture and user instructions; OCI is an important task in the field of machine vision, which involves enabling the machine to recognize and classify different objects in the image. In order to effectively manipulate objects, the VLA model must first identify the type of object it is facing, because objects of the same type usually have similar geometric features. This recognition ability is crucial for robot manipulation tasks because it directly affects the grasping strategy and the execution of the action. In the dataset of the present invention, a total of 62 common object types are collected, covering multiple fields from daily necessities to office supplies.

[0166] S202: Border detection (Position Detection): Figure 4 As shown in (b), it aims to guide the VLA model to accurately identify and mark the specific position of the target object in the three-dimensional space; to this end, a three-dimensional bounding box is used to mark the target object to ensure the accuracy of spatial positioning. The three-dimensional bounding box (BBOX) is represented by a set of eight vertex coordinates {(xi,yi,zi)}, i = 1,…, 8, each coordinate corresponding to a corner of the three-dimensional bounding box. In this representation, (xi,yi) is the two-dimensional projection position of vertex i on the image plane, that is, the pixel coordinate in the image. And zi represents the depth information to distinguish the relative spatial position of the object. The depth value zi is normalized using the maximum and minimum depth values ​​to scale its range to a standardized interval of (0,1).

[0167] S203: Axis Detection: Figure 4 (c) is used to guide the VLA model to accurately identify the type of rotation axis and the direction of movement of the target object. In three-dimensional space, the rotation axis of an object is the core feature of its movement, and through fine-tuning training of the model, it can accurately determine the type and direction of the axis. In this task, the axis type is represented by revolute (rotational joint) and prismatic (sliding joint). In order to define the direction of the rotation axis, the coordinates of two points {(xi,yi,zi)}, i=1,2, are used, that is, the two points on the axis determine the axis direction vector.

[0168] S204: Force Point Detection: Figure 4 As shown in (d), the VLA model is guided to accurately identify the optimal force point of the target object for efficient mechanical operation and control; when constructing the subtask feature set, the location of the optimal force point is crucial to the stability and force distribution of the object.

[0169] S204 The optimal force point P optimal The calculation process of the optimal force point of the target of the sliding joint type, that is, the geometric center of the exposed surface of the target object, is as follows:

[0170] S20401: Six faces defined by the object's 3D bounding box:

[0171] faces{F1,F2,...,F6},F i ∈R 4×3 (2)

[0172] In formula (2):

[0173] R represents a real number, R 4×3 Represents a two-dimensional real vector of size 4×3;

[0174] S20402: Calculate the geometric center point c of the 3D bounding box bbox and the geometric center c of each face i :

[0175]

[0176]

[0177] In formula (3) and (4):

[0178] p i Represents the three-dimensional vertices in the three-dimensional bounding box. Each three-dimensional bounding box consists of a total of 8 three-dimensional vertices p iSure;

[0179] p ij Indicates face F i The three-dimensional vertices in each face F i There are 4 three-dimensional vertices p ij Sure;

[0180] S20403: Through the geometric center point c of the 3D bounding box bbox and the geometric center point c of each face i The difference between the moving direction vector n of each face is calculated i :

[0181]

[0182] S20404: Calculate the unit vector d of the direction of motion based on any two points d1 and d2 on the axis of rotation unit :

[0183]

[0184] S20405: Calculate the moving direction vector n of each face i The unit vector d with the direction of motion unit dot_product i :

[0185] dot_product i =n i ·d unit (7)

[0186] S20406: dot product i The closer it is to 1, the closer it is to the unit vector d in the direction of motion. unit The more consistent, therefore, the face F with the largest dot product k The exposed surface:

[0187]

[0188] S20407: Calculate the face F with the largest dot product k The geometric center, that is, the best force point P optimal :

[0189]

[0190] In formula (9):

[0191] p kj The face F with the largest dot product k The three-dimensional vertices in each face F with the largest dot product k There are 4 three-dimensional vertices p kjSure.

[0192] The best force point of the target of the revolute joint type in the best force point described in S204 is far away from the axis of rotation because of its rotational freedom, which helps to improve the effect of torque and enhance the control efficiency. When constructing the feature set of this subtask, in order to optimize the selection of the force point of the revolute type target, the weight sampling method is used. The calculation process is as follows:

[0193] S20401: Different weights are assigned to different points on the target object according to their distance from the axis of rotation. During the calculation process, points that are farther from the axis of rotation are assigned a weight of 4, which is considered to have a greater impact on the object during the force application process; points close to the axis of rotation are assigned a lower weight of 1, which is considered to have a smaller contribution to the torque. This weighting method can effectively adjust the impact of different points on the calculation of the optimal force point, ensuring that the calculation results are more reasonable.

[0194] S20402: Calculate the best force point P optimal :

[0195]

[0196] In formula (10):

[0197] w i is the weight of the i-th point;

[0198] S205: Gripper Detection: Figure 4 As shown in (e), the guided VLA model can accurately identify the direction of the gripper, thereby improving the accuracy and efficiency of the robot arm's operation; in the actual articulated object manipulation experiment, the direction of the robot arm's gripper directly affects the object's grasping and force application method, which in turn determines the effect and stability of the operation. By identifying the correct direction of the gripper and combining the end point information, the robot can solve the torque of each joint through the inverse kinematics algorithm to complete precise object manipulation. For different types of target objects, the direction of the gripper is also defined differently.

[0199] For the sliding joint type target, since the gripper operates along the axial thrust of the target, the direction of the gripper is opposite to the direction of the rotation axis;

[0200] For targets with rotational joints, the orientation of the gripper is determined by the cross product of two vectors on the force-bearing surface of the target object. The advantage of this method is that it ensures that the force applied by the gripper acts precisely on the force-bearing surface of the object, thereby promoting the rotation of the object and maximizing the stability and success rate of task execution.

[0201] S206: Move planning: Figure 4 As shown in (f), the VLA model is guided to convert abstract text instructions into specific mechanical movements.

[0202] S20601: Generate a motion planning sequence consisting of multiple points based on the results of frame detection, shaft detection, force point detection, and gripper direction detection:

[0203] {(xi,yi,zi)}(11)

[0204] In formula (11):

[0205] i=1,…,4, represents the 1st to 4th points in the motion planning sequence, each of which contains the x, y, and z coordinates of the end of the robot in three-dimensional space, which accurately define the movement path of the end of the robot;

[0206] S20602: Calculate the unit vector v in the direction of the axis of rotation based on any two points d1 and d2 on the axis of rotation direction :

[0207]

[0208] In formula (12):

[0209] d1=(x1,y1,z1);

[0210] d2=(x2,y2,z2);

[0211] S20603: moving path planning;

[0212] For sliding joint type targets, calculate the sliding path p t :

[0213] p t =P optimal +t·speed·v direction ,t=0,1,2,3 (13)

[0214] In formula (13):

[0215] speed=0.1 indicates the speed of movement;

[0216] For revolute joint type targets,

[0217] S2060301: Calculate the rotation angle angle_per_second per second:

[0218] angle_per_second=speed×5 (14)

[0219] In formula (14):

[0220] speed=0.1 indicates the speed of movement;

[0221] S2060302: Calculate the corresponding rotation angle θ at each time point t:

[0222] θ=angle_per_second×t,t=0,1,...,3 (15)

[0223] S2060303: Calculate the rotation matrix R(v) by formula (15) direction ,θ):

[0224]

[0225] In formula (16):

[0226] u x ,u y ,u t Both represent the unit vector v in the direction of the rotation axis. direction The amount of

[0227] S2060304: The best force point P optimal Rotate and record the new best force point P' position:

[0228] P'=(P optimal -d1)·R(v direction ,θ)+d1 (17)

[0229] Through such a sequence, the robot arm can complete complex tasks by accurately performing each movement according to the predetermined path.

[0230] According to the above method, the present invention generates six fine-tuning data sets, totaling 1,249,654 data. Subsequently, the data is filtered to remove abnormal points and duplicate data. After filtering, the total amount of data is 310,548.

[0231] S3: Design VLA model architecture;

[0232] In the present invention, the VLA model is developed based on SPHINX-X. Figure 5As shown in FIG. 1 , unlike the Mixture Of Visual Experts (MOV) used in SPHINX-X, the present invention uses a Mixed Four Vision Encoder. The Mixed Four Vision Encoder integrates CLIP-VIT and CLIP-ConvNeXt visual encoders and DINOv2-VIT to extract local semantic features, while using Q-Former to summarize global visual features.

[0233] In order to overcome the challenge of low-resolution limitation of hybrid visual encoder and enable the VLA model to fully understand the global and local features of the input image, the following strategies are adopted:

[0234] S301: spatially dividing an input high-resolution image into a plurality of sub-images, and downsampling the high-resolution image into a low-resolution image;

[0235] S302: Multiple images are simultaneously input into the hybrid visual encoder, encoded and concatenated into a long sequence, and then input into the LLaMA2-13B model; among them, the first group of codes is responsible for global semantics, and the remaining four groups of codes record fine-grained local features; this design enables the VLA model to adaptively explore finer-grained visual semantics while maintaining coding efficiency.

[0236] S303: Design a projection layer to achieve spatial alignment of visual and language features. This design not only enhances the semantic understanding ability of the model, but also improves the accuracy of feature fusion.

[0237] S4: train the VLA model;

[0238] Use the feature set generated in S2 to train the VLA model in S3;

[0239] The command-following articulated object feature set format generated in S2 is the default visual question answering (VQA) dataset of the SPHINX framework. After modifying the configuration information, you can use the fine-tuning method VLA model provided by the SPHINX framework for fine-tuning. The present invention uses the SPHINX-v2-1k model obtained from the official resource library as the pre-training basis. Figure 5 As shown in Figure 2, the hybrid visual encoder is kept frozen throughout the fine-tuning stage, and only the projection layer and LLaMA2-13B are fine-tuned to keep the integrity of the pre-trained features.

[0240] S5: Construct a geometric thinking planner.

[0241] S501: Since the instructions designed in step 2 are coherent with the feature set of the articulated object, the subsequent reasoning task depends on the result of the previous reasoning task. In the process of using the VLA model for reasoning, the reasoning effect of the VLA model on the previous task will significantly affect the reasoning quality of the subsequent task. In order to improve the accuracy of the VLA model in reasoning the geometric center information of the articulated object and provide accurate three-dimensional manipulation waypoints based on this information, such as Figure 6 As shown, the present invention constructs a geometric thinking planner including an Advisor (instruction template) and a Strategist (scoring algorithm);

[0242] S502: In order to maximize the accuracy of the inferred geometric center information, the Advisor needs to let the VLA model list all the possible results it considers, and then the Strategist scores each result according to the specified scoring algorithm, and finally selects the best result for subsequent reasoning tasks.

[0243] S502 The Strategist scores and screens the answers to the object category recognition, including the following steps:

[0244] S50201: In order to score and screen the results of OCI, the present invention develops a scoring algorithm based on the contrastive language-image pre-training model (CLIP). The visual-semantic embedding model ViT-L / 14 is used to map images and OCI Possible Results into a shared embedding space. Specifically, the contrastive language-image pre-training model converts the input image and text candidates into high-dimensional vector representations through two independent encoders (an image encoder and a text encoder):

[0245] I=f image (x image ),T=f text (x text ) (18)

[0246] In formula (18):

[0247] x image represents the input image;

[0248] x text Represents text candidates for input;

[0249] f image An image encoder representing the CLIP model;

[0250] f text Represents the text encoder of the CLIP model;

[0251] I represents the image feature vector in the shared embedding space;

[0252] T represents the text feature vector in the shared embedding space;

[0253] S50202: Calculate the similarity between the image embedding and the text embedding, ensure that the similarity value is in the range [-1,1], and generate a score for the category prediction:

[0254] In this process, cosine similarity is used as an indicator to measure similarity.

[0255]

[0256] In formula (19):

[0257] I represents the dot product of the image embedding vector calculated for object category recognition;

[0258] T represents the dot product of the text embedding vector calculated for object category recognition;

[0259] ||I|| represents the Euclidean norm of the dot product of the image embedding vector calculated for object category recognition;

[0260] ||T|| represents the Euclidean norm of the dot product of the text embedding vector calculated for object category recognition;

[0261] S50203: Select the result with the highest similarity as the best object category recognition answer:

[0262] Best Prediction=argmax(Similarity(I,T)) (20).

[0263] This approach ensures the accuracy and objectivity of the scoring process, ultimately returning the text description that best matches the input image.

[0264] S502 The Strategist scores and screens the answers to the stress point detection, including the following steps:

[0265] The Strategist aims to select the best stress point of the BBOX from the points given by the Advisor. The distance from each point to each surface of the BBOX is calculated to select the point closest to the BBOX.

[0266] S50201: Define a plane by any three points P1, P2, P3 and calculate the normal vector of the plane

[0267]

[0268] In formula (21):

[0269]

[0270] S50202: According to the normal vector The components A, B, and C determine the equation of the plane:

[0271] Ax+By+Cz+D=0 (22)

[0272] S50203: Calculate D based on any point P1 (x0, y0, z0) on the surface:

[0273] D=-(A·x0+B·y0+C·z0) (23)

[0274] S50204: Calculate the distance from the target point P(x,y,x) to the surface:

[0275]

[0276] S50205: Calculate each point P in the given point set in turn i The distances to the six faces of the 3D bounding box;

[0277] S50206: Calculate the total score of all points (the cumulative value of all face distances) S i :

[0278]

[0279] In formula (25):

[0280] d i,j Indicates point P i To any face of the 3D bounding box F j The distance, F j is F1, F2, ..., F6;

[0281] S50207: The score reflects the overall closeness between the point and the bounding box. The lower the score, the higher the contact between the point and the bounding box. Finally, the total score S of all points is i Sort and select the point with the lowest score as the optimal point:

[0282]

[0283] S502 The Strategist scores and screens the answers to the pivot detection, including the following steps:

[0284] The purpose of this Strategist is to evaluate the alignment between a set of gripper orientations and a target axis and select the optimal grasping orientation based on the alignment.

[0285] S50201: Calculate the cosine value of the angle between each grasping direction vector and the target axis through the input grasping direction list and the target axis, so as to reflect the degree of alignment between the directions. Specifically, given the target axis vector axis and a grasping direction vector g i , calculate the cosine similarity between the two:

[0286]

[0287] In formula (27):

[0288] axis represents the three-dimensional vector of the target axis;

[0289] g i A three-dimensional vector representing the grasping direction;

[0290] ||axis|| represents the Euclidean norm of the three-dimensional vector of the target axis;

[0291] ||g i ||Euclidean norm of the three-dimensional vector representing the grasping direction;

[0292] S50202: Invalid vectors (such as vectors with zero length) are filtered out during calculation, and the cosine value and its corresponding grasping direction index are stored. The grasping direction is selected according to the different joint types:

[0293] For sliding joint type targets, select the grasping direction that is most aligned with the target direction;

[0294] For revolute joint type targets, choose the orientation that is least aligned with the target orientation.

[0295] Example 1

[0296] In order to measure the performance of the present invention, the articulated object in PartNet-Mobility and the Franka Panda Robot gripper are placed in the Sapien simulator, and then the position and direction of the Franka Panda Robot gripper are set according to the optimal force point and gripper direction inferred by the VLA model.

[0297] Evaluation criteria. This experiment aims to test the effectiveness of the model's interaction with the environment. In each task, an object is selected from the PartNet-Mobility dataset. The gripper is then guided to interact with the articulated object. The base of the object remains stationary, and the gripper must identify the movable part and master the correct actuation direction.

[0298] Comparative analysis. The success rates of gripper operations in 18 training categories (used in the training phase) and 9 test categories (not used in the training phase) were compared. The proposed method was compared with the following six baseline methods:

[0299] (1) Where2Act: This method processes point cloud data, evaluates the importance of each point by scoring it, and selects the point with the highest score as the contact point. At the same time, it also predicts the possible orientation of the end effector and selects the best orientation to determine the contact posture.

[0300] (2) UMPNet: Similar to Where2Act, UMPNet ensures that the end effector is oriented perpendicular to the object surface based on the predicted contact points.

[0301] (3) Flowbot3D: This method identifies the direction of motion in the point cloud, called “flow”, and selects the point with the largest flow to determine the interaction point. The direction of the flow determines the orientation of the end effector.

[0302] (4) Implicit3D: This method applies the Transporter network to identify key points on three-dimensional articulated objects to develop strategies for subsequent manipulation tasks. These key points help plan the pose of the end effector.

[0303] (5) ManipLLM: ManipLLM uses a visual-language model, combined with RGBD images and language cues, to predict the contact points and forward direction of the suction manipulator.

[0304] (6) A3VLM: This method is object-centric and uses a vision-language model to detect the object’s 3D bounding box, axis, joint type, and link name to guide the manipulation process.

[0305] result. Figure 7 The performance data of the present invention and six control groups were compared. Analyzing the results in the table, it can be found that in most object classifications, the performance of the present invention exceeds that of all other control groups. This performance improvement is mainly attributed to two key factors: one is the accurate recognition of the geometric center information of the object by the VLA, and the other is the accurate decision-making of the Advisor-Strategist in the geometric thinking planner. These methods support the high efficiency of the present invention in processing complex articulated objects.

[0306] Example 2

[0307] A task decomposition strategy was implemented in S2 to allow the VLA model to understand and absorb key information more deeply. Specifically, the user instructions were broken down into six unique subtasks, collectively referred to as T6. By fine-tuning the VLA model individually for these subtasks, the model's performance was optimized. To verify the effectiveness of this task decomposition approach, the three subtasks of object category identification (OCI), point detection, and gripper detection were removed from T6. After removing these subtasks, the VLA model was used to reason about the movement path planning of an articulated object, and the relevant results are shown in Figure 8 middle.

[0308] The analysis results show that after removing the OCI subtask, the VLA model encountered serious problems in identifying the BBOX, which led to errors in the identification of all related Action Affordances. When the Gripper Detection subtask was removed, the VLA model made directional errors when identifying the movement path. When the Force Point Detection subtask was removed, the VLA model made errors in identifying the starting point of the movement path, which led to deviations in the entire path. These performance degradations emphasize the importance of properly dividing subtasks, which is crucial to ensure that the model can accurately understand and process complex information.

[0309] Example 2 Results Figure 8 The figure shows the inference results obtained by fine-tuning the VLA model under different feature sets. (a) T6 shows the inference results after fine-tuning the VLA model using the full feature set. (b) T6-OCI, (c) T6-GripperDetection, and (d) T6-Force Point Detection respectively show the results obtained by fine-tuning the VLA model after removing the three subtasks of OCI, Gripper Detection, and Force Point Detection.

[0310] Example 3

[0311] The development process of the geometric thinking planner is described in detail in S5. This method aims to improve the accuracy of the VLA's understanding of the geometric center information of the articulated objects through the Advisor and Strategist, and provide accurate 3D manipulation waypoints based on this information. The goal of this section is to verify the effectiveness of the geometric thinking planner. We removed the geometric thinking planner and instead used the thinking chain prompt method to reason. This method integrates the answer obtained in the previous task into the question of the subsequent task and uses this question to ask the VLA model.

[0312] When designing the experiment, the results of the geometric thinking planner (GTP) were used as the control group, and four experimental groups were set up to compare the effect of using COT to replace GTP in different reasoning stages: the GTP-w / o OCI group replaced GTP with OCI only in the OCI stage, the GTP-w / o Point group replaced GTP with OCI only in the point detection stage, the GTP-w / o Direction group replaced GTP with OCI only in the direction detection stage, and the COT group replaced GTP in the three stages of OCI, point detection, and direction detection. This experimental design aims to evaluate the performance impact of COT on GTP replacement at each stage.

[0313] In terms of results, Fig. 9 Performance data comparing the use of COT and the control group using GTP at each stage are provided. The success rate of the reasoning results of the geometric thinking planner and COT are evaluated in the seen and unseen categories. The weighted average of the seen and unseen categories is used as the criterion for measuring performance, with the weight being the number of articulated objects of each type. The analysis results show that the performance of VLA reasoning is significantly better when the geometric thinking planner is used than when only COT is used. This performance advantage can be attributed to the fact that the Strategist in the geometric thinking planner can revisit and select the optimal solution, thereby reducing the possibility of decision errors and improving the accuracy of planning. This shows that the geometric thinking planner has significant advantages in processing complex geometric information and providing accurate three-dimensional manipulation waypoints.

[0314] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations within the meaning and range of equivalents of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0315] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A method for constructing a VLA large model for articulated object manipulation, characterized in that: The method comprises the following steps: S1: Using geometric center representation to describe the action affordance of articulated objects: A={B,D,F,M,X} (1) In formula (1): B represents the 3D bounding box of the articulated object; D represents the direction of the end effector of the mechanical gripper; F represents the best force point; M represents a movable path planning point sequence; X represents the position of the rotating shaft; S2: Construct instructions that follow the feature set of articulated objects. The feature set is in the same format as the MLLM fine-tuning dataset. The input is image and text instructions, and the output is formatted text, i.e., the Action Affordance of the articulated object. Decompose an instruction into six different subtask feature sets. S3: Design VLA model architecture; S4: train the VLA model; Use the feature set generated in S2 to train the VLA model in S3; S5: Construct a geometric thinking planner.

2. A method for constructing a VLA large model for articulated object manipulation according to claim 1, characterized in that: The subtask feature set S2 includes: S201: Object category recognition: guide the VLA model to output the name of the specific object to be operated according to the given picture and user instructions; S202: Bounding box detection: aims to guide the VLA model to accurately identify and mark the specific location of the target object in three-dimensional space; S203: Rotation axis detection: aims to guide the VLA model to accurately identify the rotation axis type and movement direction of the target object; S204: Force point detection: guide the VLA model to accurately identify the best force point of the target object to perform efficient mechanical operation and control; S205: Gripper direction detection: guiding the VLA model to accurately identify the direction of the gripper, thereby improving the accuracy and efficiency of the robot arm operation; For the sliding joint type target, since the gripper operates along the axial thrust of the target, the direction of the gripper is opposite to the direction of the rotation axis; For targets of the revolute joint type, the orientation of the gripper is determined by the cross product of two vectors on the force-bearing surface of the target object; S206: Mobile path planning: guiding the VLA model to convert abstract text instructions into specific mechanical movements.

3. A method for constructing a VLA large model for articulated object manipulation according to claim 2, characterized in that: S204 The optimal force point P optimal The calculation process of the optimal force point of the target of the sliding joint type, that is, the geometric center of the exposed surface of the target object, is as follows: S20401: Six faces defined by the object's 3D bounding box: faces{F1,F2,...,F6},F i ∈R 4×3 (2) In formula (2): R represents a real number, R 4×3 Represents a two-dimensional real vector of size 4×3; S20402: Calculate the geometric center point c of the 3D bounding box bbox and the geometric center c of each face i : In formula (3) and (4): p i Represents the three-dimensional vertices in the three-dimensional bounding box. Each three-dimensional bounding box consists of a total of 8 three-dimensional vertices p i Sure; p ij Indicates face F i The three-dimensional vertices in each face F i There are 4 three-dimensional vertices p ij Sure; S20403: Through the geometric center point c of the 3D bounding box bbox and the geometric center point c of each face i The difference between the moving direction vector n of each face is calculated i : S20404: Calculate the unit vector d of the direction of motion based on any two points d1 and d2 on the axis of rotation unit : S20405: Calculate the moving direction vector n of each face i The unit vector d with the direction of motion unit dot_product i : dot_product i =n i ·d unit (7) S20406: dot product i The closer it is to 1, the closer it is to the unit vector d in the direction of motion. unit The more consistent, therefore, the face F with the largest dot product k The exposed surface: S20407: Calculate the face F with the largest dot product k The geometric center, that is, the best force point P optimal : In formula (9): p kj The face F with the largest dot product k The three-dimensional vertices in each face F with the largest dot product k There are 4 three-dimensional vertices p kj Sure.

4. A method for constructing a VLA large model for articulated object manipulation according to claim 2, characterized in that: The best force point of the target of the rotation joint type in the best force point in S204 is at a position far away from the rotation axis because it has rotational freedom. The calculation process is as follows: S20401: assigning different weights to different points on the target object according to their distance from the rotation axis; S20402: Calculate the best force point P optimal : In formula (10): w i is the weight of the ith point.

5. A method for constructing a VLA large model for articulated object manipulation according to claim 3 or 4, characterized in that: The S206 comprises the following steps: S20601: Generate a motion planning sequence consisting of multiple points based on the results of frame detection, shaft detection, force point detection, and gripper direction detection: {(x i ,y i ,z i )} (11)In formula (11): i=1,…,4, represents the 1st to 4th points in the motion planning sequence, each of which contains the x, y, and z coordinates of the end of the robot in three-dimensional space, and the coordinates define the movement path of the end of the robot; S20602: Calculate the unit vector v in the direction of the axis of rotation based on any two points d1 and d2 on the axis of rotation direction : In formula (12): d1=(x1,y1,z1); d2=(x2,y2,z2); S20603: moving path planning; For a sliding joint type target, calculate the sliding path pt: p t =P optimal +t·speed·v direction ,t=0,1,2,3 (13) In formula (13): speed=0.1 indicates the speed of movement; For revolute joint type targets, S2060301: Calculate the rotation angle angle_per_second per second: angle_per_second=speed×5 (14) In formula (14): speed=0.1 indicates the speed of movement; S2060302: Calculate the corresponding rotation angle θ at each time point t: θ=angle_per_second×t, t=0, 1,..., 3 (15) S2060303: Calculate the rotation matrix R(v) by formula (15) direction ,θ): In formula (16): u x ,u y ,u z Both represent the unit vector v in the direction of the rotation axis. direction The weight of S2060304: The best force point P optimal Rotate and record the new best force point P′ position: P′=(P optimal -d1)·R(v direction ,θ)+d1 (17).

6. A method for constructing a VLA large model for articulated object manipulation according to claim 5, characterized in that: The S3 comprises the following steps: S301: spatially dividing an input high-resolution image into a plurality of sub-images, and downsampling the high-resolution image into a low-resolution image; S302: Multiple images are simultaneously input into the hybrid visual encoder, encoded and concatenated into a long sequence, and then input into the LLaMA2-13B model; the first group of codes is responsible for global semantics, and the remaining four groups of codes record fine-grained local features; S303: Design a projection layer to achieve spatial alignment of visual and language features.

7. A method for constructing a VLA large model for articulated object manipulation according to claim 6, characterized in that: The S5 comprises the following steps: S501: Build a geometric thinking planner including Advisor and Strategist; S502: Advisor lets the VLA model list all the results, and then the Strategist scores each result according to the specified scoring algorithm, and finally selects the best result for subsequent reasoning tasks.

8. A method for constructing a VLA large model for articulated object manipulation according to claim 7, characterized in that: S502 The Strategist scores and screens the answers to the object category recognition, including the following steps: S50201: The contrastive language-image pre-training model converts the input image and text candidates into high-dimensional vector representations through two independent encoders: I=f image (x image ),T=f text (x text ) (18) In formula (18): x image represents the input image; x text Represents text candidates for input; f image An image encoder representing the CLIP model; f text Represents the text encoder of the CLIP model; I represents the image feature vector in the shared embedding space; T represents the text feature vector in the shared embedding space; S50202: Calculate the similarity between the image embedding and the text embedding, ensure that the similarity value is in the range [-1,1], and generate a score for the category prediction: In formula (19): I represents the dot product of the image embedding vector calculated for object category recognition; T represents the dot product of the text embedding vector calculated for object category recognition; ||I|| represents the Euclidean norm of the dot product of the image embedding vector calculated for object category recognition; ||T|| represents the Euclidean norm of the dot product of the text embedding vector calculated for object category recognition; S50203: Select the result with the highest similarity as the best object category recognition answer: Best Prediction=argmax(Similarity(I,T)) (20).

9. The method for constructing a VLA large model for articulated object manipulation according to claim 7, characterized in that: S502 The Strategist scores and screens the answers to the stress point detection, including the following steps: S50201: Define a plane by any three points P1, P2, P3 and calculate the normal vector of the plane In formula (21): S50202: According to the normal vector The components A, B, and C determine the equation of the plane: Ax+By+Cz+D=0 (22) S50203: Calculate D based on any point P1 (x0, y0, z0) on the surface: D=-(A·x0+B·y0+C·z0) (23) S50204: Calculate the distance from the target point P(x,y,x) to the surface: S50205: Calculate each point P in the given point set in turn i The distances to the six faces of the 3D bounding box; S50206: Calculate the total score S of all points i : In formula (25): d i,j Indicates point P i To any face of the 3D bounding box F j The distance, F j is F1, F2, ..., F6; S50207: Total score S for all points i Sort and select the point with the lowest score as the optimal point:

10. The method for constructing a VLA large model for articulated object manipulation according to claim 7, characterized in that: S502 The Strategist scores and screens the answers to the pivot detection, including the following steps: S50201: Given a target axis vector axis and a grasping direction vector g i , calculate the cosine similarity between the two: In formula (27): In formula (27): axis represents the three-dimensional vector of the target axis; g i A three-dimensional vector representing the grasping direction; ||axis|| represents the Euclidean norm of the three-dimensional vector of the target axis; ||g i ||Euclidean norm of the three-dimensional vector representing the grasping direction; S50202: Select the grasping direction according to the different joint types: For sliding joint type targets, select the grasping direction that is most aligned with the target direction; For revolute joint type targets, choose the orientation that is least aligned with the target orientation.

Images