VLA large model construction method with efficient reasoning speed for robot control

Abstract

The invention discloses a method for constructing a VLA large model with an efficient reasoning speed for robot control, and the method is characterized in that a VLA model named as a visual language motion model ERMVLA with the efficient reasoning speed is designed, the ERMVLA takes geometric center data as a driving basis, the ERMVLA is specially designed according to the control characteristics of a hinged object, and the control characteristics of the hinged object are optimized. The tail end movement track of the mechanical arm can be precisely planned based on the current scene. Compared with a traditional scheme, the core breakthrough of the ERMVLA is to realize full-link fusion from a sensing layer to a motion control layer. Through the tail end path data generated by the ERMVLA, the mechanical arm can complete movement in a real environment by means of a spline interpolation instruction, so that diversified operation tasks are efficiently completed. In addition, a specific efficient reasoning mechanism of the model further improves the universal adaptation capability of the VLA framework, so that the VLA framework has remarkable advantages in a complex control scene.

Description

Technical Field

[0001] This invention belongs to the field of embodied intelligence and relates to a robot vision-language-action (VLA) model, specifically a method for constructing a large VLA model with high inference speed for robot control. Background Technology

[0002] The fundamental goal of robot manipulation is to enable models to understand visual scenes and perform actions accordingly. Although existing robot vision-language-action (VLA) models can handle a range of basic tasks, they still face two challenges in complex task environments: (1) limited reasoning ability, making it difficult to accurately understand and handle complex scenes; and (2) high computational costs for model training and inference, especially in resource-constrained robot systems. Among these, flexible manipulation of articulated objects is particularly crucial for the development of intelligent multi-purpose robots. These objects exhibit significant differences in motion characteristics, thus requiring specialized manipulation techniques for different objects. Meanwhile, robot manipulation tasks are essentially about interacting with objects in a dynamic environment, demanding not only human-level semantic understanding and reasoning capabilities but also accurate low-level action prediction capabilities. While existing large multimodal models (MLLMs) can handle some basic tasks, they still exhibit insufficient reasoning ability in machine scenarios, especially when encountering tasks involving complex logic and environmental understanding, where their performance drops significantly. Furthermore, attention-based MLLMs incur high computational costs during training and inference, severely limiting their application in actual robot manipulation. Therefore, developing a vision-language-action model that has high inference speed and is computationally friendly is a key challenge for achieving efficient robot control and promoting the development of embodied intelligence. Summary of the Invention

[0003] To address the problems of slow inference speed, high training difficulty, insufficient generalization ability, and high training cost in existing VLA large models in the field of robot control, this invention provides a method for constructing VLA large models with high inference speed for robot control, specifically addressing the need for intelligent control robots to quickly and accurately manipulate articulated objects.

[0004] The objective of this invention is achieved through the following technical solution:

[0005] A method for constructing a large VLA model with high inference speed for robot control includes the following steps:

[0006] Step 1: Fine-grained geometric modeling of the hinged object in a simulation environment:

[0007] In a virtual environment, the physical properties of articulated objects are combined to model them, thereby systematically extracting and labeling six key feature information: object category, bounding box, axis of rotation, contact point, gripping direction, and movement path, comprehensively describing the maneuverability of articulated objects.

[0008] Step 2: Construct the training dataset:

[0009] Based on the six key feature information extracted in step one, six training labels are generated in the virtual simulation environment: object category recognition, bounding box recognition, pivot recognition, contact point recognition, grasping direction recognition, and movement path planning, thereby realizing efficient learning and generalization of the VLA model driven by multi-dimensional supervision signals.

[0010] Step 3: Design the VLA model architecture:

[0011] The VLA model architecture is designed based on a pre-trained large-scale language model and a visual encoder. The visual encoder extracts visual features from the input image and connects the visual encoder and the language model through a multilayer perceptron, mapping the visual features to the language embedding space. The text command is encoded by pre-trained word vectors and concatenated with the visual features before being input into the Mamba language model, which outputs a response in natural language form.

[0012] Step 4: Train the VLA model:

[0013] The VLA model architecture designed in step three is jointly trained using the various training tasks constructed in step two. A loss function is designed for the output features of each task. By weighted summation of the loss functions of all tasks, a unified multi-task joint loss function is constructed as the global optimization objective during the model training phase.

[0014] The following explains the technical terms used in this invention:

[0015] Articulated objects: Articulated objects are composed of multiple rigid components connected by joints, each of which can rotate or translate around the joints. This type is commonly used in the connection of furniture, doors, windows, and other devices. Compared to ordinary rigid objects with only 6 degrees of freedom (DoF), articulated objects have a higher degree of freedom, making them more difficult for robots to understand and interact with. Articulated objects are generally divided into primamatic and revolute types. Prismatic articulated objects can only translate along the axis of rotation, while revolute articulated objects can only rotate around the axis of rotation.

[0016] Mamba Architecture: Mamba is a novel sequence modeling architecture based on the State-Space Model (SSM), possessing powerful context modeling capabilities and linear computational complexity. Unlike the traditional Transformer architecture based on attention mechanisms, Mamba performs recursive computation on the input sequence through a parameterized state-space kernel. This significantly reduces computational resource consumption while preserving long-range dependency information, making it particularly suitable for real-time systems such as robots that require efficient inference and low-latency response. Due to its inherent support for streaming processing and low memory overhead, the Mamba architecture shows broad application prospects in embodied intelligence tasks such as Vision-Language-Motion (VLA).

[0017] VLA (Vision-Language-Motion) is a multimodal artificial intelligence model that integrates visual perception, language understanding, and action generation. It can handle images, text commands, and operational tasks in complex scenes, forming a closed loop from perception to decision-making to execution. For example, a VLA model can understand natural language commands provided by the user (such as "grab the red cup on the table"), analyze visual information (such as the position and features of objects in the scene), and generate corresponding operational plans (such as the grasping path of a robotic arm). This model is widely used in fields such as robot control and industrial automation, achieving natural interaction and efficient task execution through a unified multimodal processing framework.

[0018] Compared with the prior art, the present invention has the following advantages:

[0019] This invention designs a VLA model called Efficient Reasoning for Manipulation Visual Language Action (ERMVLA). For example... Figure 1 As shown, ERMVLA, a high-performance robot VLA model, focuses on scenarios involving precise manipulation of articulated objects. Driven by geometric center data, this model is specifically designed for the manipulation characteristics of articulated objects, enabling precise planning of the robotic arm's end-effector trajectory based on the current scene. Compared to traditional solutions, ERMVLA's core breakthrough lies in achieving end-to-end fusion from the perception layer to the motion control layer. Using the end-effector path data generated by ERMVLA, the robotic arm can complete movements in the real environment using spline interpolation instructions, thus efficiently completing diverse operational tasks. Furthermore, the model's unique and efficient inference mechanism further enhances the VLA framework's versatility, demonstrating significant advantages in complex manipulation scenarios. Attached Figure Description

[0020] Figure 1 Modeling and characterizing the physical properties of articulated objects;

[0021] Figure 2 Training task and VLA model architecture diagram;

[0022] Figure 3 This is a process diagram of Example 2. Detailed Implementation

[0023] The technical solution of the present invention will be further described below with reference to the accompanying drawings, but it is not limited thereto. Any modifications or equivalent substitutions to the technical solution of the present invention that do not depart from the spirit and scope of the technical solution of the present invention should be covered within the protection scope of the present invention.

[0024] This invention provides a method for constructing a large VLA model with high inference speed for robot control. The method introduces two innovative approaches in constructing the large VLA model. First, it models the features of articulated objects based on their physical properties in a virtual environment, enabling the capture of geometric information of the articulated objects in a fine-grained manner. Second, it constructs a lightweight VLA model based on the Mamba architecture, reducing training costs and achieving high inference speed.

[0025] Input: URDF file of the hinged object.

[0026] Output: A lightweight VLA model that reduces training costs and has high inference speed, capable of inferring the end effector movement path of the robotic arm based on visual information and user commands.

[0027] The specific steps are as follows:

[0028] Step 1: Fine-grained geometric modeling of the hinged object in a simulation environment:

[0029] Existing methods for representing articulated objects typically model them based on a single instance. However, this approach often falls short in terms of expressive power and comprehensiveness when dealing with complex articulated objects containing multiple maneuverable parts. Because such objects exhibit high diversity in their internal structures, traditional single-instance, single-label modeling methods struggle to fully reflect their operational characteristics. For example, Figure 1 The illustrated "kettle" comprises two independent movable components: a "lid" and a "handle." Traditional methods often categorize the entire object as a single "kettle" instance and assign it a uniform label, ignoring the interactive potential of its internal components as independent hinged units. To overcome these shortcomings, this invention proposes a method for refining modeling by integrating the physical characteristics of hinged objects within a virtual environment. This enables accurate representation of multi-component hinged structures and enhances the model's adaptability to complex manipulation tasks.

[0030] like Figure 1As shown, this invention places the articulated object in a simulated environment for rendering, and uses the key object features of the articulated object as the core reference point to extract six core features: object category, bounding box, rotation axis, contact point, gripping direction, and movement path.

[0031] Step 2: Construct the training dataset:

[0032] This invention aims to construct a large-scale vision-language-action (VLA) model for robot manipulation of articulated objects. To achieve this goal, a series of training tasks need to be designed, with inputs including image information and text commands, and outputs covering the geometric features related to the articulated object and its motion trajectory planning. To enhance the model's understanding of articulated structure instances, this invention generates six types of training labels in a virtual simulation environment based on the key object characteristics extracted in step one:

[0033] Object Category Recognition (OCI): This label guides the VLA model to output the specific object category to be operated on based on the given image and user instructions. This invention uses the ViT-L / 14 visual-semantic embedding model from CLIP to integrate the input image and text candidate... Each vector is mapped to a shared embedding space and transformed into a high-dimensional vector. Cosine similarity is calculated, and the class with the highest similarity is selected as the label for the OCI task.

[0034] Bounding box recognition: This label guides the VLA model to use 3D bounding boxes. Mark the position of the target object to ensure accurate spatial positioning. 3D bounding box. It is through the coordinates of the eight vertices. Representation. To calculate the bounding box. First, represent the three-dimensional center point in the camera coordinate system. and size , These represent the size components of the object along the x, y, and z axes in three-dimensional space, respectively. Then, the three-dimensional center point is used... and Perform addition and subtraction operations on the values ​​to calculate the result. :

[0035] (1)

[0036] Axis Recognition: This label guides the VLA model to accurately identify the axis position of the target object. Axis Position Through the coordinates of the two vertices This indicates that they correspond to the two endpoints of the rotating shaft, respectively.

[0037] For Prismatic type targets, the axis position The starting point and ending point and the formula minus The value is related to the pivot position. For Revolute type targets, the pivot position... The starting point, the ending point, and the center point minus The value is related to:

[0038] (2)

[0039] Contact point identification: This label guides the VLA model to accurately identify the optimal contact point of the target object. For primamatic targets, the optimal contact point... It is usually located at the geometric center of the exposed surface of the target object. First, the three-dimensional bounding box is calculated based on formula (1). The six sides :

[0040] (3)

[0041] Then calculate each face. direction of movement and the direction of rotation of the shaft For each face Its geometric center .use center point and the geometric center point of each face Calculate the movement direction vector for each face. :

[0042] (4)

[0043] According to the position of the pivot in formula (2) two points and Calculate the direction of the shaft's motion. :

[0044] (5)

[0045] According to each face Relative to the center point motion direction vector Direction of rotation of the axis Choose the center point of the surfaces with the greatest similarity between the two as the optimal contact point. .

[0046] Optimal contact point for Revolute type objects Use each weight For formula (1) Perform weighted sampling calculations:

[0047] (6)

[0048] Grasping Direction Recognition: This label guides the VLA model to accurately identify the grasping direction. By identifying the correct grasping direction and combining it with end-effector position information, the robot can use inverse kinematics algorithms to calculate the torque of each joint, achieving precise control.

[0049] For Prismatic type targets, the grasping direction is the same as the axis of motion. Conversely, ensure the gripper operates along the axial thrust direction. For revolute type objects, the gripping direction should be perpendicular to the object surface to ensure the applied force acts on the correct contact surface. Gripping direction for revolute type objects. Formula (6) can be used as well as Construct a vertical vector to represent the grasping direction. :

[0050] (7)

[0051] Path planning: This label guides the VLA model to translate abstract text instructions into concrete mechanical movements, ensuring that the robotic arm's end effector executes operations along the optimal path. This invention generates a sequence of waypoints composed of multiple points based on the results of bounding box recognition, axis recognition, contact point recognition, and gripping direction recognition. .

[0052] For Prismatic type targets, based on the optimal contact point and the direction of rotation of the shaft Calculate the motion waypoint sequence :

[0053] (8)

[0054] Where c and t represent the movement speed and time step, respectively.

[0055] For Revolute type targets, records The rotation matrix is ​​calculated using Rodrigues' rotation formula. Ultimately, the optimal contact point will be determined. Rotate and calculate the motion waypoint sequence. :

[0056] (9)

[0057] Step 3: Design the VLA model architecture:

[0058] This invention designs a robot VLA model architecture with visual understanding and manipulation capabilities based on a pre-trained large language model (LLM) and a CLIP visual encoder. For example... Figure 2 As shown, the visual encoder obtains data from the input image. Extracting visual features Unlike methods that integrate feature extraction using multiple visual backbones (such as DINOv2, CLIP-ConvNeXt, CLIP-ViT), this invention selects a single visual encoder, avoiding additional computational overhead and improving the actual deployment efficiency of the system. To achieve cross-modal fusion of vision and language, a multilayer perceptron (MLP) is used to connect the visual encoder and the language model, mapping image features to the language embedding space. This avoids the high computational cost associated with integrating multiple visual models, significantly improving its practical feasibility in robotic applications. Given the real-time response requirements of robotic systems, this invention selects the Mamba language model, which possesses excellent context modeling capabilities and linear computational complexity, effectively meeting the robot's need for rapid response and making it suitable for various task scenarios. Text instructions are encoded using pre-trained word vectors. The data is then concatenated with visual features and input into the Mamba language model to achieve unified modeling of multimodal information, and finally, a natural language response is generated through decoding.

[0059] Step 4: Train the VLA model:

[0060] The Visual-Language-Action (VLA) model proposed in Step 3 is jointly trained using the various training tasks constructed in Step 2. Since the geometric information corresponding to different training tasks has its own unique physical semantics, such as object category, bounding box, axis of rotation, gripping direction, contact point, and movement path, exhibiting significant heterogeneity, this invention designs a loss function tailored to the physical properties of the output features of each task to ensure good performance of ERMVLA in inference across the six tasks.

[0061] For object category recognition tasks, the model's output is a natural language question; this invention uses cross-entropy loss. As the target of optimization:

[0062] (10)

[0063] In the formula, Indicates multimodal input, For the first Target words at each time step This allows the model to predict the probability of the word given the current context.

[0064] For bounding box recognition tasks, the bounding box output by ERMVLA is a structure composed of eight spatial points. This invention uses the predicted values ​​of these eight points... Compared with the true value The Euclidean distance between them is used as the optimization objective. Loss function The definition is as follows:

[0065] (11)

[0066] For axis recognition tasks, ERMVLA outputs the predicted axis as two spatial points. To simultaneously optimize the axis's orientation and spatial position, the loss function includes orientation consistency loss and position error terms. The orientation loss measures the consistency of the angle between the predicted axis and the true axis using the unit vector dot product, while the position loss measures the axis's position error using the L2 distance between the two points. Loss Function as follows:

[0067] (12)

[0068] In the formula, and These are the weights for the direction loss and the position loss, respectively.

[0069] In the contact point recognition task, Euclidean distance is used as the loss function to optimize the spatial accuracy of the model's output contact points. Specifically, the contact point prediction results... With real contact points Error between Defined as:

[0070] (13)

[0071] To achieve accurate prediction of the grasping direction, a loss function based on the angle between the direction vectors is employed. The model outputs a grasping direction composed of two spatial points, which are normalized using unit vectors and then compared with the true direction vector using cosine similarity calculation. This is the loss function for grasping direction recognition. for:

[0072] (14)

[0073] For the task of planning a movement path, the model will infer the planned path. This set is used to describe the trajectory of an object during operation. It is used to measure the model's predicted path. With the real path The difference between them defines the path loss function. for:

[0074] (15)

[0075] They represent and The three-dimensional coordinates of the points.

[0076] Finally, the loss function during model training. for:

[0077] (16)

[0078] The training dataset constructed in step two contains 30 different object categories. This invention selects 10 of these categories for ERMVLA class-independent capability evaluation, referred to as "unseen categories," while the remaining 20 categories are used to train the ERMVLA model, referred to as "seen categories." Figure 2 As shown, throughout the entire training phase, the hybrid visual encoder remains frozen, using only the loss function. The projection layer and Mamba-2.8B are trained to achieve the overall optimization objective, while maintaining the integrity of the pre-trained features. The batch size for training is 2, and the learning rate is set to [value missing]. In this invention, the ERMVLA model was trained on eight A800 processors (each equipped with 80GB of memory) for 16 hours.

[0079] Example 1

[0080] Evaluation Criteria: To measure the practical performance of ERMVLA, this invention references the evaluation method in MainpLLM. The articulated object from the training task and the Franka Panda Robot gripper are placed in the Sapien simulator, and their poses are set based on the waypoints and gripping directions inferred by ERMVLA. This experiment aims to test the model's ability to interact with the environment. In each task, an object is selected and placed in the simulator, and its initial joint states are recorded. Subsequently, the gripper interacts with the object, recording the final joint states and calculating the changes. If the changes exceed a threshold... If the condition is met, the operation is considered successful. The threshold is consistent with ManipLLM. At that time, it was considered that the operation was successful.

[0081] Comparative Analysis: This invention compares the success rate of ERMVLA in 20 seen categories and 10 unseen categories. This invention is compared with six baseline methods: Where2Act: Manipulates articulated objects by processing point cloud data to predict operable information for each pixel. It uses a neural network to map pixels to six basic actions. UMPNet: Similar to Where2Act, it ensures the end effector's orientation is perpendicular to the object surface when predicting contact force points. Flowbot3D: Predicts potential movement directions by representing 3D articulated flows and deploys an analytical motion planner to determine articulation strategies. Implicit3D: Employs a transmitter network to identify contact force points on articulated objects. This identification helps guide the end effector's posture when performing manipulation tasks by utilizing hybrid 3D representations, cross-attention, and implicit reconstruction mechanisms. ManipLLM: Simulates manipulation by training a visual language model, combining object category understanding, force inference, and object-centric posture prediction. A3VLM focuses on the articulation structure and motion tolerance of articulated objects, using a visual language model to detect the bounding box, axis, and joint type of the object, thereby enabling the manipulation process.

[0082] Results: Table 1 compares the manipulation performance of this invention with six baseline methods: Where2Act, UMPNet, Flowbot3D, Implicit3D, ManipLLM, and A3VLM. As can be seen from the results in Table 1, this invention outperforms the other baseline methods on 16 out of 30 object categories. For example, in the "gate" category, the success rate of this invention is 90%, 8% higher than the second-ranked A3VLM's 82%. Furthermore, as shown in Table 1, in the "seen" category, the success rate of this invention is 78%, 5% higher than the second-ranked A3VLM; in the "unseen" category, the success rate of this invention is 71%, close to the second-ranked A3VLM, indicating that this invention has a strong zero-error capability.

[0083] Table 1

[0084]

[0085] Example 2

[0086] To verify the operational performance of this invention in real-world applications, this embodiment generates articulated manipulation in a metaverse environment and conducts inference experiments in a real-world environment using a Faurecia FR5 6-DOF robotic arm. In the real-world environment, this embodiment utilizes a Teledyne DALSA TM461-E2 depth camera and a CyberTech Pincer2F120S robotic hand. Figure 3As shown, the experiment first calibrates the intrinsic and extrinsic parameters of the depth camera to capture images of the articulated object. Furthermore, to eliminate the influence of background differences between the camera and the reference image, Segment Anything is used to extract image semantics and replace the image background. Finally, based on the inference results of the ERMVLA model, the experiment utilizes end effector inverse kinematics based on a cubic spline trajectory interpolation mechanism to calculate the waypoints of the robotic arm.

[0087] Results: As shown in Table 2, this invention underwent 10 real-world experiments involving five different types of articulated objects: microwave ovens, kettles, pots, lids, and drawers. The success rates are shown in the table. The success rate for this invention was 90% in the kettle and pot categories, and 80% in the oven, lid, and drawer categories. Analysis of the experimental results demonstrates that the manipulation knowledge of this invention in real-world environments can be transferred from the metaverse environment and applied to new categories of robotic arms, proving that this method maintains category independence when transferring from the metaverse environment to the real-world environment.

[0088] Table 2

[0089]

Claims

1. A method for constructing a large VLA model with high inference speed for robot control, characterized in that... The method includes the following steps: Step 1: Fine-grained geometric modeling of the hinged object in a simulation environment: In a virtual environment, the physical properties of articulated objects are combined to model them, thereby systematically extracting and labeling six key feature information: object category, bounding box, axis of rotation, contact point, gripping direction, and movement path, comprehensively describing the maneuverability of articulated objects. Step 2: Construct the training dataset: Based on the six key feature information extracted in step one, six training labels are generated in the virtual simulation environment: object category recognition, bounding box recognition, pivot recognition, contact point recognition, grasping direction recognition, and movement path planning, thereby realizing efficient learning and generalization of the VLA model driven by multi-dimensional supervision signals. Step 3: Design the VLA model architecture: The VLA model architecture is designed based on a pre-trained large-scale language model and a visual encoder. The visual encoder extracts visual features from the input image and connects the visual encoder and the language model through a multilayer perceptron, mapping the visual features to the language embedding space. The text command is encoded by pre-trained word vectors and concatenated with the visual features before being input into the Mamba language model, which outputs a response in natural language form. Step 4: Train the VLA model: The VLA model architecture designed in step three is jointly trained using the various training tasks constructed in step two. A loss function is designed for the output features of each task. By weighted summation of the loss functions of all tasks, a unified multi-task joint loss function is constructed as the global optimization objective during the model training phase.

2. The method for constructing a large VLA model with high inference speed for robot control according to claim 1, characterized in that... In step two, object category recognition guides the VLA model to output the specific object category to be operated on based on the given image and user instructions; bounding box recognition guides the VLA model to use 3D bounding boxes. The target object's position is marked to ensure accurate spatial positioning; pivot recognition guides the VLA model to accurately identify the target object's pivot position. Contact point recognition guides the VLA model to accurately identify the optimal contact point of the target object; grasping direction recognition guides the VLA model to accurately identify the grasping direction. The motion path planning is responsible for guiding the VLA model to convert abstract text instructions into specific mechanical movements, ensuring that the end effector of the robotic arm performs operations along the optimal path.

3. The method for constructing a large VLA model with high inference speed for robot control according to claim 2, characterized in that... The object category recognition uses the ViT-L / 14 visual-semantic embedding model in CLIP to combine the input image and text candidate data. Each vector is mapped to a shared embedding space and transformed into a high-dimensional vector. Cosine similarity is calculated, and the category with the highest similarity is selected as the label for the object category recognition task.

4. The method for constructing a large VLA model with high inference speed for robot control according to claim 2, characterized in that... The three-dimensional bounding box Through the coordinates of the eight vertices This means that the three-dimensional center point is first represented in the camera coordinate system. and size , These represent the size components of the object along the x, y, and z axes in three-dimensional space, respectively. Then, the three-dimensional center point is used. and Perform addition and subtraction operations on the value to calculate the result. : （1）； The position of the rotating shaft Through the coordinates of the two vertices This indicates that, for Prismatic type targets, the two endpoints of the pivot point correspond to the pivot position. The starting point and ending point and the formula minus The value is related to the axis position for Revolute type targets. The starting point, the ending point, and the center point minus The value is related to: （2）。 5. The method for constructing a large VLA model for robot control with high inference speed according to claim 4, characterized in that... The optimal contact point The calculation method is as follows: For Prismatic type targets, the optimal contact point The geometric center of the exposed surface of the target object is located first, and the three-dimensional bounding box is calculated based on formula (1). The six sides : （3） Then calculate each face. direction of movement and the direction of rotation of the shaft For each face Its geometric center ,use center point and the geometric center point of each face Calculate the movement direction vector for each face. : （4） According to the position of the pivot in formula (2) two points and Calculate the direction of the shaft's motion. : （5） According to each face Relative to the center point motion direction vector Direction of rotation of the axis Choose the center point of the surfaces with the greatest similarity between the two as the optimal contact point. ; Optimal contact point for Revolute type objects Use each weight For formula (1) Perform weighted sampling calculations: （6）。 6. The method for constructing a large VLA model for robot control with high inference speed according to claim 5, characterized in that... The direction of grasping The method for determining it is as follows: For Prismatic type targets, the grasping direction is the same as the axis of motion. Conversely, ensure that the gripper operates with axial thrust in the direction of the gripper. For Revolute type targets, the gripping direction should be perpendicular to the object surface to ensure that the applied force acts on the correct contact surface.

7. The method for constructing a large VLA model for robot control with high inference speed according to claim 6, characterized in that... The gripping direction of the Revolute type object Using formula (6) as well as Construct a vertical vector to represent the grasping direction. : （7）。 8. The method for constructing a large VLA model for robot control with high inference speed according to claim 6, characterized in that... The motion path planning generates a sequence of motion waypoints consisting of multiple points based on the results of bounding box recognition, pivot recognition, contact point recognition, and grab direction recognition. ; For Prismatic type targets, based on the optimal contact point and the direction of rotation of the shaft Calculate the motion waypoint sequence : （8） Where c and t represent the movement speed and time step, respectively; For Revolute type targets, records The rotation matrix is ​​calculated using Rodrigues' rotation formula. Ultimately, the optimal contact point will be determined. Rotate and calculate the motion waypoint sequence. : （9）。 9. The method for constructing a large VLA model with high inference speed for robot control according to claim 8, characterized in that... In step four, for the object category recognition task, cross-entropy loss is used. As the target of optimization: （10） In the formula, Indicates multimodal input, For the first Target words at each time step To predict the probability of the word for the model under the current context; For bounding box recognition tasks, the predicted values ​​of eight points are... Compared with the true value The Euclidean distance between them is used as the optimization objective, and the loss function is... The definition is as follows: （11） For axis recognition tasks, the loss function as follows: （12） In the formula, and These are the weights for the direction loss and the position loss, respectively. In the contact point identification task, the contact point prediction result With real contact points Error between Defined as: （13） Loss of grasping orientation recognition for: （14） For the task of movement path planning, to evaluate the model's predicted path With the real path The difference between them defines the path loss function. for: （15） They represent and The three-dimensional coordinates of the points; Finally, the loss function during model training. for: （16）。

Images