A method, apparatus, equipment, and medium for training a VLA robot model.

By introducing a differentiable positive kinematics module and end-effector position loss into the VLA model training, the problem of lacking body geometric constraints of the robotic arm in the prior art is solved, thereby improving the positioning accuracy of the end effector and the success rate of robot operation.

CN123113689APending Publication Date: 2026-11-17HANGZHOU GONGSHU DISTRICT HOLOGRAPHIC INTELLIGENT TECHNOLOGY RESEARCH INSTITUTE
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611295014.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-25
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing technologies only optimize prediction errors in the joint motion space when training VLA models, lacking geometric constraints on the robot arm itself. This results in insufficient positioning accuracy of the end effector, affecting the success rate of the robot in performing tasks such as grasping.

Method used

By inputting visual observations, verbal commands, and the current joint states of the robotic arm into the VLA model, a normalized predicted action sequence is output. The predicted joint angles are converted into predicted end positions of the end effector through a differentiable positive kinematics module. The end position loss is calculated by combining the taught action sequences from the training dataset. The VLA model parameters are updated by backpropagation, and the geometric constraints of the robotic arm body are introduced.

Benefits of technology

It improves the terminal positioning accuracy of the end effector, increases the success rate of the robot performing tasks such as grasping, and eliminates the need for additional manual labeling of the end position.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123113689A_ABST
    Figure CN123113689A_ABST
Patent Text Reader

Abstract

This invention discloses a model training method, apparatus, device, and medium for a VLA robot. The method includes inputting visual observations, verbal commands, and the current joint state of the robotic arm into a VLA model, outputting a normalized predicted action sequence; extracting joint increment components and inversely normalizing them using normalized statistics to obtain predicted joint increments; adding the inversely normalized current joint state to the predicted joint increments and converting the result to radians to obtain predicted joint angles; inputting the predicted joint angles into a differentiable positive kinematics module constructed from the robotic arm model description file to obtain the predicted end effector position; obtaining a reference end effector position from the taught action sequence and calculating the end effector position loss; adding the end effector position loss to the original training loss according to preset weights and backpropagating to obtain the target VLA model. This invention introduces the geometric constraints of the robotic arm body into the training, allowing the model to directly optimize the end effector position without additional annotation, thus improving the end effector positioning accuracy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a model training method, apparatus, device, and medium for a VLA robot. Background Technology

[0002] With the development of embodied intelligence and robot imitation learning technologies, VLA (Vision-Language-Action) models are widely used in robot manipulation tasks. VLA models typically predict a sequence of actions over a future period based on visual observations captured by cameras, language commands describing the task, and the current joint states of the robotic arm. This prediction drives the robotic arm to perform tasks such as grasping and placing. During the training phase, existing technologies commonly employ methods such as image- and neural network-based robot grasping learning, neural network-based end-effector visual servo control, action chunking-based imitation learning, diffusion policy-based action generation, flow matching-based visual-language-action flow models, and robot Transformers trained on large-scale real robot data to model and generate action sequences. These methods constrain the predicted actions in joint space using action regression loss or flow matching loss to ensure they closely approximate the taught actions in the training dataset, i.e., using only the numerical error in the action dimension as the training objective.

[0003] However, the joint space error of a robotic arm is not equivalent to the end-effector position error: the joints of the robotic arm are connected to the end-effector via links, and different combinations of joint errors result in significant differences in the amount of movement of the end-effector. Furthermore, the same joint error has different effects on the three-dimensional position of the end-effector under different robotic arm postures. For grasping tasks, the final physical position reached by the end-effector directly determines whether it can accurately contact the target object. Therefore, current technologies only optimize prediction errors in the joint motion space when training VLA models, lacking constraints on the robotic arm's body geometry. This results in the trained VLA model failing to adequately constrain the end-effector's position, leading to insufficient end-effector positioning accuracy and consequently affecting the success rate of the robot performing grasping and other maneuvering tasks. Summary of the Invention

[0004] This invention provides a model training method, apparatus, device, and medium for VLA robots, aiming to solve the technical problem in the prior art where VLA model training only optimizes prediction errors in the joint motion space and lacks geometric constraints on the robot arm body, resulting in insufficient positioning accuracy of the end effector.

[0005] In a first aspect, embodiments of the present invention provide a model training method for a VLA robot, comprising: The VLA model is input into the visual observations, language commands, and current joint states of the robotic arm collected by the robot, and outputs a normalized predicted action sequence. The joint increment components of the robotic arm are extracted from the normalized predicted action sequence and denormalized according to the normalized statistics of the training dataset to obtain the predicted joint increments. The current joint state is inversely normalized and then added to the predicted joint increment. The angle unit of the addition result is converted to radians to obtain the predicted joint angle. The predicted joint angle is input into a differentiable positive kinematics module constructed based on the model description file of the robotic arm, and the predicted end position of the end effector is output. The reference terminal position is obtained by using the teaching action sequence of the training dataset, and the terminal position loss between the predicted terminal position and the reference terminal position is calculated. The terminal position loss is added to the original training loss of the VLA model according to a preset weight, and the VLA model parameters are updated by backpropagation to obtain the target VLA model.

[0006] Secondly, embodiments of the present invention provide a model training device for a VLA robot, comprising: The prediction module is used to input the visual observations, language commands and the current joint state of the robotic arm collected by the robot into the VLA model, and output a normalized predicted action sequence. The inverse normalization module is used to extract the joint increment components of the robotic arm from the normalized predicted action sequence, and perform inverse normalization according to the normalization statistics of the training dataset to obtain the predicted joint increments. The conversion module is used to inversely normalize the current joint state and add it to the predicted joint increment, and convert the angle unit of the addition result into radians to obtain the predicted joint angle; The positioning module is used to input the predicted joint angle into a differentiable positive kinematics module constructed based on the model description file of the robotic arm, and output the predicted end position of the end effector. The loss module is used to obtain the reference terminal position through the teaching action sequence of the training dataset, and to calculate the terminal position loss between the predicted terminal position and the reference terminal position. An optimization module is used to add the terminal position loss to the original training loss of the VLA model according to a preset weight, and backpropagate to update the parameters of the VLA model to obtain the target VLA model.

[0007] Thirdly, embodiments of the present invention provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the model training method for the VLA robot of the first aspect.

[0008] Fourthly, embodiments of the present invention provide a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, it implements the model training method for the VLA robot of the first aspect.

[0009] This invention provides a model training method for a VLA robot, comprising: inputting visual observations, language commands, and the current joint state of the robotic arm collected by the robot into a VLA model, and outputting a normalized predicted action sequence; extracting the joint increment components of the robotic arm from the normalized predicted action sequence, and performing inverse normalization according to the normalized statistics of the training dataset to obtain the predicted joint increment; adding the inverse normalized current joint state to the predicted joint increment, and converting the angle unit of the addition result to radians to obtain the predicted joint angle; inputting the predicted joint angle into a differentiable positive kinematics module constructed based on the model description file of the robotic arm, and outputting the predicted end effector position; obtaining a reference end effector position through the taught action sequence of the training dataset, and calculating the end effector position loss between the predicted end effector position and the reference end effector position; adding the end effector position loss to the original training loss of the VLA model according to a preset weight, and backpropagating to update the VLA model parameters to obtain the target VLA model. This invention constructs a differentiable positive kinematics module based on a model description file of a robotic arm. It maps the actions predicted by the VLA model and the taught actions from the training dataset to the end effector's terminal position by combining the current joint state. The end position loss between the predicted and reference terminal positions is added to the original training loss with preset weights for joint backpropagation. This allows the VLA model to explicitly perceive the physical position deviation of the end effector corresponding to the predicted actions during the training phase. By introducing the robotic arm's body geometric constraints into the model optimization process, this invention solves the technical problem in existing technologies where VLA model training only optimizes prediction errors in the joint motion space, lacking robotic arm body geometric constraints, resulting in insufficient end effector positioning accuracy. Simultaneously, the reference terminal position is automatically calculated from the taught action sequence, the current joint state, and the model description file, eliminating the need for additional manual labeling of the end effector position. This effectively improves the end effector's positioning accuracy while preserving the original training process and model structure, thereby increasing the success rate of the robot performing tasks such as grasping.

[0010] The present invention also provides a model training device, computer equipment, and storage medium for a VLA robot, which have the same beneficial effects as described above. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A schematic flowchart illustrating a model training method for a VLA robot provided in an embodiment of the present invention; Figure 2 This is a schematic block diagram of a VLA robot model training device provided in an embodiment of the present invention. Detailed Implementation

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0015] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0016] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0017] Please see below. Figure 1 , Figure 1 The flowchart of a VLA robot model training method provided in an embodiment of the present invention is shown, specifically including steps S101 to S106.

[0018] S101. Input the visual observations, language commands, and current joint states of the robotic arm collected by the robot into the VLA model, and output a normalized predicted action sequence. S102. Extract the joint increment components of the robotic arm from the normalized predicted action sequence, and perform inverse normalization according to the normalization statistics of the training dataset to obtain the predicted joint increments. S103. After inverse normalizing the current joint state, add it to the predicted joint increment, and convert the angle unit of the addition result to radians to obtain the predicted joint angle. S104. Input the predicted joint angle into a differentiable positive kinematics module constructed based on the model description file of the robotic arm, and output the predicted end position of the end effector. S105. Obtain the reference terminal position through the teaching action sequence of the training dataset, and calculate the terminal position loss between the predicted terminal position and the reference terminal position. S106. The terminal position loss is added to the original training loss of the VLA model according to a preset weight, and the VLA model parameters are updated by backpropagation to obtain the target VLA model.

[0019] In step S101, the VLA (Vision-Language-Action) model refers to a model that directly generates robot actions based on multimodal inputs such as vision, language, and robot state; visual observation refers to environmental images captured by the robot through cameras, for example, two cameras placed at different viewpoints can be captured simultaneously to obtain more complete scene information; language instructions refer to natural language text describing the current operation task; current joint state refers to the joint position readings of each joint of the robotic arm at the current moment. Since the actions and states in the training dataset are usually normalized before being used in training, the predicted action sequence output by the VLA model is also in the normalized numerical space, hence it is called a normalized predicted action sequence, which is an action chunk containing actions in multiple consecutive future frames.

[0020] In one embodiment, step S101 includes: The visual observations collected by the robot are processed by feature extraction and encoding to generate visual input features; Natural language processing, including word segmentation and semantic parsing, is used to obtain the language input instructions; Based on the VLA model, the visual input features, language input commands, and current joint state are fused to obtain the fused features; Noisy actions, velocity predictions, and sampling times are based on the fusion feature output stream matching process. Subtracting the product of the sampling time and the velocity prediction from the noisy action yields the normalized predicted action sequence.

[0021] In this embodiment, the VLA model employs flow matching for action generation training. Flow matching is a continuous-time generative modeling method that constructs a continuous interpolation path between training data and noise, allowing the model to learn the velocity field required to reconstruct real data from noise. After training, it can gradually generate real data from noise along this velocity field. Accordingly, the VLA model in this embodiment may include a visual encoding part, a language encoding part, a multimodal fusion part, and an action generation part (action expert), wherein the action generation part generates actions using flow matching.

[0022] Specifically, the first step is to extract and encode features from the visual observations collected by the robot to generate visual input features. For example, the robot can use two cameras to capture images of the operating scene from different perspectives. These images are then input into the visual encoding part for feature extraction and encoding, resulting in visual input features representing the target objects, obstacles, and the state of the robotic arm itself in the scene. Secondly, natural language processing, such as tokenization and semantic parsing, is performed on the language commands to obtain the language input commands. For example, when the language command is "pick up the cube on the table and put it in the storage box," after tokenization and semantic parsing, the language command is converted into a language input command that the model can understand, used to indicate the operation object and target.

[0023] Based on the VLA model, visual input features, verbal input commands, and the current joint state are fused to obtain fused features. These fused features simultaneously contain information about what's in the scene, what task needs to be completed, and the current configuration of the robotic arm, serving as conditions for subsequent action generation. Based on the fused features, the model outputs noisy action, velocity prediction, and sampling time during the flow matching process: The noisy action (denoted as x_t) is the action obtained by superimposing noise onto the normalized taught action at the sampling time; the sampling time (denoted as t, t∈[0,1]) represents the position of the noisy action on the interpolation path between the noise and the actual action; and the velocity prediction (denoted as v_t) is the velocity predicted by the model based on the fused features, used to restore the noisy action to a noise-free action.

[0024] Finally, by subtracting the product of the sampling time and the velocity prediction from the noisy action, the noise-free clean action can be estimated, resulting in the normalized predicted action sequence. This training action reconstruction process can be represented as:

[0025] pred_actions = x_t - t * v_t; Here, `pred_actions` represents the normalized predicted action sequence, i.e., the clean actions estimated by the model in the current training step; `x_t` represents the noisy actions in the flow matching process; `t` represents the sampling time; and `v_t` represents the velocity prediction. Since the above reconstruction only uses the amount obtained from a single forward computation of the model, the normalized predicted action sequence can be directly obtained at each step of training without performing a complete multi-step denoising sampling, thus ensuring that subsequent terminal position supervision is carried out within the training computation graph and gradients can be backpropagated.

[0026] In step S102, the normalized predicted action sequence is located in the normalized numerical space and contains multiple dimensions of action components, such as the robotic arm joints and grippers. It cannot be directly used for kinematic calculations. Therefore, it is necessary to first extract the joint increment components corresponding to the robotic arm joints, i.e., the changes in the position of each joint of the robotic arm relative to the current moment. Then, inverse normalization is performed using the normalized statistics of the training dataset. Normalized statistics (norm_stats) refer to the normalized parameters obtained from the statistical analysis of action and state data when constructing the training dataset, such as the mean and standard deviation of each dimension. Inverse normalization is the inverse operation of normalization performed using the normalized statistics, restoring the normalized values ​​to values ​​in the actual joint units, thereby obtaining the predicted joint increments.

[0027] In one embodiment, step S102 includes: The first five dimensions are extracted from the normalized predicted action sequence and used as the joint increment components of the robotic arm. Terminal joint increment components are selected from the joint increment components according to preset terminal frame parameters; Read the mean and standard deviation of the joint increment components from the normalized statistics of the training dataset; The terminal joint increment component is denormalized based on the mean and the standard deviation, and the result of the denormalization is used as the predicted joint increment of the robotic arm.

[0028] In this embodiment, the normalized predicted action sequence can be denoted as action block A_pred∈R^(T×6), where R represents the real number field, T represents the number of action frames contained in the action block, and 6 represents the number of dimensions of each action frame. Taking the SO101 robotic arm (a robotic arm containing five rotary joints and a gripper) as an example, its VLA model predicts approximately 40 action blocks at a time, i.e., T is approximately 40; in each action frame, the first five dimensions correspond to the joint position increments of the five joints of the robotic arm, and the sixth dimension corresponds to the opening and closing action of the gripper. Therefore, by extracting the first five dimensions of each action frame from the normalized predicted action sequence, the joint increment components of the robotic arm are obtained. The definition of the joint increment components in the action block can be expressed as: A[:, :, 0:5] = target_arm_joint_position - current_arm_joint_position; Here, A represents an action block, and A[:, :, 0:5] represents the first five dimensions of the action in each frame of the action block, i.e., the joint increment components; target_arm_joint_position represents the target joint position of the robotic arm; and current_arm_joint_position represents the current joint position of the robotic arm. In other words, the first five dimensions of the action are defined as the difference between the target joint position and the current joint position, i.e., the joint position increment (delta position).

[0029] The terminal joint increment component is selected from the joint increment components according to the preset terminal frame parameters. The terminal frame parameter (which can be denoted as terminal_xyz_horizon) is used to specify the frame in the action block that is used as the terminal for position supervision. The index of the selected terminal frame can be denoted as terminal_idx: when the terminal frame parameter is configured to 0, it means that the last frame of the action block is selected as the terminal frame; the terminal frame parameter can also be configured to a fixed prediction time domain (horizon), or aligned with the length of the randomly sampled action block during training, so that the selected terminal frame is consistent with the length of the action block used by the model in the current training; when there are padding frames in the action block, invalid padding frames can also be skipped in combination with a padding mask to ensure that the selected terminal frame is a valid action frame. For example, when T is 40 and the terminal frame parameter is configured to 0, the joint increment component of the 40th frame is selected as the terminal joint increment component.

[0030] The mean and standard deviation of the joint increment components are read from the normalized statistics of the training dataset, and then denormalized based on these mean and standard deviation. During the construction of the training dataset, the motion dimensions are normalized to a standard numerical range using the normalized statistics to stabilize training. Denormalization is performed by inversely using the same statistics: if the mean of a certain dimension in the joint increment component is mean and the standard deviation is std, and the normalized value of that dimension output by the model is a, then the true value after denormalization is a × std + mean. The result of the denormalization process is the predicted joint increment of the robotic arm, and its numerical unit is consistent with the joint unit recorded in the training dataset. This denormalization process can be expressed as: pred_delta = unnormalize(pred_actions[..., 0:5]); Here, `pred_delta` represents the predicted joint increment; `unnormalize(·)` represents the denormalization operation performed according to the normalization statistics of the training dataset; and `pred_actions[..., 0:5]` represents the first five dimensions of the joint increment components extracted from the normalized predicted action sequence `pred_actions`. It should be noted that since denormalization is performed element-wise along each dimension, selecting the terminal joint increment components based on the terminal frame parameters before performing denormalization is mathematically equivalent to performing overall denormalization on the joint increment components before selecting the terminal frame; both methods yield the predicted joint increment at the terminal frame.

[0031] In step S103, the predicted joint increment represents the change in the position of each joint of the robotic arm relative to the current moment. However, forward kinematics calculation requires the absolute joint angles of each joint of the robotic arm. Therefore, the predicted joint increment needs to be added to the current joint state to restore the absolute joint angles. Since the current joint state is also normalized during training, it needs to be denormalized according to the normalized statistics before being added to the predicted joint increment. Furthermore, since the joint angles in the training dataset are recorded in degrees, while forward kinematics calculation usually uses radians as input, the angle unit of the added result also needs to be converted from degrees to radians. Finally, the predicted joint angles that can be directly used for forward kinematics calculation are obtained.

[0032] In one embodiment, step S103 includes: Read the mean and standard deviation of the current joint state from the normalized statistics of the training dataset; The current joint state is denormalized based on the mean and the standard deviation to obtain the initial joint angle. The initial joint angle is added to the predicted joint increment according to the joint dimension to obtain the target joint angle; The unit of the target joint angle is determined to be degrees from the configuration parameters of the training dataset, and the unit of the target joint angle is converted according to the conversion relationship from degrees to radians to obtain the predicted joint angle.

[0033] In this embodiment, the mean and standard deviation corresponding to the current joint state are read from the normalized statistics of the training dataset, and the current joint state is denormalized based on the mean and standard deviation to obtain the initial joint angle. This process can be represented as: init_arm = unnormalize(observation.state.arm.position); Where init_arm represents the initial joint angle after denormalization; observation.state.arm.position represents the current joint state of the robotic arm in the observation data, that is, the joint position reading of each joint of the robotic arm at the current moment, which is stored in normalized form in the training dataset; unnormalize(·) represents the denormalization operation performed according to the normalized statistics.

[0034] The initial joint angles are added to the predicted joint increments corresponding to the joint dimensions. That is, the initial joint angle of the first joint is added to the predicted joint increment of the first joint, the initial joint angle of the second joint is added to the predicted joint increment of the second joint, and so on, to obtain the target joint angle. This process can be represented as: pred_q = init_arm + pred_delta[:, terminal_idx]; Where pred_q represents the target joint angle, i.e., the predicted absolute joint angles of each joint of the robotic arm; pred_delta[:, terminal_idx] represents the joint increment at the terminal frame index terminal_idx in the predicted joint increments. For any h-th frame in the action block, the above relationship can also be uniformly expressed as: q_pred_h = q_state + A_pred[h, 0:5]; Where q_pred_h represents the absolute joint angle of the predicted target corresponding to the h-th frame; q_state represents the joint state of the robotic arm in the current observation (the joint angle after inverse normalization); A_pred[h, 0:5] represents the first five-dimensional joint increments in the h-th frame of the predicted action block A_pred.

[0035] Finally, the unit of the target joint angle is determined from the configuration parameters of the training dataset. In this embodiment, the joint angle unit configuration parameter of the training dataset (which can be denoted as terminal_xyz_joint_unit) is in degrees (deg), meaning the target joint angle is in degrees. The differentiable positive kinematics module takes radians (rad) as input. Therefore, following the conversion from degrees to radians, the target joint angle is multiplied by π / 180 to convert the unit and obtain the predicted joint angle. This unit conversion can be denoted as the deg_to_rad(·) function. For example, if a joint in the target joint angle is 90 degrees, the predicted joint angle after unit conversion is π / 2 radians (approximately 1.5708 radians).

[0036] In step S104, forward kinematics (FK) refers to the process of calculating the pose of the end effector based on the geometry of the robotic arm, given the joint angles of each joint. The end effector is a component installed at the end of the robotic arm that directly performs the operational task, such as a gripper. The robotic arm's model description file is a description file that records the robot's links, joints, and their geometric connections, such as a URDF (Unified Robot Description Format) file. The differentiable forward kinematics module is a forward kinematics calculation module where all calculations consist of differentiable operations, enabling the gradient of the end effector position with respect to the joint angles to propagate backward along the computation graph. By inputting the predicted joint angles into the differentiable forward kinematics module, the predicted end position of the end effector, i.e., the XYZ coordinates of the end effector in three-dimensional space, can be calculated.

[0037] In one embodiment, step S104 includes: The joint chain in the model description file of the robotic arm is parsed, and the link transformation parameters, joint axis direction and end link of each joint are obtained from the joint chain. The link transformation parameters and joint axis directions are arranged in series kinematic chains in the order from the base joint to the end link. The differentiable positive kinematics module is constructed based on the cascaded kinematics chain, and the predicted joint angles are input into the differentiable positive kinematics module to calculate the homogeneous transformation matrix; The three-dimensional coordinates of the end effector are extracted from the homogeneous transformation matrix to obtain the predicted terminal position.

[0038] In this embodiment, taking the SO101 robotic arm as an example, its model description file is a URDF file. The path of this file can be specified through the configuration parameter terminal_xyz_urdf_path, for example, assets / urdf / so101_new_calib.urdf. First, the joint chain in the model description file is parsed. The URDF file records the robot's links and joints in a tree structure. Each joint records the link transformation parameters (including translation and rotation components) from the parent link coordinate system to the joint coordinate system, as well as the joint axis direction (i.e., the axis vector around which the joint rotates). For the SO101 robotic arm, the joints obtained from the joint chain are, in order, shoulder_pan (shoulder horizontal rotation joint), shoulder_lift (shoulder lift joint), elbow_flex (elbow flexion joint), wrist_flex (wrist flexion joint), and wrist_roll (wrist rotation joint), a total of five joints. The end effector is the gripper_frame_link in the gripper coordinate system.

[0039] Following the sequence from the base joints to the end effector links, i.e., shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, and finally gripper_frame_link, the link transformation parameters and joint axis directions of each joint are sequentially arranged into a series of kinematic chains. This series of kinematic chains completely describes the geometric transformation path from the robot arm's base coordinate system, through each joint and link, and finally to the end effector link coordinate system.

[0040] A differentiable positive kinematics module is constructed based on a series of kinematic chains. Specifically, for each joint in the series of kinematic chains, a fixed homogeneous transformation matrix is ​​constructed based on its link transformation parameters, and a rotational homogeneous transformation matrix corresponding to the joint rotation is constructed based on the input joint angle and joint axis direction. The homogeneous transformation matrix is ​​a 4×4 matrix that simultaneously represents the transformation matrix for 3D rotation and 3D translation. After inputting the predicted joint angle into the differentiable positive kinematics module, the fixed homogeneous transformation matrix and the rotational homogeneous transformation matrix of each joint are multiplied sequentially in series to calculate the homogeneous transformation matrix from the base coordinate system to the end-effector coordinate system. Both the matrix construction and multiplication are implemented using differentiable tensor operations, thus allowing the gradient of the end-effector position with respect to the predicted joint angle to propagate backward along the computation graph. This is crucial for the end-effector position loss to be used to train the VLA model.

[0041] Finally, the translation components, i.e., the first three elements of the last column of the homogeneous transformation matrix, are extracted from the homogeneous transformation matrix to obtain the three-dimensional coordinates of the end effector in the base coordinate system, which are used as the predicted end-effector position. The predicted end-effector position is the TCP (Tool Center Point) position of the end effector. For grasping tasks, this position directly determines whether the gripper can accurately reach and contact the target object. This calculation process can be expressed as: pred_xyz = FK_URDF(deg_to_rad(pred_q)); Where, pred_xyz represents the predicted end position of the end effector; FK_URDF(·) represents the positive kinematics calculation implemented by the differentiable positive kinematics module built based on the model description file (URDF); deg_to_rad(·) represents the unit conversion from degrees to radians in step S103; and pred_q represents the target joint angle. For any h-th frame in the action block, it can also be uniformly represented as: xyz_pred_h = FK_URDF(q_pred_h);

[0042] Where xyz_pred_h represents the predicted end-effector position corresponding to the predicted action in frame h; q_pred_h represents the predicted target absolute joint angle in frame h. It should be noted that the above construction method does not depend on the specific structure of the SO101 robotic arm. For any robotic arm with a model description file, the joint chain can be parsed and a differentiable positive kinematics module can be constructed in the same way. Therefore, the method of this embodiment can be easily extended to other robotic arms.

[0043] In step S105, the taught action sequence refers to the real action sequence collected from the training dataset through manual teaching (e.g., remotely operating a robotic arm to complete an operation task), which is the supervised target (Ground Truth, GT) for model learning. Of course, the training dataset can also be output after training a VLA model. The reference end-effector position refers to the actual end-effector position that the end effector should reach, calculated based on the taught action sequence. Since the taught action sequence and the predicted action sequence have the same joint increment representation, the taught action can be mapped to the reference end-effector position in a manner consistent with the prediction side. Then, the end-effector position loss between the predicted end-effector position and the reference end-effector position is calculated, which serves as a training signal to measure the end-effector positioning deviation of the predicted action in three-dimensional physical space.

[0044] In one embodiment, step S105 includes: Extract the teaching joint increment component corresponding to the joint increment component from the teaching action sequence; The teaching joint increment is obtained by inversely normalizing the teaching joint increment component according to the normalized statistics. The current joint state after denormalization is added to the teaching joint increment, and the units are converted to obtain the teaching joint angle. The taught joint angle is input into the differentiable positive kinematics module, and the reference terminal position that does not return gradients is output. The predicted terminal position and the reference terminal position are scaled by a preset coefficient, and the smoothing parameter loss is calculated and used as the terminal position loss.

[0045] In this embodiment, the teaching joint increment components corresponding to the joint increment components are extracted from the teaching action sequence. Specifically, the first five dimensions of the action in each frame of the teaching action sequence are extracted, and the components at the same terminal frame as the prediction side are selected based on the terminal frame parameters. Then, the teaching joint increment components are inversely normalized using normalized statistics to obtain the teaching joint increments. This process can be represented as: gt_delta = unnormalize(gt_actions[..., 0:5]); Where gt_delta represents the teaching joint increment; gt_actions represents the normalized teaching action sequence in the training dataset, gt stands for Ground Truth, representing the true annotation; gt_actions[..., 0:5] represents the first five dimensions of the action in each frame of the teaching action sequence, i.e., the teaching joint increment component; unnormalize(·) represents the inverse normalization operation performed according to the normalization statistics.

[0046] The denormalized current joint state (i.e., the initial joint angle) is added to the taught joint increment according to the joint dimension, and the units are converted according to the degree-to-radian conversion relationship to obtain the taught joint angle. This process can be expressed as: gt_q = init_arm + gt_delta[:, terminal_idx]; Where gt_q represents the summed target joint angle, which is converted to the teaching joint angle using deg_to_rad(·); init_arm represents the initial joint angle after inverse normalization; gt_delta[:,terminal_idx] represents the joint increment at the terminal frame index terminal_idx in the teaching joint increment. For any h-th frame in the action block, the above relationship can also be uniformly represented as: q_gt_h = q_state + A_gt[h, 0:5]; Where q_gt_h represents the true target absolute joint angle corresponding to the h-th frame; q_state represents the joint state of the robotic arm in the current observation; A_gt[h, 0:5] represents the first five-dimensional joint increments in the h-th frame of the taught action block A_gt (A_gt∈R^(T×6), with the same shape as the predicted action block A_pred).

[0047] The taught joint angles are input into the differentiable positive kinematics module, and the three-dimensional coordinates are calculated and extracted using a homogeneous transformation matrix to output the reference terminal position. This process can be represented as: gt_xyz = FK_URDF(deg_to_rad(gt_q)); xyz_gt_h = FK_URDF(q_gt_h); Where gt_xyz represents the reference end position of the end effector; xyz_gt_h represents the reference end position corresponding to the taught action in the h-th frame of the action block; FK_URDF(·) represents the forward kinematics calculation implemented by the differentiable forward kinematics module; deg_to_rad(·) represents the unit conversion from degrees to radians; and q_gt_h represents the true target absolute joint angle in the h-th frame. The reference end position, as the supervised target, is only used to provide the position that the end effector should reach and does not need to propagate gradients back to the reference side. Therefore, a stop gradient operation is performed on the reference end position to prevent gradient propagation back from the reference end position. The gradient only propagates backward through the predicted end position to the VLA model, avoiding perturbation of the supervised target itself.

[0048] Finally, the predicted terminal position and the reference terminal position are scaled according to preset coefficients, and the smoothing parameter loss is calculated and used as the terminal position loss. The preset coefficient (denoted as `scale`) is used to adjust the numerical scale of the terminal position, and its value corresponds to the unit of measurement used: `scale=1` for meters (m), `scale=10` for decimeters (dm), and `scale=100` for centimeters (cm). The smoothing parameter loss, also known as the SmoothL1 loss, is a smoothing loss function calculated as a quadratic function when the error is small and as a linear function when the error is large, balancing fine regression in small error intervals with robustness to larger errors. The calculation of the terminal position loss can be expressed as: L_xyz = SmoothL1(scale * xyz_pred_h, scale * stop_gradient(xyz_gt_h)); Where L_xyz represents the terminal position loss; SmoothL1(·,·) represents the function for calculating the smoothing parameter loss; scale represents the preset coefficients; xyz_pred_h represents the predicted terminal position; xyz_gt_h represents the reference terminal position; and stop_gradient(·) represents stopping the gradient operation. Equivalently, this calculation can also be expressed as: xyz_loss = SmoothL1(pred_xyz * scale, stop_gradient(gt_xyz) * scale); Where xyz_loss represents the terminal position loss; pred_xyz represents the predicted terminal position; and gt_xyz represents the reference terminal position. In this embodiment, the preset coefficients default to a decimeter scale, i.e., scale=10, so that the terminal position error at the centimeter to decimeter level, after scaling, falls exactly within a suitable numerical range for the smoothing parameter loss. For example, when the predicted terminal position differs from the reference terminal position by 2cm, the error value in decimeters is 0.2, which is within the fine regression range of the second segment of the smoothing parameter loss. If a centimeter scale (scale=100) is used, the loss value is too large, which may excessively interfere with the original training loss of the VLA model; if a meter scale (scale=1) is used, the loss value is too small, and the geometric constraint effect is insufficient. By selecting the preset coefficients, the numerical magnitudes of the terminal position loss and the original training loss can be kept balanced.

[0049] In step S106, the original training loss refers to the loss corresponding to the original training objective of the VLA model, such as the flow matching action loss and other auxiliary losses of the original model; the preset weights refer to the weighting coefficients configured for the terminal position loss, used to control the proportion of the terminal position loss in the total loss. The terminal position loss is added to the original training loss according to the preset weights to obtain the total loss, and the VLA model parameters are updated through backpropagation, so that the model optimizes the terminal positioning objective of the end effector while optimizing the action prediction objective; after the training meets the preset conditions, the target VLA model is obtained. The terminal position loss is an auxiliary training objective that can be switched on or off, and does not change the labeling of the training dataset, the structure of the VLA model, or the original training process.

[0050] In one embodiment, step S106 includes: Configure the preset weight to a value greater than the preset value, and simultaneously enable the terminal position loss superposition process; Multiply the terminal location loss by the preset weight to obtain the weighted terminal location loss term; The terminal position loss term is added to the original training loss of the VLA model to obtain the total loss of the VLA model; Calculate the gradient of the total loss with respect to the VLA model parameters, and update the VLA model parameters by backpropagation according to the gradient; Repeat the above steps and record the terminal position loss until the total loss meets the preset conditions to obtain the target VLA model.

[0051] In this embodiment, the preset weight (which can be denoted as terminal_xyz_loss_weight) is configured to a value greater than the preset value, and the terminal position loss stacking process is enabled. In this embodiment, the preset value is 0, meaning that the training process only performs the aforementioned calculation and stacking of terminal position loss when the preset weight terminal_xyz_loss_weight is greater than 0; when the preset weight is configured to 0, the training process degenerates into the original training of the VLA model, thereby realizing the on / off control of terminal position loss as an auxiliary training objective. In this embodiment, the preset weight is configured to 0.2. In addition to applying terminal position loss to the terminal frame, an optional full-trajectory terminal position auxiliary loss weight (terminal_xyz_trajectory_loss_weight) can also be configured to apply terminal position supervision to all frames of the action block. In this embodiment, this weight is configured to 0.0, meaning that only the terminal position at the terminal frame is supervised.

[0052] The terminal position loss is multiplied by a preset weight to obtain a weighted terminal position loss term. This terminal position loss term is then added to the original training loss of the VLA model to obtain the total loss of the VLA model. This process can be represented as: L_total = L_vla + L_depth + weight_xyz * L_xyz; Where L_total represents the total loss of the VLA model; L_vla represents the flow matching action loss of the VLA model; L_depth represents the original depth estimation auxiliary loss of the VLA model, and L_vla and L_depth together constitute the original training loss of the VLA model in this embodiment; weight_xyz represents the preset weights; L_xyz represents the terminal position loss, and the product of weight_xyz and L_xyz is the weighted terminal position loss term. Equivalently, the calculation of the total loss can also be expressed as: total_loss = vla_loss + depth_loss + weight * xyz_loss; Where total_loss represents the total loss; vla_loss represents the flow matching action loss; depth_loss represents the depth estimation auxiliary loss; weight represents the preset weights; and xyz_loss represents the terminal position loss.

[0053] The gradient of the total loss with respect to the VLA model parameters is calculated, and the VLA model parameters are updated by backpropagation according to the gradient. Since the differentiable positive kinematics module and the entire computational link from the normalized predicted action sequence to the predicted terminal position are differentiable, the gradient of the terminal position loss can be backpropagated through the predicted terminal position, predicted joint angle, and predicted joint increment to the action expert of the VLA model, so that the model parameters are updated simultaneously in the direction of reducing action prediction error and reducing end-effector physical position deviation.

[0054] Finally, repeat the above steps and record the terminal position loss. During training, print out metrics such as the terminal position loss (which can be recorded as XYZ_Loss) and the terminal position error converted to centimeters (which can be recorded as XYZ_Err(cm)) step by step, and write them to the training log file (e.g., loss.jsonl); at the same time, calculate the validation loss on the validation set at set intervals and write it to the validation log file (e.g., val_loss.jsonl), and save the model checkpoints; until the total loss meets the preset conditions, such as the total loss converging to a stable interval or the training reaches the preset number of steps, the target VLA model is obtained.

[0055] In a specific experimental example, an open evaluation of the end-effector position was previously performed on checkpoints of several baseline models without end-effector position loss. This revealed a significant end-effector position bias only in models optimized for joint motion space, which is the direct motivation for introducing end-effector position loss in this embodiment. Based on this, with preset weights `terminal_xyz_loss_weight=0.2`, joint angle units `terminal_xyz_joint_unit=deg`, and end-effector position loss numerical scale `terminal_xyz_loss_unit=dm`, the VLA model was continuously trained for 5000 steps. The training log shows that the average end-effector position error was 2.76 cm in the first 500 steps, decreasing to 1.62 cm in the next 500 steps; the average total loss in the latest 100 steps was 0.177346, the average end-effector position loss was 0.453949, and the average end-effector position error was 1.55 cm; and the average total loss in the latest 20 steps was 0.173083, the average end-effector position loss was 0.421665, and the average end-effector position error was 1.47 cm. In terms of validation, the total validation loss (val_loss) at step 5000 (the 4th training epoch) is 0.2128162533044815, of which the validation flow matching action loss (val_vla_loss) is 0.1517678201198578 and the validation depth estimation auxiliary loss (val_depth_loss) is 0.0. It can be seen that as training progresses, the end effector's terminal position error continuously decreases from approximately 2.76 cm to approximately 1.5 cm, indicating that the terminal position loss effectively guides the VLA model to improve the end effector's terminal localization accuracy.

[0056] In summary, the VLA robot model training method provided in this embodiment of the invention inputs the visual observations, language commands, and current joint states of the robotic arm collected by the robot into the VLA model to obtain a normalized predicted action sequence. After extracting joint increment components, inverse normalization, addition with the current joint state, and angle unit conversion, the predicted joint angle is obtained. Then, a differentiable positive kinematics module based on the robotic arm's model description file is used to obtain the predicted end effector position. This end effector position loss is calculated by combining it with the reference end effector position obtained from the taught action sequence. The VLA model parameters are then updated through backpropagation with the original training loss according to preset weights. Thus, during the training phase, the geometric constraints of the robotic arm are explicitly introduced into the VLA model optimization process, allowing the model to directly perceive and optimize the physical end effector position corresponding to the predicted action, rather than merely constraining abstract joint action values. The reference end effector position is automatically calculated by the taught action sequence, current joint state, and model description file, eliminating the need for additional manual labeling of the end effector position. The end effector position loss serves as an on / off auxiliary training objective, preserving the original training process and model structure, and its magnitude can be adjusted through preset coefficients, and different action block lengths can be adapted through end effector frame parameters. This invention solves the technical problem in the prior art where VLA model training only optimizes prediction errors in the joint motion space and lacks geometric constraints on the robot arm itself, resulting in insufficient end-effector positioning accuracy. It effectively improves the end-effector positioning accuracy of the VLA model's predicted actions on the real robot arm, thereby improving the success rate and reliability of the robot performing grasping and other operations.

[0057] Combination Figure 2 As shown, Figure 2 This is a schematic block diagram of a VLA robot model training device 200 provided in an embodiment of the present invention. The VLA robot model training device 200 includes: The prediction module 201 is used to input the visual observations, language commands and current joint states of the robotic arm collected by the robot into the VLA model and output a normalized predicted action sequence. The inverse normalization module 202 is used to extract the joint increment components of the robotic arm from the normalized predicted action sequence, and perform inverse normalization according to the normalization statistics of the training dataset to obtain the predicted joint increments. The conversion module 203 is used to inversely normalize the current joint state and add it to the predicted joint increment, and convert the angle unit of the addition result into radians to obtain the predicted joint angle; The positioning module 204 is used to input the predicted joint angle into a differentiable positive kinematics module constructed based on the model description file of the robotic arm, and output the predicted end position of the end effector. The loss module 205 is used to obtain the reference terminal position through the teaching action sequence of the training dataset, and to calculate the terminal position loss between the predicted terminal position and the reference terminal position. The optimization module 206 is used to add the terminal position loss to the original training loss of the VLA model according to a preset weight, and backpropagate to update the parameters of the VLA model to obtain the target VLA model.

[0058] In this embodiment, the prediction module 201 inputs the visual observations, language commands, and current joint states of the robotic arm collected by the robot into the VLA model and outputs a normalized predicted action sequence; the denormalization module 202 extracts the joint increment components of the robotic arm from the normalized predicted action sequence, performs denormalization according to the normalized statistics of the training dataset, and obtains the predicted joint increment; the conversion module 203 adds the denormalized current joint state to the predicted joint increment, and converts the angle unit of the addition result to radians to obtain the predicted joint angle; the localization module 204 inputs the predicted joint angle into a differentiable positive kinematics module constructed based on the model description file of the robotic arm, and outputs the predicted end position of the end effector; the loss module 205 obtains the reference end position through the taught action sequence of the training dataset, and calculates the end position loss between the predicted end position and the reference end position; the optimization module 206 adds the end position loss to the original training loss of the VLA model according to a preset weight, backpropagates to update the VLA model parameters, and obtains the target VLA model.

[0059] In one embodiment, the prediction module 201 is specifically used for: The visual observations collected by the robot are processed by feature extraction and encoding to generate visual input features; Natural language processing, including word segmentation and semantic parsing, is used to obtain the language input instructions; Based on the VLA model, the visual input features, language input commands, and current joint state are fused to obtain the fused features; Noisy actions, velocity predictions, and sampling times are based on the fusion feature output stream matching process. Subtracting the product of the sampling time and the velocity prediction from the noisy action yields the normalized predicted action sequence.

[0060] In one embodiment, the denormalization module 202 is specifically used for: The first five dimensions are extracted from the normalized predicted action sequence and used as the joint increment components of the robotic arm. Terminal joint increment components are selected from the joint increment components according to preset terminal frame parameters; Read the mean and standard deviation of the joint increment components from the normalized statistics of the training dataset; The terminal joint increment component is denormalized based on the mean and the standard deviation, and the result of the denormalization is used as the predicted joint increment of the robotic arm.

[0061] In one embodiment, the conversion module 203 is specifically used for: Read the mean and standard deviation of the current joint state from the normalized statistics of the training dataset; The current joint state is denormalized based on the mean and the standard deviation to obtain the initial joint angle. The initial joint angle is added to the predicted joint increment according to the joint dimension to obtain the target joint angle; The unit of the target joint angle is determined to be degrees from the configuration parameters of the training dataset, and the unit of the target joint angle is converted according to the conversion relationship from degrees to radians to obtain the predicted joint angle.

[0062] In one embodiment, the positioning module 204 is specifically used for: The joint chain in the model description file of the robotic arm is parsed, and the link transformation parameters, joint axis direction and end link of each joint are obtained from the joint chain. The link transformation parameters and joint axis directions are arranged in series kinematic chains in the order from the base joint to the end link. The differentiable positive kinematics module is constructed based on the cascaded kinematics chain, and the predicted joint angles are input into the differentiable positive kinematics module to calculate the homogeneous transformation matrix; The three-dimensional coordinates of the end effector are extracted from the homogeneous transformation matrix to obtain the predicted terminal position.

[0063] In one embodiment, the loss module 205 is specifically used for: Extract the teaching joint increment component corresponding to the joint increment component from the teaching action sequence; The teaching joint increment is obtained by inversely normalizing the teaching joint increment component according to the normalized statistics. The current joint state after denormalization is added to the teaching joint increment, and the units are converted to obtain the teaching joint angle. The taught joint angle is input into the differentiable positive kinematics module, and the reference terminal position that does not return gradients is output. The predicted terminal position and the reference terminal position are scaled by a preset coefficient, and the smoothing parameter loss is calculated and used as the terminal position loss.

[0064] In one embodiment, the optimization module 206 is specifically used for: Configure the preset weight to a value greater than the preset value, and simultaneously enable the terminal position loss superposition process; Multiply the terminal location loss by the preset weight to obtain the weighted terminal location loss term; The terminal position loss term is added to the original training loss of the VLA model to obtain the total loss of the VLA model; Calculate the gradient of the total loss with respect to the VLA model parameters, and update the VLA model parameters by backpropagation according to the gradient; Repeat the above steps and record the terminal position loss until the total loss meets the preset conditions to obtain the target VLA model.

[0065] Since the embodiments of the apparatus and the embodiments of the method correspond to each other, please refer to the description of the embodiments of the method for the embodiments of the apparatus, which will not be repeated here.

[0066] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed, can perform the steps provided in the above embodiments. The storage medium may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0067] This invention also provides a computer device, which may include a memory and a processor. The memory stores a computer program, and when the processor calls the computer program in the memory, it can implement the steps provided in the above embodiments. Of course, the computer device may also include various network interfaces, a power supply, a graphics card, etc., to utilize the graphics card's performance to operate the model, such as for inference and training.

[0068] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0069] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A model training method for a VLA robot, characterized in that, include: The VLA model is input into the visual observations, language commands, and current joint states of the robotic arm collected by the robot, and outputs a normalized predicted action sequence. The joint increment components of the robotic arm are extracted from the normalized predicted action sequence and denormalized according to the normalized statistics of the training dataset to obtain the predicted joint increments. The current joint state is inversely normalized and then added to the predicted joint increment. The angle unit of the addition result is converted to radians to obtain the predicted joint angle. The predicted joint angle is input into a differentiable positive kinematics module constructed based on the model description file of the robotic arm, and the predicted end position of the end effector is output. The reference terminal position is obtained by using the teaching action sequence of the training dataset, and the terminal position loss between the predicted terminal position and the reference terminal position is calculated. The terminal position loss is added to the original training loss of the VLA model according to a preset weight, and the VLA model parameters are updated by backpropagation to obtain the target VLA model; The process involves obtaining a reference terminal position from the taught action sequence in the training dataset and calculating the terminal position loss between the predicted terminal position and the reference terminal position. This includes: extracting a taught joint increment component corresponding to the joint increment component from the taught action sequence, wherein the taught joint increment component and the joint increment component are selected from the same action frame; denormalizing the taught joint increment component according to the normalized statistics to obtain the taught joint increment; adding the denormalized current joint state to the taught joint increment and performing unit conversion to obtain the taught joint angle; inputting the taught joint angle into the differentiable positive kinematics module and outputting a reference terminal position that does not backpropagate gradients, wherein a gradient-stopping operation is performed on the reference terminal position so that the gradient only propagates back to the VLA model via the predicted terminal position; scaling the predicted terminal position and the reference terminal position by a preset coefficient, calculating a smoothing parameter loss and using it as the terminal position loss, wherein the preset coefficient is used to adjust the numerical scale of the terminal position, and the value of the preset coefficient corresponds to the position unit used by the predicted terminal position and the reference terminal position.

2. The model training method for the VLA robot according to claim 1, characterized in that, The VLA model is input into visual observations, verbal commands, and the current joint states of the robotic arm collected by the robot, and outputs a normalized predicted sequence of actions, including: The visual observations collected by the robot are processed by feature extraction and encoding to generate visual input features; Natural language processing, including word segmentation and semantic parsing, is used to obtain the language input instructions; Based on the VLA model, the visual input features, language input commands, and current joint state are fused to obtain the fused features; Noisy actions, velocity predictions, and sampling times are based on the fusion feature output stream matching process. Subtracting the product of the sampling time and the velocity prediction from the noisy action yields the normalized predicted action sequence.

3. The model training method for the VLA robot according to claim 1, characterized in that, The joint increment components of the robotic arm are extracted from the normalized predicted action sequence, and then denormalized according to the normalized statistics of the training dataset to obtain the predicted joint increments, including: The first five dimensions are extracted from the normalized predicted action sequence and used as the joint increment components of the robotic arm. Terminal joint increment components are selected from the joint increment components according to preset terminal frame parameters; Read the mean and standard deviation of the joint increment components from the normalized statistics of the training dataset; The terminal joint increment component is denormalized based on the mean and the standard deviation, and the result of the denormalization is used as the predicted joint increment of the robotic arm.

4. The model training method for the VLA robot according to claim 1, characterized in that, The current joint state is inversely normalized and then added to the predicted joint increment. The angle unit of the addition result is converted to radians to obtain the predicted joint angle, including: Read the mean and standard deviation of the current joint state from the normalized statistics of the training dataset; The current joint state is denormalized based on the mean and the standard deviation to obtain the initial joint angle. The initial joint angle is added to the predicted joint increment according to the joint dimension to obtain the target joint angle; The unit of the target joint angle is determined to be degrees from the configuration parameters of the training dataset, and the unit of the target joint angle is converted according to the conversion relationship from degrees to radians to obtain the predicted joint angle.

5. The model training method for the VLA robot according to claim 1, characterized in that, The predicted joint angles are input into a differentiable positive kinematics module constructed based on the model description file of the robotic arm, and the predicted end effector position is output, including: The joint chain in the model description file of the robotic arm is parsed, and the link transformation parameters, joint axis direction and end link of each joint are obtained from the joint chain. The link transformation parameters and joint axis directions are arranged in series kinematic chains in the order from the base joint to the end link. The differentiable positive kinematics module is constructed based on the cascaded kinematics chain, and the predicted joint angles are input into the differentiable positive kinematics module to calculate the homogeneous transformation matrix; The three-dimensional coordinates of the end effector are extracted from the homogeneous transformation matrix to obtain the predicted terminal position.

6. The model training method for the VLA robot according to claim 1, characterized in that, The terminal position loss is added to the original training loss of the VLA model according to a preset weight, and the VLA model parameters are updated by backpropagation to obtain the target VLA model, including: Configure the preset weight to a value greater than the preset value, and simultaneously enable the terminal position loss superposition process; Multiply the terminal location loss by the preset weight to obtain the weighted terminal location loss term; The terminal position loss term is added to the original training loss of the VLA model to obtain the total loss of the VLA model; Calculate the gradient of the total loss with respect to the VLA model parameters, and update the VLA model parameters by backpropagation according to the gradient; Repeat the above steps and record the terminal position loss until the total loss meets the preset conditions to obtain the target VLA model.

7. A model training device for a VLA robot, characterized in that, include: The prediction module is used to input the visual observations, language commands and the current joint state of the robotic arm collected by the robot into the VLA model, and output a normalized predicted action sequence. The inverse normalization module is used to extract the joint increment components of the robotic arm from the normalized predicted action sequence, and perform inverse normalization according to the normalization statistics of the training dataset to obtain the predicted joint increments. The conversion module is used to inversely normalize the current joint state and add it to the predicted joint increment, and convert the angle unit of the addition result into radians to obtain the predicted joint angle; The positioning module is used to input the predicted joint angle into a differentiable positive kinematics module constructed based on the model description file of the robotic arm, and output the predicted end position of the end effector. The loss module is used to obtain a reference terminal position from the taught action sequence of the training dataset and calculate the terminal position loss between the predicted terminal position and the reference terminal position. Specifically, the loss module is used to: extract the taught joint increment component corresponding to the joint increment component from the taught action sequence, wherein the taught joint increment component and the joint increment component are selected from the same action frame; denormalize the taught joint increment component according to the normalized statistics to obtain the taught joint increment; add the denormalized current joint state to the taught joint increment and perform unit conversion to obtain the taught joint angle; input the taught joint angle into the differentiable positive kinematics module and output a reference terminal position that does not backpropagate gradients, wherein a gradient stopping operation is performed on the reference terminal position so that the gradient only propagates back to the VLA model through the predicted terminal position; scale the predicted terminal position and the reference terminal position according to a preset coefficient, calculate the smoothing parameter loss and use it as the terminal position loss, wherein the preset coefficient is used to adjust the numerical scale of the terminal position and the value of the preset coefficient corresponds to the position unit used by the predicted terminal position and the reference terminal position; An optimization module is used to add the terminal position loss to the original training loss of the VLA model according to a preset weight, and backpropagate to update the parameters of the VLA model to obtain the target VLA model.

8. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the model training method for the VLA robot as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the model training method for the VLA robot as described in any one of claims 1 to 6.