Robot control method based on visual language action model and related equipment thereof

Abstract

The invention provides a robot control method based on a visual language action model and related equipment thereof, and relates to the technical field of intelligent robots, and the method comprises the steps: obtaining visual observation information, a target language instruction and robot body state information of a target robot; reasoning the visual observation information, the target language instruction and the robot body state information through a pre-trained visual language action model to obtain a predicted low-dimensional potential vector; reconstructing the predicted low-dimensional potential vector into a robot action instruction through a pre-trained high-dimensional action reconstruction module; wherein the dimension of the robot action instruction is matched with the action space dimension of the target robot; and the target robot is driven to execute the high-dimensional robot action instruction. According to the method, the limitation of the existing visual language action model on the action output dimension can be solved, so that the high-degree-of-freedom robot is effectively controlled.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to, but are not limited to, the technical field of intelligent robots, and particularly relate to a robot control method based on a visual language action model and a related device thereof. BACKGROUND

[0002] High-degree humanoid robots are becoming an important platform for realizing general artificial intelligence. As an end-to-end solution, the visual language action model can directly map visual observations and language instructions into robot actions, greatly simplifying the robot programming process.

[0003] In related technologies, the visual language action model has inherent limitations in architecture. The action output dimension of the model is pre-set and limited. When the model is applied to high-degree robots, the limited output dimension of the model is far lower than the action space required for the robot to perform whole-body coordinated fine operations. This dimension mismatch makes it difficult for existing visual language action models to be directly applied to the control of high-degree robots. SUMMARY

[0004] The present application aims to at least solve one of the technical problems existing in the prior art. To this end, the present application provides a robot control method based on a visual language action model and a related device thereof, which can solve the limitation of the existing visual language action model on the action output dimension, and effectively control high-degree robots.

[0005] To achieve the above-mentioned purpose, a first aspect of embodiments of the present application proposes a robot control method based on a visual language action model, the method comprising: obtaining visual observation information of a target robot and a target language instruction; performing inference on the visual observation information and the target language instruction through a pre-trained visual language action model to obtain a predicted low-dimensional latent vector; reconstructing the predicted low-dimensional latent vector into a robot action instruction through a pre-trained high-dimensional action reconstruction module; wherein the dimension of the robot action instruction matches the action space dimension of the target robot; driving the target robot to execute the high-dimensional robot action instruction.

[0006] In some embodiments, the pre-training process of the visual language action model and the high-dimensional action reconstruction module comprises: obtaining a high-dimensional action dataset comprising a plurality of high-dimensional action vectors, and training visual observation information and training language instructions time-aligned with the high-dimensional action vectors; compressing the high-dimensional action vectors through an encoder module to generate a target low-dimensional latent vector; train the visual language action model with the target low-dimensional latent vector as a training target and the training visual observation information and the training language instruction as training inputs, so that the predicted low-dimensional latent vector output by the visual language action model approximates the target low-dimensional latent vector; train the high-dimensional action reconstruction module with the high-dimensional action vector as a reconstruction target and the target predicted low-dimensional latent vector as a training input, so that the high-dimensional robot action instruction output by the high-dimensional action reconstruction module approximates the high-dimensional action vector; jointly train the encoder module, the visual language action model, and the high-dimensional action reconstruction module based on a difference between the reconstructed high-dimensional robot action instruction and the high-dimensional action vector.

[0007] In some embodiments, the compressing the high-dimensional action vector through the encoder module to generate a target low-dimensional latent vector includes: mapping, by a preset multi-layer nonlinear transformation matrix of the encoder module, the high-dimensional action vector to a continuous probability distribution in a low-dimensional latent space; sampling from the continuous probability distribution to generate the target low-dimensional latent vector.

[0008] In some embodiments, the training the visual language action model with the target low-dimensional latent vector as a training target and the training visual observation information and the training language instruction as training inputs, so that the predicted low-dimensional latent vector output by the visual language action model approximates the target low-dimensional latent vector includes: performing multimodal fusion understanding on the training visual observation information and the training language instruction by the visual language action model to obtain multimodal fusion features; mapping the multimodal fusion features to the predicted low-dimensional latent vector, and training the visual language action model iteratively until a difference between the predicted low-dimensional latent vector and the target low-dimensional latent vector converges to a preset first loss threshold, with the difference between the predicted low-dimensional latent vector and the target low-dimensional latent vector as a loss.

[0009] In some embodiments, the training the high-dimensional action reconstruction module with the high-dimensional action vector as a reconstruction target and the target predicted low-dimensional latent vector as a training input, so that the high-dimensional robot action instruction output by the high-dimensional action reconstruction module approximates the high-dimensional action vector includes: performing up-sampling processing on the target predicted low-dimensional latent vector by a preset up-sampling matrix in the high-dimensional action reconstruction module to obtain the high-dimensional robot action instruction; The difference between the high-dimensional robot action command and the high-dimensional action vector is used as the loss to iteratively train the high-dimensional action reconstruction module until the difference converges to a preset second loss threshold.

[0010] In some embodiments, the joint training of the encoder module, the visual language action model, and the high-dimensional action reconstruction module based on the difference between the reconstructed high-dimensional robot action command and the high-dimensional action vector includes: Calculate the first loss between the predicted low-dimensional latent vector output by the visual language action model and the target low-dimensional latent vector; Calculate the second loss between the high-dimensional robot motion command output by the high-dimensional motion reconstruction module and the high-dimensional motion vector; Calculate the divergence loss between the continuous probability distribution output by the encoder module and the preset standard probability distribution; Construct a joint loss function based on the first loss, the second loss, and the divergence loss; The encoder module, the visual language action model, and the high-dimensional action reconstruction module are iteratively trained by minimizing the joint loss function using supervised gradient descent until the joint loss function converges to a preset total loss threshold.

[0011] In some embodiments, the encoder module and the high-dimensional motion reconstruction module are designed to be decoupled from the backbone architecture of the visual language motion model. The encoder module and the high-dimensional motion reconstruction module together form a general adaptation layer to adapt the visual language motion model so that it can control the target robot.

[0012] A second aspect of this application provides a robot control device based on a visual language action model. This device is used to implement the robot control method based on a visual language action model as described in the first aspect. The robot control device includes: The acquisition module is used to acquire the target robot's visual observation information, target language commands, and robot body state information; The reasoning module is used to reason about the visual observation information, the target language instructions and the robot body state information through a pre-trained visual language action model to obtain a predicted low-dimensional potential vector. The reconstruction module is used to reconstruct the predicted low-dimensional latent vector into robot action instructions through a pre-trained high-dimensional action reconstruction module; wherein the dimension of the robot action instructions matches the dimension of the action space of the target robot. A drive module is used to drive the target robot to execute high-dimensional robot motion commands.

[0013] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the robot control method based on a visual language action model as described in the first aspect.

[0014] To achieve the above objectives, a fourth aspect of the present application provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the robot control method based on a visual language action model as described in the first aspect.

[0015] The robot control method and related equipment based on visual language action models proposed in this application have the following beneficial effects: This application first acquires visual observation information, target language commands, and robot body state information of the target robot. Then, it utilizes a pre-trained low-dimensional vector prediction module to perform inference and output predicted low-dimensional latent vectors. This transforms the complex visual language understanding task into a low-dimensional feature prediction task, reducing the learning difficulty and computational burden of the visual language action model and providing a compact representation containing key semantic features for subsequent high-dimensional action generation. Next, a pre-trained high-dimensional action reconstruction module reconstructs the predicted low-dimensional latent vectors into high-dimensional robot action commands that match the dimensions of the target robot's action space. Utilizing the nonlinear mapping relationship from the latent space to the high-dimensional action space pre-learned by this module, it ensures that the generated action commands retain the semantic logic required for the task while possessing high-dimensional accuracy, avoiding information loss and action distortion caused by traditional linear dimensionality reduction methods. Based on this, by driving the target robot to execute high-dimensional robot action commands, a control closed loop from visual language perception to high-dimensional physical execution is achieved. Compared with related technologies that are limited by the inherent architecture of the model and cannot adapt to complex ontology, the method of this application embodiment achieves decoupling and adaptation between low-dimensional model output and high-dimensional robot control requirements through the collaborative work of the above prediction and reconstruction steps. Thus, this application can solve the limitation of existing visual language action models on the dimension of action output, and effectively control high-degree-of-freedom robots.

[0016] Other features and advantages of this disclosure will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the disclosure. The objectives and other advantages of this disclosure may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the robot control method based on a visual language action model provided in an embodiment of this application. Figure 2 This is another flowchart illustrating the robot control method based on a visual language action model provided in this application embodiment; Figure 3 This is another flowchart illustrating the robot control method based on a visual language action model provided in this application embodiment; Figure 4 This is another flowchart illustrating the robot control method based on a visual language action model provided in this application embodiment; Figure 5 This is another flowchart illustrating the robot control method based on a visual language action model provided in this application embodiment; Figure 6 This is another flowchart illustrating the robot control method based on a visual language action model provided in this application embodiment; Figure 7 A schematic diagram of a robot control device based on a visual language action model provided in an embodiment of this application; Figure 8 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0021] With the development of embodied intelligence and robotics, humanoid robots, due to their inherent compatibility with human environments, have become an important vehicle for realizing general artificial intelligence. These robots typically integrate two arms, two dexterous hands, a head, a waist, and a mobile chassis, with body degrees of freedom often exceeding 50, enabling them to perform complex, anthropomorphic collaborative tasks. To reduce the difficulty of robot control, the Vision-Language-Action Model (VLA), as an end-to-end solution, can directly receive visual observations and natural language commands, mapping them to the robot's action output, simplifying the intermediate steps in traditional robot programming.

[0022] However, when applying VLA models to high-degree-of-freedom humanoid robots, a dimensionality mismatch problem arises. Existing VLA models have inherent limitations in their architectural design; the dimensions of their motion output heads are usually pre-defined and fixed. For example, some VLA models only support a maximum of 32-dimensional motion output, a value far lower than the 50+ dimensional motion space required for humanoid robots to perform full-body fine manipulation. This dimensionality difference prevents existing VLA models from directly driving high-degree-of-freedom robots.

[0023] To address the aforementioned problems, this invention proposes a robot control method and related equipment based on a visual language action model. First, it acquires the target robot's visual observation information, target language commands, and robot body state information. Then, it uses a pre-trained low-dimensional vector prediction module to infer and output predicted low-dimensional latent vectors. This transforms the complex visual language understanding task into a low-dimensional feature prediction task, reducing the learning difficulty and computational burden of the visual language action model and providing a compact representation containing key semantic features for subsequent high-dimensional action generation. Next, a pre-trained high-dimensional action reconstruction module reconstructs the predicted low-dimensional latent vectors into high-dimensional robot action commands that match the target robot's action space dimension. Utilizing the nonlinear mapping relationship from the latent space to the high-dimensional action space pre-learned by this module, it ensures that the generated action commands retain the semantic logic required for the task while possessing high-dimensional accuracy, avoiding information loss and action distortion caused by traditional linear dimensionality reduction methods. Based on this, by driving the target robot to execute high-dimensional robot action commands, a control closed loop from visual language perception to high-dimensional physical execution is achieved. Compared with related technologies that are limited by the inherent architecture of the model and cannot adapt to complex ontology, the method of this application embodiment achieves decoupling and adaptation between low-dimensional model output and high-dimensional robot control requirements through the collaborative work of the above prediction and reconstruction steps. Thus, this application can solve the limitation of existing visual language action models on the dimension of action output, and effectively control high-degree-of-freedom robots.

[0024] The robot control method and related equipment based on visual language action model provided in this application are specifically described through the following embodiments. First, the robot control method based on visual language action model in this application is described.

[0025] The robot control method based on visual language action model in this application can be illustrated through the following embodiments.

[0026] Figure 1 This is an optional flowchart of the robot control method based on a visual language action model provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps 101 to 104.

[0027] Step 101: Obtain the visual observation information, target language commands, and robot body state information of the target robot.

[0028] Step 102: The visual observation information, target language instructions and robot body state information are inferred by the pre-trained visual language action model to obtain the predicted low-dimensional potential vector.

[0029] Step 103: The predicted low-dimensional latent vectors are reconstructed into robot action commands through a pre-trained high-dimensional motion reconstruction module.

[0030] Step 104: Drive the target robot to execute high-dimensional robot motion commands.

[0031] In step 101 of some embodiments, visual observation information typically refers to image or video stream data acquired by cameras mounted on the robot or third-party perspective cameras placed in the environment. This data reflects the physical state of the robot's current environment, the position of target objects, and the distribution of obstacles. Target language commands are natural language text or voice commands issued by the user, such as "Please put my coffee cup on the table," which are acquired via microphone and processed through speech-to-text, or directly input through the interactive interface. In addition, the robot's body state information needs to be acquired synchronously through joint encoders, inertial measurement units, and other devices configured on each joint. This information characterizes the robot's body perception data, specifically covering the robot's base position coordinates in the global coordinate system, the pose parameters of the end effector in three-dimensional space, and the current rotation angle and angular velocity of each joint. This step ensures that the control flow can perceive the external environment and understand the user's intentions, providing multimodal raw input data for subsequent decision-making.

[0032] In step 102 of some embodiments, the pre-trained visual-language-action model serves as a core inference engine, receiving the aforementioned visual observation information and target language command robot ontology state information. This model can be based on deep learning architectures such as Transformer, utilizing attention mechanisms to perform cross-modal alignment and fusion understanding of image and text features. Unlike traditional models that directly output high-dimensional joint control signals, this model outputs only a predicted low-dimensional latent vector. This predicted low-dimensional latent vector is a highly compressed abstract feature representation with a dimension far lower than the robot's actual degrees of freedom (e.g., only 32 dimensions). It does not directly correspond to a specific joint angle but rather encodes the semantic intent and temporal features required to complete the current action, such as high-level strategies for "grasping" or "moving."

[0033] In step 103 of some embodiments, the pre-trained high-dimensional motion reconstruction module acts as a decoder from abstract intent to specific physical action. It receives the predicted low-dimensional latent vector from the previous step and, through its internally learned nonlinear mapping, upsamples or decompresses this low-dimensional vector to restore it as a robot motion command. The dimension of the generated robot motion command strictly matches the dimension of the target robot's motion space. For example, for a humanoid robot with 52 degrees of freedom, the command is a vector containing 52 components. This command specifies in detail the angles, angular velocities, or torques required for each joint of the robot at the current time step, thereby restoring the compressed physical details while ensuring the semantic correctness of the motion, thus ensuring the smoothness and physical feasibility of the motion.

[0034] In step 104 of some embodiments, the generated high-dimensional robot motion commands are sent to the robot's underlying controller or joint servo driver. The underlying controller parses these high-dimensional commands and converts them into specific current or voltage signals to drive the coordinated movement of the robot's motors or actuators throughout its body. For example, when performing fine manipulation tasks, these commands can simultaneously control the robot's arm movement, wrist rotation, and dexterous hand finger bending, thereby accurately completing complex actions such as grasping, placing, or manipulating tools.

[0035] Through steps 101 to 104 above, this embodiment of the application utilizes a pre-trained low-dimensional vector prediction module to transform the complex perception and understanding task of high-dimensional action space into feature prediction in low-dimensional space, reducing the computational burden on the model. Subsequently, a pre-trained high-dimensional action reconstruction module is used to restore the abstract low-dimensional features into detailed high-dimensional physical instructions, realizing the transformation from a low-dimensional decision space to a high-dimensional execution space. This hierarchical processing mechanism not only avoids the convergence problem caused by directly training a high-dimensional output model, but also ensures the physical realism of the action through an independent reconstruction module. Thus, this application can solve the limitation of existing visual language action models on the dimension of action output, thereby effectively controlling high-degree-of-freedom robots.

[0036] Please see Figure 2 In some embodiments, the pre-training process of the visual language action model and the high-dimensional action reconstruction module may include, but is not limited to, steps 201 to 205.

[0037] Step 201: Obtain a high-dimensional action dataset including multiple high-dimensional action vectors, as well as training visual observation information and training language instructions that are time-aligned with the high-dimensional action vectors.

[0038] Step 202: The high-dimensional action vector is compressed by the encoder module to generate the target low-dimensional latent vector.

[0039] Step 203: Using the target low-dimensional latent vector as the training objective and training visual observation information and training language instructions as training inputs, train the visual language action model so that the predicted low-dimensional latent vector output by the visual language action model approximates the target low-dimensional latent vector.

[0040] Step 204: Using the high-dimensional motion vector as the reconstruction target and the target prediction low-dimensional latent vector as the training input, the high-dimensional motion reconstruction module is trained so that the high-dimensional robot motion commands output by the high-dimensional motion reconstruction module approximate the high-dimensional motion vector.

[0041] Step 205: Based on the difference between the reconstructed high-dimensional robot action instructions and the high-dimensional action vectors, the encoder module, the visual language action model, and the high-dimensional action reconstruction module are jointly trained.

[0042] In step 201 of some embodiments, a high-dimensional motion dataset containing a large number of robot operation demonstration samples is first loaded. High-dimensional motion vectors refer to a multi-dimensional set of values ​​that can describe the robot's motion state within a specific time step or continuous time period. Specifically, this includes physical quantities in various dimensions such as the rotation angle and angular velocity of each joint of the robotic arm, the position coordinates and attitude of the end effector in three-dimensional space, and the opening and closing degree or torque feedback of the gripper. The time-aligned training visual observation information is typically a series of image frames acquired by cameras or depth radar installed on the robot body or in the operating environment, reflecting the environmental context when the robot performs the task. The training language instructions are natural language descriptions of the current operational task intent, such as "move the water glass on the table to the left tray." The time-alignment process ensures that each frame of visual data and each language instruction establishes a precise timestamp index relationship with the high-dimensional motion vector at a specific moment, thereby providing the model with a "vision-language-action" supervision signal.

[0043] In step 202 of some embodiments, an encoder module is used to extract features and perform dimensionality reduction mapping on complex high-dimensional action data. Since the original high-dimensional action vector space is large and exhibits data sparsity, directly establishing a mapping from visual input to high-dimensional action can easily lead to model overfitting or difficulty in convergence. The encoder module (e.g., a variational autoencoder network or a multilayer perceptron structure) uses nonlinear transformation operations to compress and map high-dimensional action vectors containing temporal dependencies into a lower-dimensional latent space, thereby generating a target low-dimensional latent vector. This target low-dimensional latent vector eliminates noise and unnecessary details from the original action data, retaining only the spatiotemporal structural features and key motion trends of the action sequence, serving as an intermediate representation target for subsequent learning of the visual language action model.

[0044] Please see Figure 3 In some embodiments, step 202 may include, but is not limited to, steps 301 to 302.

[0045] Step 301: The high-dimensional action vector is mapped to a continuous probability distribution in the low-dimensional latent space by the preset multi-layer nonlinear transformation matrix of the encoder module.

[0046] Step 302: Sample from the continuous probability distribution to generate a low-dimensional potential vector of the target.

[0047] In step 301 of some embodiments, the encoder module is configured as a neural network architecture with deep feature extraction capabilities. The preset multi-layer nonlinear transformation matrix refers to the computational logic formed by the weight parameters of multiple fully connected or convolutional layers in the neural network, combined with nonlinear activation functions. In this step, a high-dimensional motion vector containing physical quantities such as robotic arm joint angles and end-effector pose is input to the encoder module. After layers of matrix multiplication and nonlinear transformation, the input is no longer mapped to a fixed coordinate point in a low-dimensional space, but rather to statistical parameters describing a continuous probability distribution in the low-dimensional latent space. This continuous probability distribution can be a multidimensional Gaussian distribution, and the specific output of the encoder module is the mean vector and variance vector (or log-variance) of this distribution. This processing method transforms discrete, sparse high-dimensional motion samples into continuous, dense probability regions in the latent space, so that physically similar motions overlap or are adjacent to each other in the latent space.

[0048] In step 302 of some embodiments, sampling from a continuous probability distribution refers to calculating a definite numerical vector based on the statistical parameters output in step 301 combined with the introduced random noise variable following a standard normal distribution. To ensure gradient differentiability during model training, this sampling process can obtain the sampling result by adding the mean vector to the product of the standard deviation vector and the random noise. The generated target low-dimensional latent vector is the specific instance obtained in this sampling; it is a compressed representation of high-dimensional action information. Although the dimension of this vector is much lower than the original action space, it contains the key structural information required to generate the original action, and because it originates from a probability distribution, it has local smoothness.

[0049] Through steps 301 to 302 described above, this embodiment of the application implements a variational feature encoding mechanism based on probability generation. Unlike traditional deterministic autoencoders that map actions to isolated points, this scheme maps high-dimensional action vectors to continuous probability distributions in a low-dimensional latent space and samples them, giving the latent space good continuity and completeness. This mechanism ensures that semantically similar or physically similar actions are tightly clustered in the low-dimensional space. For the visual language action model responsible for predicting the vector, this smooth and regularized latent space manifold significantly reduces the difficulty of the regression task, enabling the model to predict latent vectors within the effective distribution interval even when faced with unseen visual scenes or language instructions, thereby improving the robustness and generalization ability of robot action generation.

[0050] In step 203 of some embodiments, the visual language action model is trained to understand cross-modal inputs and plan action intentions. The model receives training visual observation information after feature extraction (such as image feature embeddings extracted via a convolutional neural network or visual Transformer), training language instructions (such as semantic vectors generated by a text encoder), and the low-dimensional latent vector generated in step 202 as input conditions. It outputs a predicted low-dimensional latent vector with the goal of fitting the low-dimensional latent vector generated in step 202. During training, an optimization algorithm adjusts the weight parameters of the visual language action model by minimizing the distributional or geometrical distance between the predicted and target low-dimensional latent vectors. This process enables the visual language action model to learn how to infer a low-dimensional latent vector representing the correct action intention based on the currently seen scene and received instructions, achieving a precise mapping from perceptual input to low-dimensional latent vectors.

[0051] Please refer to 4. In some embodiments, step 203 may include, but is not limited to, steps 401 to 402.

[0052] Step 401: Perform multimodal fusion understanding of training visual observation information and training language instructions through the visual language action model to obtain multimodal fusion features.

[0053] Step 402: Map the multimodal fusion features to predict low-dimensional latent vectors, and use the difference between the predicted low-dimensional latent vectors and the target low-dimensional latent vectors as the loss to train the visual language action model iteratively until the difference between the predicted low-dimensional latent vectors and the target low-dimensional latent vectors converges to a preset first loss threshold.

[0054] In step 401 of some embodiments, the visual language action model first processes heterogeneous input data separately through an internally integrated feature extraction network. For training visual observation information, spatial features including environmental geometry and object positions can be extracted using a convolutional neural network or a visual encoder such as a visual Transformer; for training language instructions, semantic features including task intent and operational logic can be extracted using a Transformer-based text encoder. Multimodal fusion understanding refers to the processor using a cross-attention mechanism or a multimodal Transformer layer to perform deep interactive alignment operations on the aforementioned spatial and semantic features. In this process, language features are used as query vectors to retrieve key regions in visual features, or the two are concatenated and weighted along the feature channel dimension to generate multimodal fusion features. This feature is not merely a simple superposition of single-modal information, but forms a unified high-level semantic representation that combines environmental perception and task orientation, accurately depicting the context state required to execute a specific language instruction in the current visual scene.

[0055] In step 402 of some embodiments, mapping the multimodal fusion features to a predicted low-dimensional latent vector refers to compressing and transforming the high-dimensional fusion features into a predefined low-dimensional latent space through a fully connected layer, generating a predicted low-dimensional latent vector representing the model's current policy inference. Subsequently, the numerical difference between this predicted vector and the target low-dimensional latent vector, which serves as a supervision signal, is calculated, using mean squared error or KL divergence as the loss function. Using this difference as the optimization objective, backpropagation can be employed using optimization algorithms such as stochastic gradient descent or Adam to update the weight parameters of the visual-language-action model. The iterative training process continues, constantly correcting the model's prediction bias until the loss value stably decreases and falls below a preset first loss threshold, indicating that the model has learned how to accurately deduce the correct action intent based on visual and linguistic input.

[0056] Through steps 401 to 402 described above, this embodiment of the application utilizes the target low-dimensional latent vector after feature compression as a supervision value, enabling the visual language action model to learn action intentions in the latent space. This approach transforms the physical action regression task into a low-dimensional intention feature matching task, effectively avoiding the data sparsity problem in high-dimensional action spaces and the semantic gap between multimodal heterogeneous data. For the policy network, this alignment at the feature level significantly reduces the difficulty of fitting the mapping function, allowing the model to focus more on understanding the logical relationship between the visual environment and language instructions, thereby accelerating the training convergence process and improving the accuracy of intention prediction for complex tasks.

[0057] In step 204 of some embodiments, the high-dimensional motion reconstruction module acts as a decoder, responsible for restoring abstract decision variables into specific, executable physical control commands. This module takes the predicted low-dimensional latent vector output by the visual language motion model (or the target low-dimensional latent vector used during training to accelerate convergence) as input, and projects the low-dimensional information back into the original high-dimensional motion space through a series of upsampling, deconvolution, or nonlinear mapping operations to generate high-dimensional robot motion commands. The optimization goal of training is to make the output high-dimensional robot motion commands numerically approximate the original high-dimensional motion vector as closely as possible, ensuring that the motion details recovered from the compressed features (such as the precise amplitude and temporal trajectory of joint rotations) are sufficiently realistic, thereby enabling the physical robot to perform refined physical operations.

[0058] Please see Figure 5 In some embodiments, step 204 may include, but is not limited to, steps 501 to 502.

[0059] Step 501: Upsample the target prediction low-dimensional latent vector using the preset upsampling matrix in the high-dimensional motion reconstruction module to obtain high-dimensional robot motion commands.

[0060] Step 502: Using the difference between the high-dimensional robot action command and the high-dimensional action vector as the loss, iteratively train the high-dimensional action reconstruction module until the difference converges to the preset second loss threshold.

[0061] In step 501 of some embodiments, the high-dimensional motion reconstruction module is typically constructed as a multilayer perceptron or deconvolutional neural network architecture, serving as a decoder component in the overall model framework. It receives the target prediction low-dimensional latent vector and reconstructs it into control parameters for physical space. The preset upsampling matrix refers to the set of weight parameters fixed in the network layers within this module. Its function is to expand the feature dimension of the input data layer by layer through matrix operations. During this process, the target prediction low-dimensional latent vector is input to this module. After a series of linear matrix multiplication operations combined with nonlinear activation function processing, the highly abstracted low-dimensional latent vector is remapped back to the original motion state space, thereby generating a high-dimensional robot motion command. This command contains all the low-level control parameters required to drive the robot to perform a specific task, specifically covering the target rotation angle, angular velocity vector, position coordinates and attitude quaternions of each joint of the robotic arm, and the opening / closing state or torque value of the gripper. Its data dimension is consistent with the original high-dimensional motion vector, ensuring the integrity of the motion information.

[0062] In step 502 of some embodiments, the numerical difference between the high-dimensional robot motion command generated in step 501 and the original high-dimensional motion vector used as the ground truth is calculated. Mean squared error or L1 norm can be used as the quantization loss metric. This difference is used as the loss function to backpropagate and update the parameters of the upsampling matrix within the high-dimensional motion reconstruction module. This iterative training process continues, constantly adjusting the weight distribution of the decoder until the error between the reconstructed output and the real motion sample decreases and stabilizes within a preset second loss threshold. This process establishes a precise inverse mapping relationship from the low-dimensional latent space to the high-dimensional physical space, ensuring that the module can recover complex robotic arm motion details losslessly or with minimal loss from the compressed feature distribution.

[0063] In step 205 of some embodiments, to eliminate cascade errors that may be introduced by independent training of modules and to improve the adaptability of the overall architecture, joint fine-tuning of the entire link is performed. First, the difference loss between the finally reconstructed high-dimensional robot action command and the original real high-dimensional action vector is calculated, and the gradient signal generated by this difference is simultaneously backpropagated to the encoder module, the visual language action model, and the high-dimensional action reconstruction module. This joint training strategy makes the latent space generated by the encoder not only easier to reconstruct but also easier to predict by the visual language action model; at the same time, it makes the predicted vector generated by the visual language action model contain more effective physical information that is conducive to action recovery. This end-to-end optimization ensures the inherent consistency of the three stages of feature compression, intent prediction, and action reconstruction.

[0064] Please see Figure 6 In some embodiments, step 205 may include, but is not limited to, steps 601 to 605.

[0065] Step 601: Calculate the first loss between the predicted low-dimensional latent vector output by the visual language action model and the target low-dimensional latent vector.

[0066] Step 602: Calculate the second loss between the high-dimensional robot motion command output by the high-dimensional motion reconstruction module and the high-dimensional motion vector.

[0067] Step 603: Calculate the divergence loss between the continuous probability distribution output by the encoder module and the preset standard probability distribution.

[0068] Step 604: Construct a joint loss function based on the first loss, the second loss, and the divergence loss.

[0069] Step 605: Minimize the joint loss function using supervised gradient descent, and iteratively train the encoder module, visual language action model, and high-dimensional action reconstruction module until the joint loss function converges to the preset total loss threshold.

[0070] In step 601 of some embodiments, it is necessary to calculate the numerical difference between the predicted low-dimensional latent vector generated by the visual language action model based on the current environment image and language instructions, and the target low-dimensional latent vector generated by the encoder module based on real action data as the supervision ground truth; this difference is the first loss. The first loss can be called the prediction loss or consistency loss, and mathematically it can be measured using mean squared error (MSE) or cosine similarity distance. This calculation process aims to evaluate the policy network's understanding ability at the abstract feature level, that is, to determine whether the model can accurately deduce the latent intention features that match the expert demonstration actions based solely on visual and linguistic cues, thereby providing direct gradient signals for cross-modal alignment.

[0071] In step 602 of some embodiments, the physical parameter deviation between the high-dimensional robot motion commands decoded and generated by the high-dimensional motion reconstruction module and the original high-dimensional motion vector is calculated. This deviation can be determined by calculating the L2 norm distance or the L1 norm of the absolute error average of the two high-dimensional vectors in Euclidean space. Since the high-dimensional motion vector contains key control parameters such as the robot arm joint angles, angular velocities, and end-effector pose, the second loss directly reflects the physical fidelity of the motion reconstruction. This calculation process ensures that after compression and decompression, the generated commands can still drive the robot to reproduce fine movements that are highly consistent with the expert demonstration samples in terms of motion trajectory and mechanical properties, preventing the loss of motion details due to feature compression.

[0072] In step 603 of some embodiments, a statistical difference algorithm, such as KL divergence, can be used to calculate the statistical distance between the continuous probability distribution output by the encoder module and a preset standard probability distribution. Here, the continuous probability distribution is defined by the mean vector and log-variance vector generated by the encoder, while the standard probability distribution is typically set to an isotropic standard normal distribution. The divergence loss, as a regularization constraint, forces the generated low-dimensional latent vector distribution to approximate the standard normal distribution, preventing overfitting in the latent space and ensuring good smoothness and complete sampling characteristics, which is beneficial for subsequent generative tasks.

[0073] In step 604 of some embodiments, the calculated first loss, second loss, and divergence loss are integrated into a scalar joint loss function through linear weighted summation. During this process, specific hyperparameter weight coefficients can be introduced to balance the priorities of different optimization objectives according to task requirements. For example, a higher weight can be assigned to the second loss to prioritize the physical safety and accuracy of action execution, while appropriate weights can be assigned to the first loss and divergence loss to balance the correctness of intent understanding with the regularization characteristics of the latent space. The joint loss function unifies and integrates the three independent optimization dimensions of feature alignment, action reconstruction, and distribution regularization, providing a unique global gradient guidance direction for the collaborative optimization of multiple modules.

[0074] In step 605 of some embodiments, the gradient of the joint loss function with respect to all trainable parameters in the encoder module, visual language action model, and high-dimensional action reconstruction module is calculated using the backpropagation algorithm, and the weights of these modules are updated synchronously using supervised gradient descent. This iterative training process is repeated on a massive action dataset, continuously fine-tuning the parameters of each module to find the global optimum, until the value of the joint loss function decreases and stabilizes within a preset total loss threshold. This signifies that the model has reached the optimal balance between feature compression efficiency, intent prediction accuracy, and action reconstruction accuracy.

[0075] Through steps 601 to 605 above, this embodiment of the application solves the target mismatch problem that may be caused by staged training by constructing a composite loss function that includes prediction consistency, action reconstruction accuracy, and distribution regularization constraints. Joint training forces the latent space generated by the encoder to simultaneously satisfy the two conditions of "easy to be predicted by the policy network" and "easy to be reconstructed by the decoder," thereby significantly improving the representation quality of latent features. The model finally trained not only has high action execution accuracy (guaranteed by reconstruction loss) but also cross-modal generalization ability (guaranteed by prediction loss and divergence loss), which can efficiently transform natural language instructions and visual environment into robot control policies.

[0076] Through steps 201 to 205 above, this embodiment first uses an encoder to compress complex high-dimensional robotic arm movements into low-dimensional latent vectors, reducing the learning difficulty of the visual language action model and enabling it to focus on understanding the logical connection between visual semantics and action intent. Then, the high-dimensional action reconstruction module accurately restores the predicted low-dimensional latent vectors into fine control instructions, ensuring the physical accuracy of robot operation. Finally, through a joint training mechanism, the model can improve the robot's success rate in unstructured environmental tasks and its generalization response to new instructions while ensuring action representation capabilities.

[0077] In some embodiments, the encoder module and the high-dimensional motion reconstruction module are constructed as modular components independent of the core inference network of the visual language action model. The visual language action model serves as a general intent inference backbone, focusing on cross-modal semantic understanding and policy planning, without being limited by the kinematic parameters of specific hardware. Together, the encoder module and the high-dimensional motion reconstruction module constitute a general adaptation layer connecting the abstract policy space and the physical control space. This adaptation layer acts as a standardized interface, shielding the underlying differences between different robot hardware, enabling the upper backbone network to operate within a unified low-dimensional latent space.

[0078] During the adaptation process, the general adaptation layer is responsible for bidirectional feature transformation. When faced with target robots with different degrees of freedom, joint configurations, or dynamic characteristics, only the corresponding encoder module and high-dimensional motion reconstruction module need to be trained for the motion data of that specific target robot, without needing to retrain the massive visual language motion model backbone. The encoder module is responsible for compressing the high-dimensional ontology perception data of the target robot into general latent features that the model can understand, while the high-dimensional motion reconstruction module is responsible for accurately mapping the general latent prediction vector output by the model back to the specific high-dimensional control commands required by the target robot. This mechanism allows the same visual language motion model to flexibly control various types of target robots by attaching different adaptation layer parameters.

[0079] Through the above embodiments, this application embodiment isolates the robot's specific kinematic features in a general adaptation layer, decoupling general reasoning capabilities from specific execution capabilities, thereby reducing the training cost and data requirements when transferring visual language action models to new types of robots.

[0080] Please see Figure 7 This application also provides a robot control device based on a visual language action model, which can implement the above-mentioned robot control method based on a visual language action model, including: The acquisition module is used to obtain visual observation information, target language instructions, and robot body status information; The reasoning module is used to obtain a predicted low-dimensional potential vector from visual observation information, target language instructions, and robot body state information through a pre-trained visual language action model. The reconstruction module is used to reconstruct the predicted low-dimensional latent vectors into robot action commands through a pre-trained high-dimensional action reconstruction module; wherein the dimension of the robot action command matches the dimension of the target robot's action space. The drive module is used to drive the target robot to execute high-dimensional robot motion commands.

[0081] Please see Figure 8 , Figure 8 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 801 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 802 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and is called and executed by the processor 801. The 803 input / output interface is used to implement information input and output. The communication interface 804 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 805 transmits information between various components of the device (e.g., processor 801, memory 802, input / output interface 803, and communication interface 804); The processor 801, memory 802, input / output interface 803, and communication interface 804 are connected to each other within the device via bus 805.

[0082] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0083] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0084] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0085] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0086] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0087] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0088] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0089] It should be understood that in this application, "at least one" and "several" refer to one or more, and "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0090] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0091] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0092] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0093] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0094] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A robot control method based on a visual language action model, characterized by, The method comprises: obtaining visual observation information of a target robot, target language instructions, and robot body state information; inference is performed on the visual observation information, the target language instructions, and the robot body state information by a pre-trained visual language action model to obtain a predicted low-dimensional latent vector; the predicted low-dimensional latent vector is reconstructed into a robot action instruction by a pre-trained high-dimensional action reconstruction module; wherein the dimension of the robot action instruction matches the action space dimension of the target robot; the target robot is driven to execute the high-dimensional robot action instruction. 2.The robot control method based on visual language action model according to claim 1, wherein, The pre-training process of the visual language action model and the high-dimensional action reconstruction module comprises: obtaining a high-dimensional action data set comprising a plurality of high-dimensional action vectors, and training visual observation information and training language instructions that are time-aligned with the high-dimensional action vectors; the high-dimensional action vectors are compressed by an encoder module to generate a target low-dimensional latent vector; the visual language action model is trained with the target low-dimensional latent vector as the training target and the training visual observation information and the training language instructions as the training input, so that the predicted low-dimensional latent vector output by the visual language action model approximates the target low-dimensional latent vector; the high-dimensional action reconstruction module is trained with the high-dimensional action vectors as the reconstruction target and the target predicted low-dimensional latent vector as the training input, so that the high-dimensional robot action instruction output by the high-dimensional action reconstruction module approximates the high-dimensional action vectors; the encoder module, the visual language action model, and the high-dimensional action reconstruction module are jointly trained based on the difference between the reconstructed high-dimensional robot action instruction and the high-dimensional action vectors. 3.The robot control method based on visual language action model according to claim 2, wherein, The high-dimensional action vectors are compressed by the encoder module to generate a target low-dimensional latent vector, comprising: the high-dimensional action vectors are mapped to a continuous probability distribution in a low-dimensional latent space by a pre-set multi-layer nonlinear transformation matrix of the encoder module; the target low-dimensional latent vector is generated by sampling from the continuous probability distribution. 4.The robot control method based on visual language action model according to claim 2, wherein, The visual language action model is trained with the target low-dimensional latent vector as the training target and the training visual observation information and the training language instructions as the training input, so that the predicted low-dimensional latent vector output by the visual language action model approximates the target low-dimensional latent vector, comprising: multimodal fusion features are obtained by multimodal fusion understanding of the training visual observation information and the training language instructions by the visual language action model; the multimodal fusion features are mapped to the predicted low-dimensional latent vector, and the difference between the predicted low-dimensional latent vector and the target low-dimensional latent vector is taken as a loss to train the visual language action model iteratively until the difference between the predicted low-dimensional latent vector and the target low-dimensional latent vector converges to a pre-set first loss threshold. 5.The robot control method based on visual language action model according to claim 2, wherein, The high-dimensional action reconstruction module is trained by taking the high-dimensional action vector as a reconstruction target and taking the target predicted low-dimensional latent vector as a training input, so that the high-dimensional robot action instruction output by the high-dimensional action reconstruction module approximates the high-dimensional action vector, including: The target predicted low-dimensional latent vector is up-sampled by a pre-set up-sampling matrix in the high-dimensional action reconstruction module to obtain the high-dimensional robot action instruction; The high-dimensional action reconstruction module is iteratively trained by taking the difference between the high-dimensional robot action instruction and the high-dimensional action vector as a loss until the difference converges to a pre-set second loss threshold. 6.The robot control method based on visual language action model according to claim 3, wherein, The encoder module, the visual language action model and the high-dimensional action reconstruction module are jointly trained based on the difference between the reconstructed high-dimensional robot action instruction and the high-dimensional action vector, including: A first loss between the predicted low-dimensional latent vector output by the visual language action model and the target low-dimensional latent vector is calculated; A second loss between the high-dimensional robot action instruction output by the high-dimensional action reconstruction module and the high-dimensional action vector is calculated; A divergence loss between the continuous probability distribution output by the encoder module and a pre-set standard probability distribution is calculated; A joint loss function is constructed according to the first loss, the second loss and the divergence loss; The encoder module, the visual language action model and the high-dimensional action reconstruction module are iteratively trained by minimizing the joint loss function through a supervised gradient descent method until the joint loss function converges to a pre-set total loss threshold. 7.The robot control method based on visual language action model according to claim 2, wherein, The encoder module and the high-dimensional action reconstruction module are designed to be decoupled from the backbone architecture of the visual language action model, and the encoder module and the high-dimensional action reconstruction module together form a general adaptation layer to adapt the visual language action model so that it can control the target robot.

8. A robot control device based on a visual language action model, characterized by, The device includes: An acquisition module configured to acquire visual observation information of a target robot, a target language instruction and robot body state information; An inference module configured to infer the visual observation information, the target language instruction and the robot body state information through a pre-trained visual language action model to obtain a predicted low-dimensional latent vector; A reconstruction module configured to reconstruct the predicted low-dimensional latent vector into a robot action instruction through a pre-trained high-dimensional action reconstruction module, wherein the dimension of the robot action instruction matches the action space dimension of the target robot; A driving module configured to drive the target robot to execute the high-dimensional robot action instruction.

9. An electronic device, comprising: Including: A memory and a processor, the memory stores a computer program, and the processor implements the robot control method based on the visual language action model in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The storage medium stores a program, and the program is executed by the processor to implement the robot control method based on the visual language action model in any one of claims 1 to 7.

Images