A robot control method based on a lightweight multi-modal large model

Abstract

This invention provides a robot control method based on a lightweight multimodal large model, relating to the fields of embodied intelligence, robot control, and multimodal large model technology. Addressing the problems of low computational efficiency, high training resource requirements, and high data dependency in practical deployments of existing vision-language-action models, this invention constructs a sparse activation control submodule, selectively activating certain network layers or attention branches in the pre-trained multimodal backbone network; and constructs a temporal action decoding module that fuses current multimodal hidden layer features with historical temporal context features to generate robot action commands. This invention also employs a two-stage training method that decouples offline feature extraction and action policy learning, caching multimodal hidden layer features and training the temporal action decoding module based on this cache, thereby reducing redundant backbone forward computation and gradient backpropagation.

Description

Technical Field

[0001] This invention belongs to the fields of embodied intelligence, robot control and multimodal large model technology, and particularly relates to a robot control method based on a lightweight multimodal large model. Background Technology

[0002] Vision-Language-Motion (VLA) models are a core technology for realizing universal control of embodied intelligent robots. They can directly map visual observations and natural language commands into action commands for the robot, enabling it to complete complex real-world tasks. In recent years, technologies such as RT-2, Diffusion-VLA, and OpenVLA have emerged as key advancements in this field. Representative works such as these have continuously refreshed the task performance benchmarks of VLA models, demonstrating the enormous application potential of this technology in the field of robotics.

[0003] However, existing mainstream VLA models generally suffer from three major technical bottlenecks, severely restricting their deployment and application in real-world scenarios: First, extremely high computational resource requirements. Existing model training typically requires multi-GPU / TPU clusters. For example, OpenVLA requires 512 TPUs-v4 hours of computational resources for a single training round, and Diffusion-VLA's diffusion process requires no less than 100 sampling computations, which is difficult for ordinary academic institutions and industrial scenarios to support. Second, excessive GPU memory consumption. Traditional end-to-end training methods require gradient backpropagation throughout the backbone network, resulting in extremely high GPU memory consumption during training. Typically, a professional computing card with more than 32GB of GPU memory is needed to complete training, making deployment extremely difficult. Third, high data dependence. Existing VLA models typically require large-scale datasets of more than 10,000 demonstration trajectories to complete effective training, while the cost of collecting high-quality robot demonstration data is extremely high, further limiting the practical application of the models.

[0004] Therefore, establishing a lightweight, low-resource-demand, and low-data-dependency robot control method to solve the core problems of high deployment threshold and low efficiency in existing technologies has become a key technical problem that urgently needs to be solved in the field of embodied intelligence. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a robot control method based on a lightweight multimodal large model, comprising the following steps:

[0006] Acquire the robot's historical action sequences, current visual observation images, and natural language commands;

[0007] Construct a lightweight vision-language-action model, train the lightweight vision-language-action model, and obtain a trained lightweight vision-language-action model;

[0008] The robot's historical action sequence, current visual observation image, and current natural language command are input into the trained lightweight vision-language-action model to obtain action commands; the robot is then controlled to perform the operation tasks corresponding to the natural language commands based on the action commands.

[0009] The lightweight vision-language-action model includes a visual encoder, a text segmenter, a multimodal fusion module, and a temporal action decoding module. The visual encoder performs spatial-aware encoding on the current visually observed image and outputs a visual token sequence. The text segmenter segments the current natural language instruction and outputs a text token sequence. The multimodal fusion module extracts the current multimodal hidden features based on the visual token sequence and the text token sequence. The temporal action decoding module decodes the current multimodal hidden features to generate action instructions.

[0010] Furthermore, the visual encoder is constructed based on a spatially perceptive visual coding network;

[0011] The specific method for the visual encoder to perform spatially perceptual encoding on the current visual observation image is as follows: the current visual observation image is divided into blocks or feature mapping encoding to obtain a visual token sequence.

[0012] Furthermore, the multimodal fusion module includes a token splicing submodule, a pre-trained multimodal backbone network, and a sparse activation control submodule;

[0013] The Token splicing submodule performs semantic alignment and joint encoding on the visual token sequence and the text token sequence through at least one of attention fusion, cross attention fusion or route attention fusion, and inputs them into the pre-trained multimodal backbone network to generate the current multimodal hidden layer features based on the pre-trained multimodal backbone network.

[0014] The pre-trained multimodal backbone network is a multimodal backbone network capable of processing visual token sequences and text token sequences, including multiple network layers or multiple attention branches, each network layer and attention branch including corresponding learnable hybrid weights;

[0015] The sparse activation control submodule is used to selectively enable certain network layers or attention branches in a multimodal backbone network.

[0016] Furthermore, the specific method for selectively activating certain network layers or attention branches in the multimodal backbone network is as follows:

[0017] Based on the layer selection strategy, task input features, or learnable hybrid weights, activation labels corresponding to each network layer or attention branch of the backbone network are generated. The activated network layer or activated attention branch is selected based on the activation labels. Forward computation is performed on the activated network layer or activated attention branch, and the inactive network layer or inactive attention branch is skipped.

[0018] The layer selection strategy includes: setting a threshold. Enable layer index greater than threshold For network layers, the layer index should be less than or equal to a threshold. The weights of the network layers are set to 0;

[0019] The task input features are obtained by extracting and aggregating features from the current visual observation image, the current natural language command, and the historical action sequence.

[0020] Furthermore, the temporal action decoding module includes a temporal context encoding unit, a feature fusion unit, and an action mapping unit;

[0021] Temporal context coding unit is used to temporally encode historical action sequences to obtain temporal context features;

[0022] The feature fusion unit is used to fuse the current multimodal hidden layer features and temporal context features to obtain the fused features;

[0023] The action mapping unit is used to output action commands based on the fused features;

[0024] The temporal context coding unit includes an embedding layer and a temporal encoder, which can be any one of a Transformer encoder, a recurrent neural network, a temporal convolutional network, or a gated temporal network.

[0025] Furthermore, the specific method for temporal encoding of historical action sequences is as follows:

[0026] The embedding layer is used to process the time steps from the first time step to the second time step. Time-step robot historical action sequence Embedding is performed to obtain the intermediate sequence;

[0027] Temporal features are extracted from the intermediate sequence using a temporal encoder to obtain a token sequence; the last token in the token sequence is selected as the temporal context feature.

[0028] Furthermore, when the timing action decoding module decodes the current multimodal hidden features, it decodes the current multimodal hidden features based on one of the lightweight decoding scheme and the high-performance decoding scheme, and maps the current multimodal hidden layer features to the action commands of the robot end effector.

[0029] Furthermore, the lightweight decoding scheme includes: skipping the temporal context encoding unit and the feature fusion unit, and mapping the current multimodal hidden features into fixed-dimensional action instructions based on the action mapping unit.

[0030] Furthermore, the high-performance decoding scheme includes: performing temporal encoding on the historical action sequence based on the temporal context encoding unit to obtain temporal context features; fusing the current multimodal hidden layer features and temporal context features based on the feature fusion unit to obtain fused features; concatenating the current multimodal hidden layer features and temporal context features based on the action mapping unit to obtain concatenated features; and mapping the concatenated features to fixed-dimensional action instructions.

[0031] Furthermore, the specific method for training the lightweight vision-language-action model is as follows:

[0032] A dataset is constructed based on several robot trajectories and action labels. Data augmentation is then performed on the dataset to obtain a preprocessed dataset. Each trajectory includes an RGB visual observation image acquired at a preset frequency, the pose and gripper state of a 7-DOF end effector, and natural language annotations for two atomic actions. Data augmentation includes at least one of the following: temporal augmentation, viewpoint synthesis augmentation, image perturbation augmentation, or action trajectory perturbation augmentation.

[0033] Set a freezing threshold to freeze the parameters of the pre-trained multimodal backbone network, so that the total proportion of frozen parameters exceeds the freezing threshold; perform forward inference based on the preprocessed dataset to extract and cache the corresponding multimodal hidden layer features; store the multimodal hidden layer features offline in the form of tensor files, key-value databases, feature cache tables or fragmented feature files, and establish corresponding identifiers;

[0034] Read the cached multimodal hidden layer features, use action labels as supervision signals, and train the temporal action decoding module or train the trainable parameters in the sparse activation control submodule of the temporal action decoding module and the multimodal fusion module.

[0035] Set optimization objectives and training rounds. The optimization objectives include action prediction loss and parameter regularization terms. The action prediction loss is used to constrain the difference between the predicted action and the real action label. The parameter regularization terms are used to constrain the trainable parameters in the temporal action decoding module or the sparse activation control submodule.

[0036] After reaching the maximum number of training rounds, Hausdorff distance is used as an example evaluation metric to evaluate the similarity between the trajectory of the action command execution task generated based on the lightweight vision-language-action model and the actual demonstration trajectory, thereby quantifying the task success rate.

[0037] Based on the evaluation results, a well-trained lightweight vision-language-action model is obtained.

[0038] The beneficial effects of adopting the above technical solution are as follows: The robot control method based on a lightweight multimodal large model provided by this invention selectively activates some network layers or attention branches in the pre-trained multimodal backbone network through a sparse activation control module, which can reduce unnecessary multimodal fusion computation and reduce computational overhead during inference or training. Through a training method that decouples multimodal hidden layer feature caching from action policy learning, cached features can be reused when the temporal action decoding module structure is adjusted or parameters are optimized, reducing repeated backbone forward computation and gradient backpropagation, thereby reducing memory usage and training costs. By fusing current multimodal hidden layer features with historical temporal context features in the temporal action decoding module, the continuity of actions and trajectory stability in continuous control tasks can be improved. This invention can be combined with data augmentation methods such as temporal enhancement, viewpoint synthesis enhancement, image perturbation enhancement, or action trajectory perturbation enhancement to improve the utilization efficiency of robot demonstration data and reduce dependence on large-scale demonstration data. Furthermore, the decoupled training method of this invention allows the structural search and iterative optimization of the temporal action decoding module to be based on cached multimodal hidden layer features, which helps reduce the experimental costs between different temporal action decoding module structures. Attached Figure Description

[0039] Figure 1 Flowchart of a robot control method based on a lightweight multimodal large model provided in Embodiment 1 of the present invention;

[0040] Figure 2 The training flowchart for a lightweight vision-language-action model provided in Embodiment 1 of the present invention;

[0041] Figure 3 The schematic diagram of the operation task effect corresponding to executing different natural language instructions provided in Embodiment 1 of the present invention is as follows: Figure 3 As shown, (a) is the robot trajectory corresponding to the instruction to open the drawer, (b) is the robot trajectory corresponding to the instruction to close the drawer, (c) is the robot trajectory corresponding to the instruction to pick up the frying pan and place it in the sink area, (d) is the robot trajectory corresponding to the instruction to take the pot out of the sink and place it on the countertop, (e) is the robot trajectory corresponding to the instruction to grab the frying pan and place it steadily on the stove, and (f) is the robot trajectory corresponding to the instruction to remove the frying pan from the stove and put it back on the countertop. Detailed Implementation

[0042] The specific implementation methods of this application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0043] It should be understood and explained that the following embodiments are only used to illustrate the technical solutions of this application and not to limit it. Although this application has been described in detail with reference to the following embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of this application. Any modifications or equivalent substitutions that do not depart from the scope of this application should be covered within the protection scope of this application.

[0044] In this specification, the visual token sequence refers to the visual feature sequence obtained by encoding the robot's visual observation image by the visual encoding module; the text token sequence refers to the text feature sequence obtained by encoding natural language instructions; the current multimodal hidden layer feature refers to the fused feature obtained by the multimodal fusion module based on the current visual token sequence and the text token sequence; the temporal context feature refers to the context feature obtained by encoding the historical state sequence or historical action sequence; and the temporal action decoding module refers to the decoding module used to fuse the current multimodal hidden layer feature and the temporal context feature and output the robot's action instructions.

[0045] Example 1:

[0046] A robot control method based on a lightweight multimodal large model, such as Figure 1 As shown, it includes the following steps:

[0047] Step 1: Acquire the robot's historical action sequence, current visual observation image, and current natural language command;

[0048] Step 2: Construct a lightweight vision-language-action model, train the lightweight vision-language-action model to obtain a trained lightweight vision-language-action model;

[0049] Step 3: Input the robot's historical action sequence, current visual observation image, and current natural language command into the trained lightweight vision-language-action model to obtain action commands; control the robot's end effector to execute the operation task corresponding to the natural language command according to the action commands.

[0050] In step 2, the lightweight vision-language-action model includes a visual encoder, a text segmenter, a multimodal fusion module, and a temporal action decoding module. The visual encoder performs spatial-aware encoding on the current visually observed image, outputting a visual token sequence; the text segmenter segments the current natural language command, outputting a text token sequence; the multimodal fusion module extracts the current multimodal hidden layer features based on the visual token sequence and the text token sequence; and the temporal action decoding module maps the current multimodal hidden layer features to action commands for the robot's end effector, achieving closed-loop vision-language-action control.

[0051] The visual encoder is constructed based on a spatially perceptive visual coding network. The specific method of the visual encoder for spatially perceptual coding of the current visual observation image is as follows: the current visual observation image is divided into blocks for coding or feature mapping coding, and a visual token sequence containing spatial location information is output to provide visual feature input with spatial details for the subsequent multimodal fusion module.

[0052] The multimodal fusion module includes a token splicing submodule, a pre-trained multimodal backbone network, and a sparse activation control submodule. The token splicing submodule performs semantic alignment and joint encoding on the visual token sequence and the text token sequence through at least one of attention fusion, cross attention fusion, or route attention fusion, and inputs it into the pre-trained multimodal backbone network to generate the current multimodal hidden layer features, providing a multimodal semantic foundation for subsequent action decoding.

[0053] The pre-trained multimodal backbone network is a multimodal backbone network capable of processing visual token sequences and text token sequences. It includes multiple network layers or multiple attention branches, and each network layer and attention branch includes corresponding learnable hybrid weights.

[0054] The sparse activation control submodule is used to selectively enable certain network layers or attention branches in a multimodal backbone network to reduce unnecessary forward computation;

[0055] Specifically, the sparse activation control submodule generates activation labels corresponding to each network layer or attention branch of the backbone network based on the layer selection strategy, task input features, or learnable hybrid weights. It then selects the activated network layer or attention branch based on the activation labels, performs forward computation on the activated network layer or attention branch, and skips the inactive network layer or attention branch, thereby reducing the computational overhead and memory usage during model inference and training.

[0056] The layer selection strategy includes setting a threshold. Enable layer index greater than threshold For network layers, the layer index should be less than or equal to a threshold. The weights of the network layers are set to 0;

[0057] The task input features are obtained by feature extraction and aggregation of the current visual observation image, the current natural language command, and the historical action sequence;

[0058] The temporal action decoding module includes a temporal context encoding unit, a feature fusion unit, and an action mapping unit. The temporal context encoding unit encodes historical action sequences temporally to obtain temporal context features. The feature fusion unit fuses current multimodal hidden layer features and temporal context features to obtain fused features. The action mapping unit outputs action commands based on the fused features. The temporal context encoding unit includes an embedding layer and a temporal encoder. The temporal encoder can be any of a Transformer encoder, a recurrent neural network, a temporal convolutional network, or a gated temporal network, and is used to capture the temporal correlation information of the robot's historical action sequences, improving the action coherence of continuous operation tasks.

[0059] The specific method by which the timing action decoding module decodes the current multimodal hidden features is as follows:

[0060] The current multimodal hidden features are decoded using either a lightweight decoding scheme or a high-performance decoding scheme, and the current multimodal hidden layer features are mapped to action commands of the robot's end effector.

[0061] The lightweight decoding scheme includes: skipping the temporal context encoding unit and the feature fusion unit, and mapping the current multimodal hidden features into fixed-dimensional action instructions based on the action mapping unit.

[0062] The high-performance decoding scheme includes: performing temporal encoding on historical action sequences based on a temporal context encoding unit to obtain temporal context features; fusing current multimodal hidden layer features and temporal context features based on a feature fusion unit to obtain fused features; concatenating current multimodal hidden layer features and temporal context features based on an action mapping unit to obtain concatenated features; and mapping the concatenated features to fixed-dimensional action instructions.

[0063] The visual encoder in this embodiment is built based on the SigLIP visual word segmenter, and the pre-trained multimodal backbone network is preferably a Janus-type multimodal backbone network. This embodiment uses the SigLIP visual word segmenter and a Janus-type multimodal backbone network to construct a lightweight vision-language-action model. In other embodiments, the SigLIP visual word segmenter and a Janus-Pro multimodal backbone network can be used as lower-level implementations, but this embodiment is not limited to this specific model combination. The temporal action decoding module in this embodiment is built based on Transformer-MLP hybrid decoding; the architecture includes a visual encoder, a multimodal fusion module, and a temporal action decoding module. The mathematical expression is:

[0064] ;

[0065] in, In this embodiment, the SigLIP visual word segmenter serves as the visual encoder. In this embodiment, the Janus-Pro multimodal backbone network serves as the multimodal fusion module. This is one implementation of the action mapping unit in the timing action decoding module of this embodiment. This is a function composition operator, representing the sequential execution of each network module, with the output of the previous module serving as the input of the next. The specific network names, dimensions, and degrees of freedom for output actions mentioned above are parameters of preferred embodiments and do not constitute a limitation on the scope of protection of this application.

[0066] This embodiment of the visual encoder abandons the traditional ViT segmentation method and adopts the spatially perceptual segmentation scheme of the SigLIP visual segmenter to encode the input RGB observation image of the robot. The size of the input visual encoder is... RGB current visual observation image The output visual token sequence is :

[0067] ;

[0068] in, For the first A visual token The total number of visual tokens. The spatially-aware word segmentation scheme achieves a 4-fold improvement in spatial resolution with the same computational overhead as the CLIP-style word segmenter, preserving richer spatial detail information for robot operation tasks.

[0069] The multimodal fusion module in this embodiment uses the Janus-Pro-1.3B multimodal backbone network, retaining the Janus architecture's native ability to process arbitrary modalities, while optimizing the spatiotemporal sparse activation mode for robot control tasks. The Janus-Pro-1.3B multimodal backbone network performs multimodal fusion of the visual token sequence and the text token sequence of natural language instructions through dynamic attention routing to obtain the current multimodal hidden layer features. As shown in the formula below:

[0070] ;

[0071] in, For the first Learnable hybrid weights of layers This represents the total number of layers in the Janus-Pro-1.3B multimodal backbone network. For visual token sequences, It is a sequence of text tokens.

[0072] To reduce computational overhead, the sparse activation control submodule in this embodiment uses a layer selection strategy based on dynamic sparse activation to apply learnable hybrid weights. The sparsity treatment is performed as shown in the following formula:

[0073] ;

[0074] in, This embodiment sets the layer index for the Janus-Pro-1.3B multimodal backbone network. This means that only the last 12 Transformer layers of the Janus-Pro-1.3B multimodal backbone network are used for computation, while the weights of the first 12 layers are reset to zero. Compared with using the entire backbone network, this strategy reduces the floating-point computation by 41% when training only the temporal action decoding module, while retaining the core capability of multimodal fusion.

[0075] This embodiment provides two preferred timing action decoding module decoding schemes: a lightweight decoding scheme and a high-performance decoding scheme. These schemes are suitable for both ultra-lightweight and high-performance application scenarios, and ultimately output the pose and gripper state commands of the robot end effector. The specific output dimensions can be adjusted according to the degrees of freedom of the robot end effector.

[0076] The specific method of the lightweight decoding solution is as follows:

[0077] This embodiment directly maps the 2048-dimensional multimodal hidden layer features into 7-dimensional action instructions based on an MLP-only decoder. The calculation method is as follows:

[0078] ;

[0079] in, , and These are the three layers of parameters for the MLP-only decoder. The MLP-only decoder contains only 2.62 million trainable parameters, and its floating-point operation volume is only 0.15G at a 30Hz inference frequency, enabling real-time inference on embedded platforms.

[0080] The specific method of the high-performance decoding scheme is as follows:

[0081] This embodiment decodes 2048-dimensional multimodal hidden layer features based on a Transformer enhanced decoder. The high-performance decoding scheme, building upon MLP decoding, incorporates temporal context features from the robot's historical action sequences to improve the coherence of trajectory execution and task stability. The feature is fused with multimodal hidden layer features, and the action command is output through the MLP layer, as shown in the following formula:

[0082] ;

[0083] in, for Action commands for time steps, For the first time step to the second The robot's historical action sequence in time steps. This is a timing coding operation;

[0084] Among them, the robot's historical action sequence The specific process for timing coding is as follows:

[0085] First, the embedding layer is used to process time steps from the first time step to the second time step. Time-step robot historical action sequence Embedding is performed to obtain the intermediate sequence. :

[0086] ;

[0087] Then, the intermediate sequence is processed by a time encoder. Temporal feature extraction is performed to obtain the token sequence. :

[0088] ;

[0089] Select token sequence The last token in the sequence is used as a temporal context feature. :

[0090] ;

[0091] The number of parameters and floating-point operations of the Transformer enhanced decoder can be adjusted according to the number of layers, hidden dimension, and action output dimension of the temporal encoder. In the configuration of this embodiment, the decoder contains 3.94 million trainable parameters, and the floating-point operation is 7.23G at a frequency of 30Hz and 300 time steps.

[0092] like Figure 2 As shown, the specific method for training the lightweight vision-language-action model is as follows:

[0093] S1. A dataset is constructed based on several robot trajectories and action labels. Data augmentation is performed on the dataset to obtain a preprocessed dataset. Each trajectory includes an RGB visual observation image acquired at a preset frequency, the pose and gripper state of a 7-DOF end effector, and natural language annotations for two atomic actions. Data augmentation includes at least one of temporal augmentation, viewpoint synthesis augmentation, image perturbation augmentation, or action trajectory perturbation augmentation.

[0094] S2, set the freeze threshold to 80%, freeze the parameters of the pre-trained multimodal backbone network so that the total proportion of frozen parameters exceeds the freeze threshold; perform forward inference based on the preprocessed dataset, extract and cache the corresponding multimodal hidden layer features;

[0095] Multimodal hidden layer features are stored offline in the form of tensor files, key-value databases, feature cache tables, or fragmented feature files, and corresponding identifiers are established.

[0096] S3, read the cached multimodal hidden layer features, and use the robot action labels as supervision signals to train the trainable parameters in the sparse activation control submodule of the temporal action decoding module or the multimodal fusion module; wherein, the cached multimodal hidden layer features are used to avoid repeatedly executing the forward computation and gradient backpropagation of the pre-trained multimodal backbone network during the structural iteration or parameter optimization of the temporal action decoding module.

[0097] The optimization objective is set, including action prediction loss and parameter regularization term, wherein the action prediction loss is used to constrain the difference between the predicted action and the real action label, and the parameter regularization term is used to constrain the trainable parameters in the temporal action decoding module or the sparse activation control submodule.

[0098] This embodiment provides a lightweight two-stage decoupled training process for vision-language-action models, such as... Figure 2 As shown, the specific implementation process is illustrated using the UCSD Kitchens dataset and a single NVIDIA A30 Tensor Core GPU as an example. The dataset, hardware platform, training rounds, and learning rate are all configured in the preferred embodiment.

[0099] This embodiment uses the UCSD Kitchens dataset as the training benchmark. The UCSD Kitchens dataset contains 16,000 robot demonstration trajectories, each trajectory containing 15Hz. RGB visual observation images, 7-DOF end effector pose and gripper state, and natural language annotations for 2 atomic actions—the UCSD Kitchens dataset is only a fraction of the dataset required for traditional VLA models. To improve the small-data learning ability of lightweight vision-language-action models, temporal augmentation and viewpoint synthesis augmentation were performed on the UCSD Kitchens dataset to obtain a preprocessed training set. :

[0100] ;

[0101] in, For timing enhancement, For perspective fusion items, A set of perspective changes For trajectory samples after time warping, A set of perspective changes From the perspective of The transformation number for timing enhancement is assigned a value. , representing 5 different temporal distortion and scaling parameters, corresponding to 5 sets of trajectories after time axis perturbation. Original trajectory go through The newly generated trajectory samples obtained by perspective transformation synthesis, namely perspective enhancement generated samples, effectively expand the diversity of training samples and improve the generalization ability of the model through this enhancement strategy.

[0102] Load the model weights of the pre-trained Janus-Pro-1.3B multimodal backbone network and freeze them completely. Perform forward inference on all samples of the pre-processed UCSD Kitchens dataset to pre-compute the 2048-dimensional multimodal hidden layer features corresponding to each trajectory.

[0103] ;

[0104] in, The total number of trajectories in the dataset. The trajectory corresponds to a visual-text multimodal input. Pre-computed multimodal hidden layer features are stored as .pt files with bfloat32 precision, occupying a total storage space of 157.13GB, completing the construction of the feature cache data. This stage can be executed only once; subsequent optimization and architecture iteration of the temporal action decoding module can reuse this feature cache data, eliminating the need to repeatedly perform feature extraction from the pre-trained multimodal backbone network. The pre-computed feature cache data is loaded to optimize the trainable parameters of the temporal action decoding module, eliminating the need to load and backpropagate the Janus-Pro-1.3B multimodal backbone network, thus reducing memory usage and computational overhead during training. In other embodiments, the trainable parameters in the temporal action decoding module and the sparse activation control submodule can also be trained jointly. The optimization objective of this embodiment is the mean squared error loss function combined with L2 regularization, calculated as follows:

[0105] ;

[0106] in, These are the trainable parameters for the timing action decoding module. This is the mapping function for the timing action decoding module. For the pre-calculated first Multimodal hidden layer features corresponding to visual-text multimodal inputs for each trajectory For the first The real action labels for the visual-text multimodal input corresponding to each trajectory. is the L2 regularization coefficient.

[0107] This embodiment uses the AdamW optimizer, with a batch size of 256, an initial learning rate of 3e-4, and 50 training epochs. The entire training process can be completed on a single NVIDIA A30 GPU. Testing shows that compared to traditional end-to-end training, this embodiment reduces GPU memory usage by 68%, from 12.4GB to 4.0GB, while significantly shortening the iteration time per epoch and substantially improving training efficiency.

[0108] This embodiment also provides a quantitative evaluation and verification method for lightweight vision-language-action models. Hausdorff distance (HD) is used as an exemplary evaluation metric to assess the similarity between the execution trajectory generated by the lightweight vision-language-action model and the actual demonstration trajectory, while simultaneously quantifying the task success rate. The Hausdorff distance evaluation metric is only used to illustrate the verification process of this embodiment and does not constitute a necessary limitation on the model structure or training method of this embodiment.

[0109] Hausdorff distance can simultaneously capture the spatial precision and temporal coherence of a trajectory, and its calculation method is as follows:

[0110] ;

[0111] in, For the generated execution trajectory, To realistically demonstrate the trajectory, The Euclidean distance between the points on the trajectory;

[0112] When the Hausdorff distance between the generated execution trajectory and the real trajectory satisfies When the distance threshold is reached, the task is considered successfully executed. .

[0113] Based on the evaluation method described above, the two decoder schemes were tested on the UCSD Kitchens dataset, and the results are as follows:

[0114] The lightweight decoding scheme based on the MLP-only baseline decoder achieved a task success rate of 56.4%, while the high-performance decoding scheme based on the Transformer Enhanced (TMLP) decoder achieved a task success rate of 70.7%. The results show that the TMLP decoder designed in this embodiment significantly improves the task execution success rate with only a small increase in computational overhead, achieving a balanced optimization of efficiency and performance.

[0115] Based on the robot control method based on a lightweight multimodal large model provided in this embodiment, the schematic diagram of the operation task effect corresponding to different natural language commands output from the UCSD Kitchens dataset is shown below. Figure 3 As shown, (a) is the robot trajectory corresponding to the instruction to open the drawer, (b) is the robot trajectory corresponding to the instruction to close the drawer, (c) is the robot trajectory corresponding to the instruction to pick up the frying pan and place it in the sink area, (d) is the robot trajectory corresponding to the instruction to take the pot out of the sink and place it on the countertop, (e) is the robot trajectory corresponding to the instruction to grab the frying pan and place it steadily on the stove, and (f) is the robot trajectory corresponding to the instruction to remove the frying pan from the stove and put it back on the countertop.

[0116] Example 2:

[0117] This embodiment proposes an electronic device, including: one or more processors, and a memory, wherein the memory is used to store instructions, and when the instructions are executed by the one or more processors, the one or more processors execute the robot control method based on a lightweight multimodal large model.

[0118] The electronic device can be a mobile phone, computer, or tablet computer, etc., and includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements the robot control method based on a lightweight multimodal large model as described in the embodiments. It is understood that the electronic device may also include input / output (I / O) interfaces and communication components.

[0119] The processor is used to execute all or part of the steps in the robot control method based on a lightweight multimodal large model as described in the above embodiments. The memory is used to store various types of data, which may include, for example, instructions for any application or method in the electronic device, as well as application-related data.

[0120] The processor can be implemented as an Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor, or other electronic components, and is used to execute the robot control method based on lightweight multimodal large model described in the above embodiments.

[0121] Example 3:

[0122] This embodiment proposes a computer-readable storage medium that stores executable instructions. When these instructions are executed, if they are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0123] The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the robot control method based on a lightweight multimodal large model described in the various embodiments of this application.

[0124] The aforementioned storage media include: flash memory, hard disk, multimedia card, card-type memory (e.g., SD (Secure Digital Memory Card) or DX (Memory Data Register, MDR) memory, etc.), random access memory (RAM), static random-access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, disk, optical disk, server, APP (Application) application store, and other media capable of storing program verification codes. These media store computer programs, which, when executed by a processor, can implement the various steps of the aforementioned robot control method based on a lightweight multimodal large model.

[0125] Example 4:

[0126] This embodiment proposes a computer program product, including a computer program or instructions, which, when executed by a processor, implements the robot control method based on a lightweight multimodal large model.

[0127] Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a computer program product.

[0128] The various embodiments in this application are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0129] The scope of protection of this application is not limited to the embodiments described above. Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from the scope and spirit of this disclosure. If such modifications and variations fall within the scope of this disclosure and its equivalents, then the intent of this disclosure also includes these modifications and variations.

Claims

1. A robot control method based on a lightweight multimodal large model, characterized in that, Includes the following steps: Acquire the robot's historical action sequences, current visual observation images, and natural language commands; Construct a lightweight vision-language-action model, train the lightweight vision-language-action model, and obtain a trained lightweight vision-language-action model; The robot's historical action sequence, current visual observation image, and current natural language command are input into the trained lightweight vision-language-action model to obtain action commands; the robot is then controlled to perform the operation tasks corresponding to the natural language commands based on the action commands. The lightweight vision-language-action model includes a visual encoder, a text segmenter, a multimodal fusion module, and a temporal action decoding module. The visual encoder performs spatial-aware encoding on the current visual observation image and outputs a visual token sequence. The text segmenter segments the current natural language instruction and outputs a text token sequence. The multimodal fusion module extracts the current multimodal hidden layer features based on the visual token sequence and the text token sequence. The timing action decoding module decodes the current multimodal hidden features and generates action instructions.

2. The robot control method based on a lightweight multimodal large model according to claim 1, characterized in that, The visual encoder is constructed based on a spatially perceptive visual coding network; The specific method for the visual encoder to perform spatially perceptual encoding on the current visual observation image is as follows: the current visual observation image is divided into blocks or feature mapping encoding to obtain a visual token sequence.

3. The robot control method based on a lightweight multimodal large model according to claim 1, characterized in that, The multimodal fusion module includes a token splicing submodule, a pre-trained multimodal backbone network, and a sparse activation control submodule; The Token splicing submodule performs semantic alignment and joint encoding on the visual token sequence and the text token sequence through at least one of attention fusion, cross attention fusion or route attention fusion, and inputs them into the pre-trained multimodal backbone network to generate the current multimodal hidden layer features based on the pre-trained multimodal backbone network. The pre-trained multimodal backbone network is a multimodal backbone network capable of processing visual token sequences and text token sequences, including multiple network layers or multiple attention branches, each network layer and attention branch including corresponding learnable hybrid weights; The sparse activation control submodule is used to selectively enable certain network layers or attention branches in a multimodal backbone network.

4. The robot control method based on a lightweight multimodal large model according to claim 3, characterized in that, The specific method for selectively activating certain network layers or attention branches in the multimodal backbone network is as follows: Based on the layer selection strategy, task input features, or learnable hybrid weights, activation labels corresponding to each network layer or attention branch of the backbone network are generated, and the activated network layer or attention branch is selected based on the activation labels. Perform forward computation on activated network layers or activated attention branches, and skip inactive network layers or inactive attention branches; The layer selection strategy includes: setting a threshold. Enable layer index greater than threshold For network layers, the layer index should be less than or equal to a threshold. The weights of the network layers are set to 0; The task input features are obtained by extracting and aggregating features from the current visual observation image, the current natural language command, and the historical action sequence.

5. The robot control method based on a lightweight multimodal large model according to claim 1, characterized in that, The temporal action decoding module includes a temporal context encoding unit, a feature fusion unit, and an action mapping unit; Temporal context coding unit is used to temporally encode historical action sequences to obtain temporal context features; The feature fusion unit is used to fuse the current multimodal hidden layer features and temporal context features to obtain the fused features; The action mapping unit is used to output action commands based on the fused features; The temporal context coding unit includes an embedding layer and a temporal encoder, which can be any one of a Transformer encoder, a recurrent neural network, a temporal convolutional network, or a gated temporal network.

6. The robot control method based on a lightweight multimodal large model according to claim 5, characterized in that, The specific method for temporal encoding of historical action sequences is as follows: The embedding layer is used to process the time steps from the first time step to the second time step. Time-step robot historical action sequence Embedding is performed to obtain the intermediate sequence; The token sequence is obtained by extracting temporal features from the intermediate sequence using a temporal encoder. The last token in the token sequence is selected as the temporal context feature.

7. A robot control method based on a lightweight multimodal large model according to claim 5, characterized in that, When the timing action decoding module decodes the current multimodal hidden features, it decodes the current multimodal hidden features based on one of the lightweight decoding scheme and the high-performance decoding scheme, and maps the current multimodal hidden layer features into action commands of the robot end effector.

8. A robot control method based on a lightweight multimodal large model according to claim 7, characterized in that, The lightweight decoding scheme includes: skipping the temporal context encoding unit and the feature fusion unit, and mapping the current multimodal hidden features into fixed-dimensional action instructions based on the action mapping unit.

9. A robot control method based on a lightweight multimodal large model according to claim 7, characterized in that, The high-performance decoding scheme includes: performing temporal encoding on historical action sequences based on a temporal context encoding unit to obtain temporal context features; fusing current multimodal hidden layer features and temporal context features based on a feature fusion unit to obtain fused features; concatenating current multimodal hidden layer features and temporal context features based on an action mapping unit to obtain concatenated features; and mapping the concatenated features to fixed-dimensional action instructions.

10. A robot control method based on a lightweight multimodal large model according to claim 1, characterized in that, The specific method for training a lightweight vision-language-action model is as follows: A dataset is constructed based on several robot trajectories and action labels. Data augmentation is then performed on the dataset to obtain a preprocessed dataset. Each trajectory includes an RGB visual observation image acquired at a preset frequency, the pose and gripper state of a 7-DOF end effector, and natural language annotations for two atomic actions. Data augmentation includes at least one of the following: temporal augmentation, viewpoint synthesis augmentation, image perturbation augmentation, or action trajectory perturbation augmentation. Set a freezing threshold to freeze the parameters of the pre-trained multimodal backbone network, so that the total proportion of frozen parameters exceeds the freezing threshold; perform forward inference based on the preprocessed dataset to extract and cache the corresponding multimodal hidden layer features; store the multimodal hidden layer features offline in the form of tensor files, key-value databases, feature cache tables or fragmented feature files, and establish corresponding identifiers; Read the cached multimodal hidden layer features, use action labels as supervision signals, and train the temporal action decoding module or train the trainable parameters in the sparse activation control submodule of the temporal action decoding module and the multimodal fusion module. Set optimization objectives and training rounds. The optimization objectives include action prediction loss and parameter regularization terms. The action prediction loss is used to constrain the difference between the predicted action and the real action label. The parameter regularization terms are used to constrain the trainable parameters in the temporal action decoding module or the sparse activation control submodule. After reaching the maximum number of training rounds, Hausdorff distance is used as an example evaluation metric to evaluate the similarity between the trajectory of the action command execution task generated based on the lightweight vision-language-action model and the actual demonstration trajectory, thereby quantifying the task success rate. Based on the evaluation results, a well-trained lightweight vision-language-action model is obtained.

Images