Multi-modal large model robot control method based on meta-learning fine tuning

Abstract

The invention relates to a multi-modal large model robot control method based on meta-learning fine tuning, and the method comprises the steps: carrying out the meta-learning pre-training of an OpenVLA model, and obtaining a pre-trained VLA model; performing task specific fine tuning to obtain a fine-tuned VLA model; and inputting the current visual data, the language cue word, the robot sensor data and the task target data graph into the fine-tuned VLA model, and outputting to obtain a decision control instruction for correspondingly controlling the working state of the robot. Compared with the prior art, the method has the advantages that meta-learning pre-training is carried out on diversified tasks, and universal features and strategies are extracted, so that the generalization ability of the model is improved; in addition, in a specific task, rapid fine tuning is performed through a small number of new samples, so that the model can adapt to new task requirements in a short time; and multi-modal information of vision, character cues and robot control is combined, so that the perception and decision-making capabilities of the model are improved, and the adaptability and operation efficiency of the robot in a complex environment are effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of robot control technology, and in particular to a multi-modal large-model robot control method based on meta-learning fine-tuning. Background Art

[0002] With the rapid development of robotics technology, robots are increasingly used in the fields of industry, medicine, and services. However, existing technologies still face many challenges in dealing with complex life scenarios. In modern life scenarios, robot control systems face the challenge of processing samples that are much larger than training samples. Existing multimodal large models (MLMS) often have insufficient generalization capabilities when dealing with various complex and changeable life scenarios. This is mainly reflected in the following aspects:

[0003] 1. Insufficient generalization ability: Existing large multimodal models often show poor adaptability when dealing with new scenarios and new tasks. Due to the limited and insufficient diversity of training data, the model is prone to overfitting when faced with new samples. For example, studies have shown that the performance of vision-language models in cross-domain tasks is significantly reduced, which shows that existing models have obvious deficiencies in generalization ability. [1] In practical applications,

[0004] 2. When the robot encounters an unseen environment or task, the existing model is usually unable to adapt effectively, resulting in operational failure or inefficiency [2] .

[0005] 3. Low fine-tuning efficiency: Existing model fine-tuning methods usually require a lot of computing resources and time when fine-tuning on large-scale datasets. This makes fast fine-tuning impractical in practical applications. Although current fine-tuning methods such as BERT and GPT have improved the performance of the model to a certain extent, their high computing overhead and long training process are still a major bottleneck. [3] For example, fine-tuning GPT-3 requires thousands of GPU hours of computation time, which is unrealistic in many practical scenarios. [4] .

[0006] 4. High complexity: The structure of large multimodal models is complex and has many parameters. The fine-tuning process requires very fine tuning and adjustment, which increases the difficulty and cost of implementation. For multimodal models, especially those that need to process multiple inputs such as vision, language, and motion control, the parameter tuning process is very complex and often requires a lot of experiments and debugging. [5] For example, when dealing with vision-language tasks, the model needs to optimize both the visual feature extraction and language understanding modules at the same time, which makes the tuning process complex and time-consuming. [6] .

[0007] 5. Lack of flexibility: When processing multimodal data, existing models are often over-optimized for specific tasks and data sets, and lack flexibility in new tasks and data sets. This results in the need for a large amount of retraining and tuning of the model when facing different tasks, making it difficult to achieve rapid deployment and adaptation in practical applications. [7] For example, the performance of some vision-language models on new tasks may be significantly lower than the performance on the training tasks, indicating that the model lacks the necessary adaptability and flexibility. [8] .

[0008] In summary, the existing large multimodal models have many shortcomings in terms of generalization ability, fine-tuning efficiency, complexity and flexibility, which limits their wide application in real life scenarios, resulting in robots being unable to quickly adapt to new tasks and new scenarios, and having difficulty performing related operations efficiently and accurately.

[0009] Prior art literature:

[0010] [1] A. Radford, J. Wu, C. Child, et al., "Learning Transferable Visual ModelsFrom Natural Language Supervision," in Proceedings of the International Conference on Machine Learning (ICML), 2021.

[0011] [2] Y.Bengio, A.Courville, P.Vincent, "Representation Learning: AReviewand New Perspectives," IEEE Transactions on Pattern Analysis and MachineIntelligence, vol.35, no.8, pp.1798-1828, 2013.

[0012] [3] J.Devlin, M.-W.Chang, K.Lee, K.Toutanova, "BERT: Pre-training of DeepBidirectional Transformers for Language Understanding," arXiv preprint arXiv:1810.04805, 2018.

[0013] [4] T. Brown, B. Mann, N. Ryder, et al., "Language Models are Few-ShotLearners," in Advances in Neural Information Processing Systems (NeurIPS), 2020.

[0014] [5] D.Bahdanau, K.Cho, Y.Bengio, "Neural Machine Translation by JointlyLearning to Align and Translate," arXiv preprint arXiv:1409.0473, 2014.

[0015] [6] H.Tan, M.Bansal, "LXMERT: Learning Cross-Modality EncoderRepresentations from Transformers," in Proceedings of the 2019Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019.

[0016] [7]CGAtkeson,AWMoore,S.Schaal,"Locally Weighted Learning,"Artificial Intelligence Review,vol.11,no.1-5,pp.11-73,1997.

[0017] [8] S. Antol, A. Agrawal, J. Lu, et al., "VQA: Visual Question Answering," in Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015. Summary of the invention

[0018] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a multimodal large-model robot control method based on meta-learning fine-tuning, which can improve the adaptability and operational efficiency of the robot in complex environments.

[0019] The purpose of the present invention can be achieved by the following technical solution: A multi-modal large model robot control method based on meta-learning fine-tuning, comprising the following steps:

[0020] S1. Perform meta-learning pre-training on the OpenVLA (Vision Language Action) model to obtain a pre-trained VLA model;

[0021] S2, performing task-specific fine-tuning on the pre-trained VLA model to obtain a fine-tuned VLA model;

[0022] S3, inputting the current visual data, language prompt words, robot sensor data and task target data graph into the fine-tuned VLA model, and outputting the decision control instructions;

[0023] S4. According to the decision control instructions, the working state of the robot is controlled accordingly.

[0024] Furthermore, the OpenVLA model in step S1 is composed of a visual encoder, a projector and a large language model (LLM). In the visual encoder part, OpenVLA uses two pre-trained visual models: SigLIP and DinoV2, which are responsible for extracting low-level spatial information and high-level semantic information respectively;

[0025] The visual features output by SigLIP and DinoV2 are concatenated and passed to a large language model to map the visual information to the action space of robot control and generate action decisions.

[0026] Furthermore, the meta-learning pre-training in step S1 specifically discretizes the robot's actions to form a series of discrete action tags, so that the robot can predict these action tags based on visual input;

[0027] After processing, the action tags are input as tokens into the large language model for training. The large language model learns how to generate appropriate action tags based on visual and language inputs by predicting the next token. During training, the model optimizes the target by calculating the cross entropy loss for each discrete action tag.

[0028] Furthermore, the action discretization specifically divides the robot action space into 256 bins, each bin contains a discretized version of an action, and percentiles are used for discretization.

[0029] Furthermore, the input data of the pre-trained VLA model in step S1 includes visual data, sensor data and task target data;

[0030] The output data includes control instructions and task adaptability feedback. The control instructions are the specific execution actions generated by the model based on the current task, and the task adaptability feedback is the way the model adjusts its control strategy based on the execution results. The execution of each task will provide feedback, including information on whether the task was successfully completed.

[0031] Furthermore, the process of task-specific fine-tuning in step S2 includes:

[0032] Prepare new task data, including new visual input and corresponding language prompts;

[0033] Using new task data and a meta-learning method, the parameters of the pre-trained VLA model are updated and fine-tuned through gradient updates of the inner and outer loops and combined with task execution evaluation.

[0034] Furthermore, the visual data includes an RGB image, a depth image or an RGB-D image.

[0035] Furthermore, the language prompt words are specifically task instructions or goals described using natural language, including task descriptions, object indications, and operation instructions.

[0036] Furthermore, the sensor data includes force / tactile sensor data, robotic arm state information, and IMU (inertial measurement unit) data.

[0037] Furthermore, the mission target data includes target object information, mission completion criteria, and environmental status.

[0038] Compared with the prior art, the present invention has the following advantages:

[0039] The present invention performs meta-learning pre-training on the OpenVLA model to obtain a pre-trained VLA model. By performing meta-learning pre-training on a variety of tasks, common features and strategies are extracted, thereby improving the generalization ability of the model. Through the meta-learning technology, the rapid adaptability of the model in new tasks and new scenarios is improved. Among them, the pre-trained VLA model consists of a visual encoder, a projector and a large language model (LLM). The visual encoder is responsible for extracting low-level spatial information and high-level semantic information to help the model perform complex spatial reasoning; the large language model is used to map the visual information to the action space controlled by the robot, and the action decision is generated by the language model. Through the collaboration of these modules, the meta-learning pre-training of OpenVLA can achieve rapid task migration and adaptation between multiple tasks and environments, so that the robot can efficiently perform complex operations.

[0040] The present invention performs task-specific fine-tuning on a pre-trained VLA model. OpenVLA only needs a small amount of new task data (such as a few examples or operation trajectories) to quickly fine-tune the model and perform well in the new task. Through task-specific fine-tuning, OpenVLA can adapt to new tasks in a very short time and quickly make optimization decisions, which enables the robot to perform multiple tasks more efficiently and reduces the delay when the system switches tasks. In addition, task-specific fine-tuning enables the model to flexibly adapt to various changing environments and task requirements and reduces the consumption of computing resources. Since the fine-tuning process only requires a small number of iterations for the data of the new task, more efficient optimization can also be achieved in terms of computing resources.

[0041] The present invention combines multimodal data (visual data, language prompts, sensor data, and task target data) for meta-learning pre-training, which enables the model to be trained through multiple tasks, and can quickly adapt to new tasks. Visual data is used to provide spatial and appearance information for the model, language prompts are used to clarify task goals, sensor data is used to provide real-time feedback, and task target data is used to define the criteria for success. Through the synergy of these data, OpenVLA can make rapid strategy adjustments when facing different operation tasks, thereby improving the generalization ability and execution efficiency of tasks, ensuring that the model can process different types of information during task migration, and comprehensively considering various inputs to generate the best task execution plan. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0043] Figure 2 It is a schematic diagram of the application framework in the embodiment;

[0044] Figure 3 Schematic diagram of the process of task-specific fine-tuning in the embodiment. DETAILED DESCRIPTION

[0045] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] Example

[0047] like Figure 1 As shown, a multimodal large model robot control method based on meta-learning fine-tuning includes the following steps:

[0048] S1. Perform meta-learning pre-training on the OpenVLA model to obtain a pre-trained VLA model;

[0049] S2, performing task-specific fine-tuning on the pre-trained VLA model to obtain a fine-tuned VLA model;

[0050] S3, inputting the current visual data, language prompt words, robot sensor data and task target data graph into the fine-tuned VLA model, and outputting the decision control instructions;

[0051] S4. According to the decision control instructions, the working state of the robot is controlled accordingly.

[0052] This embodiment applies the above solution to optimize the control of the 7-dimensional robotic arm, and the main contents are:

[0053] 1. Meta-learning pre-training

[0054] This solution proposes a fast fine-tuning solution for large models based on meta-learning. Specifically, a new fine-tuning method is proposed for optimizing the multi-modal large model of OpenVlA, a vision-language-robotic arm control program. The core idea of ​​this method is to improve the model's ability to quickly adapt to new tasks and new scenarios through meta-learning technology. The specific steps are as follows:

[0055] The meta-learning pre-training method based on OpenVLA is mainly a deep learning framework that combines visual information and control strategies to quickly adapt to tasks through a small number of examples. The following is a detailed description of its input data, processing process, output data, and model architecture:

[0056] 1. Input data

[0057] The input data of OpenVLA's meta-learning pre-trained model includes visual data, sensor data, and task target data. Visual data is mainly RGB images, depth images, or RGB-D images acquired by cameras. RGB images provide color information of the environment, depth images contain the distance information from each pixel to the camera, and RGB-D images combine RGB with depth information to form a more comprehensive description of the environment. In addition to visual data, the model can also receive data from sensors, which can be force sensors, inertial sensors, or state information of the robot itself, such as joint angles, end effector position, speed, etc. These data help the model understand the position and state of the robot in the environment, and then optimize task execution. Task target data is used to guide model learning, including the position and posture of the target object to be grasped, and the criteria for task success. These target data help the model determine the specific actions to be taken. In addition, the meta-learning task set is also part of the input, which contains data sets of multiple tasks. During the training process, the model will learn how to quickly adjust in these tasks and be able to adapt quickly when faced with new tasks.

[0058] 2. Processing

[0059] The core goal of meta-learning is to enable the model to quickly adapt when faced with new tasks. OpenVLA's meta-learning pre-training method is carried out through the following steps:

[0060] 2.1 Task Distribution Modeling and Rapid Adaptation

[0061] In the framework of meta-learning, OpenVLA does not just train a fixed model, but allows the model to learn how to adapt quickly from multiple tasks through multi-task learning. Specifically, OpenVLA uses Model-Agnostic Meta-Learning (MAML) or similar meta-learning methods to optimize the model so that it can quickly adjust its strategy when it sees new tasks. This is done through the gradient update method of inner and outer loops: the inner loop updates the model parameters for a specific task, and the outer loop optimizes the global parameters of the model so that each task can adapt quickly with less data and fewer training steps.

[0062] In this way, OpenVLA learns how to generate effective action commands based on visual-language input in different operation tasks and environments. The meta-learning model enables OpenVLA to quickly find effective control strategies through a small number of examples and feedback adjustments when facing unseen tasks.

[0063] 2.2 Task Discretization and Action Prediction

[0064] During the training process, OpenVLA discretizes the robot's actions into a series of discrete action tags, which the robot can predict based on visual input. Specifically, OpenVLA divides the robot's action space into 256 bins, each containing a discretized version of an action. In order to make the action prediction more stable and accurate, OpenVLA uses percentiles for discretization instead of directly using the minimum-maximum boundary, which can avoid the impact of outliers in the data on the discretization of the action space.

[0065] After the action tags are processed, they are input as tokens into the Llama2 language model for training. The language model learns how to generate appropriate action tags based on visual and language inputs by predicting the next token. During training, the model optimizes the objective by calculating the cross entropy loss for each discrete action tag.

[0066] 3. Output data

[0067] The output of the OpenVLA meta-learning pre-trained model usually includes two aspects: control instructions and task adaptability feedback. Control instructions are specific execution actions generated by the model based on the current task, including the target angle of the robot arm joint, the target position and posture of the end effector, and decisions such as whether to grasp. Depending on the task, the specific form of the control instructions will also vary. If the task is to grasp an object, the model may output a decision on how to grasp it, whether the grasping posture needs to be adjusted, or how to apply the appropriate grasping force to the object.

[0068] Task adaptability feedback is how the model adjusts its control strategy based on the results of the execution. Each task execution provides feedback, usually including information on whether the task was successfully completed. For example, whether the grasping was successful, whether the object was placed in the correct position, etc. This feedback information is used to adjust the internal parameters of the model so that the model can improve its strategy when faced with similar tasks. Reinforcement learning methods are often used for this feedback adjustment. By evaluating the success or failure of each operation, the model will optimize future performance by updating the strategy.

[0069] 4. Model Architecture

[0070] This solution can use various multi-modal large models. This embodiment is mainly based on the OpenVLA model for creation, such as Figure 2 As shown in Figure 1, the core architecture of the model is a vision-language-action model (VLA), which consists of a visual encoder, a projector, and a large language model (LLM). In the visual encoder part, OpenVLA uses two pre-trained visual models, SigLIP and DinoV2, which are responsible for extracting low-level spatial information and high-level semantic information, respectively, to help the model perform complex spatial reasoning. DinoV2 places special emphasis on spatial reasoning capabilities, which is critical for robot control because robots must not only understand the appearance of objects, but also understand the positional relationship of objects in space.

[0071] These visual features are concatenated and passed to the language model part, namely Llama2 (7B parameters). Llama2 is the language model of OpenVLA. Its task is to map visual information to the action space of robot control and generate action decisions through the language model.

[0072] The design decision of OpenVLA is based on an end-to-end fine-tuning approach, which directly integrates the robot control actions into the prediction process of the language model based on the pre-trained vision-language model. This means that when performing meta-learning pre-training, OpenVLA adopts an integrated approach to generate outputs related to robot actions through the vision-language model, without relying on traditional separately trained vision or control modules.

[0073] Through the collaboration of these modules, OpenVLA's meta-learning pre-training can achieve rapid task migration and adaptation between multiple tasks and environments, allowing the robot arm to efficiently perform grasping and other complex operations. By performing meta-learning pre-training on a large number of diverse tasks, the common features and strategies of the model are extracted. During the pre-training process, this embodiment selects multiple different types of tasks (such as image classification, language understanding, robot arm operation, etc.), and improves the generalization ability of the model through gradient accumulation and strategy optimization.

[0074] 2. Task-Specific Fine-tuning

[0075] The goal of task-specific fine-tuning is to use the pre-trained vision-language-action model (VLA) as a basis and enable the model to be efficiently adjusted to the needs of a specific task through the fine-tuning process. This process not only reduces the dependence on a large amount of labeled data, but also improves the adaptability and flexibility of the model in practical applications.

[0076] The key feature of task-specific fine-tuning is fast adaptation. Even if the model has not been trained on a large scale on the task, it can be effectively adjusted with a small number of samples. This is one of the core advantages of meta-learning, because in traditional machine learning models, a large amount of training data is usually required to effectively adapt the model to a new task. Through task-specific fine-tuning, OpenVLA can use the pre-training capabilities of meta-learning to quickly adjust parameters based on a small number of new samples when faced with new tasks to adapt to the needs of new tasks.

[0077] 1. Effective use of a small number of new samples

[0078] In task-specific fine-tuning, OpenVLA can adjust the model with a small number of new samples. Meta-learning algorithms (such as MAML, Model-Agnostic Meta-Learning, and Model Diagnostic Meta-Learning) enable the model to use a small number of new task samples to fine-tune the model's parameters through fast gradient updates, thereby generating the optimal control strategy for the task. The "small number of new samples" in this process means that the model uses a small number of examples for the task and uses its existing knowledge to make quick adjustments, rather than learning from scratch.

[0079] 2. Quickly adapt to new tasks

[0080] The purpose of task-specific fine-tuning is to quickly adjust the model within a given time and computing resources to adapt to new tasks or scenarios. This ability to quickly adapt enables the robot to handle different tasks in a variety of dynamic environments. For example, during the execution of a task, the robot may encounter new objects or new operating instructions. Task-specific fine-tuning enables the model to understand these new requirements in a short period of time and respond appropriately. By inputting task-specific data (such as images of specific objects and corresponding language instructions) into the model, the robot can quickly learn how to complete the task.

[0081] 3. Improve generalization ability

[0082] Task-specific fine-tuning not only helps the model adapt quickly in a specific task, but also improves the model's generalization ability. By fine-tuning on multiple tasks, OpenVLA can not only perform known tasks, but also better handle new, unseen tasks. For example, if OpenVLA has been pre-trained on multiple grasping tasks, it can adapt to a new grasping task through task-specific fine-tuning, or add new object types to the task, and then quickly learn how to handle these new tasks or objects.

[0083] 4. Implementation process of task-specific fine-tuning

[0084] 4.1 Task Data Preparation

[0085] When performing task-specific fine-tuning, you first need to prepare data related to the new task. This data usually includes new visual input (such as images or videos) and corresponding language prompts (such as task descriptions or instructions). For example, for a new object grasping task, you may need to input an image of the object, a grasping instruction (such as "grasp the red cup"), and the location information of the target object. This data enables OpenVLA to fine-tune the new task.

[0086] 4.2 Fast Fine-tuning with Small Amounts of Data

[0087] Through meta-learning methods such as MAML, the model will quickly adjust based on a small amount of data from a new task. The core of this fine-tuning method is to optimize the model's performance on new tasks through gradient updates in the inner loop and the outer loop. The inner loop updates parameters by training on the new task data in a small scale, while the outer loop optimizes the shared parameters of the entire model, allowing the model to quickly adapt between different tasks. In this process, OpenVLA can adapt to new tasks after just a few iterations without the need for a large number of training samples.

[0088] 4.3 Optimization control strategy

[0089] During task-specific fine-tuning, the model's control strategy is further optimized. In robotics tasks, this usually means that the model needs to adjust its predictions of the robot's actions. For example, in a new object grasping task, OpenVLA needs to adjust how to generate grasping instructions based on the new object information and task description. The model optimizes the action predictions based on the new visual input and task goal data and ensures that the robot performs the appropriate actions (such as grasping force, grasping method, etc.).

[0090] 4.4 Evaluation and Adjustment

[0091] After task-specific fine-tuning, the model is further tuned by executing the task and evaluating its performance. The evaluation process usually includes checking whether the model successfully completed the task and whether it can perform actions stably in different scenarios. Based on the success or failure of the task, the model further adjusts its strategy through a feedback mechanism to improve its performance in new tasks. If the execution result of the task is not ideal, OpenVLA will adjust the fine-tuning strategy based on the evaluation results to further optimize the control decision.

[0092] 5. Advantages of Task-Specific Fine-tuning

[0093] 5.1 Reduce data requirements

[0094] Task-specific fine-tuning significantly reduces the need for large-scale labeled data. In traditional machine learning methods, each new task requires a large amount of labeled data for training, while in OpenVLA's meta-learning framework, the model has learned a wide range of task representations and control strategies through pre-training. Therefore, OpenVLA only needs a small amount of new task data (such as a few examples or operation trajectories) to quickly fine-tune the model and perform well in new tasks.

[0095] 5.2 Improve task execution efficiency

[0096] Through task-specific fine-tuning, OpenVLA can adapt to new tasks in a very short time and make optimization decisions quickly. This enables the robot to perform multiple tasks more efficiently and reduces the delay when the system switches tasks. For example, the robot may need to switch from one task to another (such as switching from grasping an object to placing an object). Task-specific fine-tuning makes this switching process smoother and does not require large-scale training on each task.

[0097] 5.3 Improving the flexibility and adaptability of the model

[0098] Task-specific fine-tuning enables the model to flexibly adapt to various changing environments and task requirements. Whether in a new work scene, facing new objects, or processing different types of task instructions, OpenVLA can quickly adjust its control strategy. This flexibility enables the robot to handle a variety of different tasks and has higher efficiency and adaptability in practical applications.

[0099] 5.4 Optimizing computing resources

[0100] Task-specific fine-tuning not only reduces the need for data, but also reduces the consumption of computing resources. Since the fine-tuning process only requires a small number of iterations on the data of the new task, it can also achieve more efficient optimization in terms of computing resources. This is especially important for deployment to devices with limited computing resources or edge computing environments.

[0101] In this embodiment, in a specific task, a small amount of new samples are used for rapid fine-tuning. The steps include: selecting samples of the new task, calculating the task-specific loss function, and optimizing the model parameters by gradient descent to make it quickly adapt to the new task. The calculation process of the meta gradient is as follows: Figure 3 As shown in the figure, we first obtain key-value pairs from the input edit prompt-answer pairs and cache gradients by back-propagating the standard fine-tuning loss. Next, we generate updated model parameters through the hypernetwork, and input equivalent and irrelevant prompt-answer pairs into the post-edit model. Finally, we calculate the meta-loss and optimize the hypernetwork parameters. The overall process aims to reduce memory consumption and optimize the efficiency of large-scale editing operations by caching and calculating in steps.

[0102] 3. Multimodal Fusion

[0103] OpenVLA combines multimodal data for meta-learning pre-training, which enables the model to be trained through multiple tasks and quickly adapt when seeing new tasks. The core of meta-learning is to use a large amount of multimodal task data to enable the model to not only learn the basic strategy of each task, but also quickly adjust its strategy in new environments or task scenarios with a small number of examples. Multimodal fusion plays a key role in this process. It ensures that the model can process different types of information when migrating tasks, and comprehensively considers various inputs to generate the best task execution plan.

[0104] 1. Visual Data

[0105] Visual data is an important part of OpenVLA input data. It provides the model with image information in the environment and helps the robot understand the appearance, location and relationship of objects with other objects. Visual data includes:

[0106] RGB image: provides color information of the scene and describes the visual features of objects such as shape, color, and texture.

[0107] Depth Image: describes the distance between the object and the camera, helping the model understand the spatial layout of the object.

[0108] RGB-D images: Combine RGB images and depth images to provide a richer description of the scene by fusing color and spatial information. This data input is particularly important for understanding complex environments. These visual inputs are processed by visual encoders in OpenVLA, such as SigLIP and DinoV2. The visual encoder is responsible for extracting high-level visual features from these images, allowing the model to capture the location, spatial structure, and key visual cues of objects.

[0109] 2. Language prompts

[0110] Language prompts are another key input data type in OpenVLA. They refer to task instructions or goals described in natural language. Through the language model, the robot can understand and execute these text instructions. Language prompts can include:

[0111] Task description: For example, “grab the red cup” or “place the object on the table”, which clearly describes the goal of the task.

[0112] Object instructions: For example, “the object on the table” or “the object on the right”, which helps the robot understand the target location or specific features of an object.

[0113] Operation instructions: For example, "grasp", "move" or "rotate", etc., instructing the robot how to interact with the object. These language prompts are processed by a language model in OpenVLA (such as Llama2), which maps natural language input to the predicted space of robot actions, helping the robot understand and execute instructions.

[0114] 3. Sensor Data

[0115] Sensor data complements the robot’s physical perception information and is critical for the robot to perform tasks, especially when interacting with the environment. Sensor data can include:

[0116] Force / tactile sensor data: This senses the pressure, friction, and other information when the robot’s end effector contacts an object, helping the model determine whether the object is grasped and whether the force used is appropriate.

[0117] Robot arm status information: such as joint angles, end effector position, motion speed, etc., providing robot action and status information. Through these sensor data, the model can understand the current state of the robot and adjust the control strategy to adapt to the task requirements.

[0118] IMU (Inertial Measurement Unit) data: Provides dynamic changes of the robot in space. Especially when moving or performing complex operations, IMU data can help the robot accurately determine its own posture changes.

[0119] These sensor data, combined with visual and language information, provide the model with comprehensive environmental perception capabilities, enabling the robot to not only "see" objects, but also "feel" the state of objects and their interaction with the environment.

[0120] 4. Mission target data

[0121] The task target data specifies the task that the robot needs to complete and the target state. The task target data includes:

[0122] Target object information: For example, the type, color, shape, etc. of the object, which can help the model identify the correct object in the visual input.

[0123] Task completion criteria: For example, after grabbing an object, you need to place it in a specified location, or complete the task within a certain time. Task goal data provides the robot with task evaluation criteria to help the model determine whether the task is successful.

[0124] Environmental status: For example, whether the object is in the specified location, whether the object is visible, etc. This information helps the robot determine whether the task meets the requirements.

[0125] These target data are combined with visual, language, and sensor data to provide the model with specific goals that need to be achieved during the task execution, promoting the precise execution of the task. In this process, visual data provides the model with spatial and appearance information, language prompts clarify the task goals, sensor data provides real-time feedback, and task goal data defines the criteria for success. Through the synergy of these data, OpenVLA can quickly adjust its strategies when facing different operation tasks, thereby improving the generalization ability and execution efficiency of the task.

[0126] This embodiment enhances the perception and decision-making capabilities of the model by fusing multimodal information of vision, language, and robotic arm control. The steps include: extracting visual features, processing language information through a language model, optimizing the robotic arm control strategy through reinforcement learning, and finally fusing multimodal information to form a unified decision output.

[0127] In summary, this solution has the following advantages:

[0128] 1. Rapid adaptability: Through meta-learning pre-training, the model can be quickly fine-tuned in new tasks and new scenarios, greatly reducing training time and resource consumption.

[0129] 2. Enhanced generalization ability: Meta-learning technology can effectively improve the generalization ability of the model, making it more stable and reliable when facing a variety of samples and tasks.

[0130] 3. Multimodal fusion: By integrating multimodal information of vision, language and robotic arm control, the model’s comprehensive perception and decision-making capabilities are enhanced, and the robot’s operational performance in complex environments is improved.

Claims

1. A multimodal large model robot control method based on meta-learning fine-tuning, characterized in that: The following steps are involved: S1. Perform meta-learning pre-training on the OpenVLA model to obtain a pre-trained VLA model; S2, performing task-specific fine-tuning on the pre-trained VLA model to obtain a fine-tuned VLA model; S3, inputting the current visual data, language prompt words, robot sensor data and task target data graph into the fine-tuned VLA model, and outputting the decision control instructions; S4. According to the decision control instructions, the working state of the robot is controlled accordingly.

2. According to claim 1, a multi-modal large model robot control method based on meta-learning fine-tuning is characterized in that: The OpenVLA model in step S1 is composed of a visual encoder, a projector, and a large language model. In the visual encoder part, OpenVLA uses two pre-trained visual models: SigLIP and DinoV2, which are responsible for extracting low-level spatial information and high-level semantic information respectively; The visual features output by SigLIP and DinoV2 are concatenated and passed to a large language model to map the visual information to the action space of robot control and generate action decisions.

3. According to claim 2, a multi-modal large model robot control method based on meta-learning fine-tuning is characterized in that: The meta-learning pre-training in step S1 specifically discretizes the robot's actions to form a series of discrete action tags, so that the robot can predict these action tags based on visual input; After processing, the action tags are input as tokens into the large language model for training. The large language model learns how to generate appropriate action tags based on visual and language inputs by predicting the next token. During training, the model optimizes the target by calculating the cross entropy loss for each discrete action tag.

4. The multimodal large model robot control method based on meta-learning fine-tuning according to claim 3 is characterized in that: The action discretization specifically divides the robot action space into 256 bins, each bin contains a discretized version of an action, and percentiles are used for discretization.

5. The multimodal large model robot control method based on meta-learning fine-tuning according to claim 1 is characterized in that: The input data of the pre-trained VLA model in step S1 includes visual data, sensor data and task target data; The output data includes control instructions and task adaptability feedback. The control instructions are the specific execution actions generated by the model based on the current task, and the task adaptability feedback is the way the model adjusts its control strategy based on the execution results. The execution of each task will provide feedback, including information on whether the task was successfully completed.

6. The multimodal large model robot control method based on meta-learning fine-tuning according to claim 1 is characterized in that: The process of task-specific fine-tuning in step S2 includes: Prepare new task data, including new visual input and corresponding language prompts; Using new task data and a meta-learning method, the parameters of the pre-trained VLA model are updated and fine-tuned through gradient updates of the inner and outer loops and combined with task execution evaluation.

7. The multimodal large model robot control method based on meta-learning fine-tuning according to claim 1 is characterized in that: The visual data includes an RGB image, a depth image or an RGB-D image.

8. The multimodal large model robot control method based on meta-learning fine-tuning according to claim 1 is characterized in that: The language prompt words are specifically task instructions or goals described using natural language, including task descriptions, object instructions, and operation instructions.

9. The multimodal large model robot control method based on meta-learning fine-tuning according to claim 1, characterized in that: The sensor data includes force / tactile sensor data, robotic arm status information, and IMU data.

10. The multi-modal large model robot control method based on meta-learning fine-tuning according to claim 1, characterized in that: The task target data includes target object information, task completion criteria, and environmental status.

Images