Robot control method based on multi-modal large model

Abstract

The invention discloses a robot control method based on a multi-modal large model, solves the problems of generalization and adaptability of an existing robot control method based on a large model, and belongs to the field of robot control. The method comprises the steps that a multi-modal planning large model deconstructs a subtask sequence according to the requirements of a user and historical behavior records, and an instruction generation model generates a corresponding control instruction according to the deconstructed subtask sequence and stores the control instruction into a pointer drawing queue; the planning pointer manages the control instructions in the planning pointer queue, and the normally executed control instructions are moved into a historical behavior record; during training, planning control is judged through man-machine interaction, planning content with a good execution effect is stored in an excellent data set, and the excellent data set is utilized to perform fine tuning optimization on the multi-modal planning model and the instruction generation model; and controlling the robot by using the two optimized models and the planning pointer.

Description

Technical Field

[0001] The invention relates to a robot control method based on a multi-modal large model, and belongs to the field of robot control. Background Art

[0002] It is difficult for traditional robots to complete all the complex actions required by users (for example, "clean the kitchen for me") with just a vague instruction. Instead, they rely on manual design and optimization of specific tasks. This task-specific control method makes robots often show low adaptability and flexibility when facing new and unseen tasks. In addition, robots cannot always be in an ideal environment. When robots are in a dynamic and complex environment, various factors will affect the performance of the robots (such as passers-by who are always moving).

[0003] With the rapid development of artificial intelligence technology, large language models (LLMs) have made revolutionary breakthroughs in many fields. Large language models (LLMs) refer to deep learning models with massive parameters (usually ranging from billions to hundreds of billions), which are mainly used to process and generate natural language text. These models are pre-trained on large-scale text datasets and learn various language-related knowledge. They can generate coherent and natural texts like humans and complete various language tasks. Multimodal large models (MM-LLMs) not only have the same text processing capabilities as LLMs, but can also combine inputs of multiple different types of modalities to assist understanding. It integrates information from different modalities and learns the associations and interactions between them, so as to understand and generate information more comprehensively and accurately, and provide more powerful support for various complex tasks. Due to the outstanding capabilities of multimodal large models, the application of multimodal large models to robot control has gradually become a research hotspot.

[0004] Large model fine-tuning technology generally refers to the process of further training the model based on a pre-trained large deep learning model using a private, task-specific dataset, aiming to enable the model to adapt to new, specific tasks or fields, thereby improving the practicality and performance of the model.

[0005] In summary, the limitations of traditional robots in complex tasks and dynamic environments have prompted researchers to explore more advanced technical solutions. The rapid development of large language models and multimodal large models has provided new possibilities for robot control, while large model fine-tuning technology has provided an effective means to improve the adaptability and performance of these models. The combination of these technologies can significantly enhance the robot's human-computer interaction capabilities and complex environment processing capabilities, enabling it to operate effectively in a wider range of application scenarios. At the same time, the existing robot control methods of multimodal large models or large language models have the following defects:

[0006] High computational cost: The complexity of existing large multimodal models or large language models requires a lot of computing resources in the training and reasoning process, especially when processing large-scale data sets, which requires large-scale parallel computing capabilities, which limits their widespread deployment in practical robotics applications. For example, some models require a lot of computing resources and time for fine-tuning, which is impractical in practical applications. This high training cost makes it difficult for many small research institutions and individuals to afford it, thus limiting the widespread application of the model.

[0007] A large amount of data is required for specific tasks: The construction of mature and excellent robotics datasets also requires a lot of time and manpower, and small-scale datasets have poor generalization capabilities. In addition, the quality assessment and improvement of multimodal datasets is a complex problem because it requires considering the relationship and interaction between different types of data.

[0008] The generalization ability of a single model is limited: After being fine-tuned on a specific dataset, existing large models perform well on that dataset, but have poor generalization ability on other datasets. For example, some models perform well on specific tasks but perform poorly in new tasks or environments.

[0009] It is difficult for a single model to achieve multiple goals: Existing large models can only perform extremely well on specific or individual tasks, but when a single model is expected to achieve multiple functions, it will result in unsatisfactory performance on each function. Summary of the invention

[0010] In view of the problems of generalization and adaptability of existing robot control methods based on large models, the present invention provides a robot control method based on a multi-modal large model.

[0011] A robot control method based on a multi-modal large model of the present invention comprises:

[0012] S1. Select the base multimodal large model and the large language model, and combine the robot's multimodal fine-tuning dataset and the fine-tuning dataset of the subtask-machine instruction pair to fine-tune the base multimodal large model and the large language model to obtain the multimodal planning large model and the instruction generation model; the multimodal planning large model is used to deconstruct the subtask sequence according to the user's requirements and historical behavior records, and the instruction generation model is used to generate corresponding control instructions according to the deconstructed subtask sequence;

[0013] S2, controlling the robot through a multimodal planning model, an instruction generation model, and a planning pointer;

[0014] Among them, the planning pointer queue is used to store the corresponding control instructions generated by the instruction generation model in the order of subtasks. The planning pointer points to the control instruction currently being executed in the planning instruction queue. The control instruction that has been executed normally is moved into the historical behavior record for storage. If the user issues a new request during the execution process, the planning pointer will point to the newly input control instruction, and the control instruction corresponding to the user's previous request will be discarded;

[0015] S3. Evaluate the planning control through human-computer interaction. If the execution effect is good, the content of this plan will be stored in the excellent data set. If the execution effect is poor, the content of this plan will be discarded.

[0016] S4, repeatedly execute S3 until the number of executions reaches the set value. If the success rate reaches more than 99%, then go to S5. Otherwise, use the excellent data set to fine-tune and optimize the multimodal planning model and instruction generation model, and then go to S3.

[0017] S5. Use the planning pointer, the fine-tuned optimized multimodal planning model and the instruction generation model to control the robot.

[0018] Preferably, the S5 comprises:

[0019] S51, controlling the robot through the planning pointer, the fine-tuned optimized multimodal planning model, the instruction generation model and the planning pointer;

[0020] S52, judging whether the control instruction is executed correctly, if not, sending a signal to the multimodal planning model, the instruction generation model and the planning pointer;

[0021] After receiving the signal, the multimodal planning model and the instruction generation model re-plan and output new control instructions;

[0022] After receiving the signal, the planning pointer stops the currently executing control instruction, discards the control instruction corresponding to the user's current request, and waits for a new control instruction.

[0023] Preferably, in S5, after receiving the signal, the multimodal planning model and the instruction generation model re-plan and output a new control instruction:

[0024] Adjust the multimodal planning model and instruction generation model by analyzing the causes of errors and the correct solutions;

[0025] The adjusted multi-modal planning model and instruction generation model are used to re-plan the user's requirements and output new control instructions.

[0026] As a preferred method, in S1, the base multimodal large model and the large language model are fine-tuned:

[0027] Low-rank matrices A and B are added to the base multimodal large model and the large language model. The low-rank matrices A and B are fine-tuned and updated through the robot's multimodal fine-tuning dataset and the fine-tuning dataset of the subtask-machine instruction pairs, and the weights of the multimodal planning large model and the instruction generation model are obtained as ΔW+AB.

[0028] Preferably, the method of the present application further includes: writing corresponding prompt words for the multimodal planning model and the instruction generation model by using thought chains and methods based on a small number of examples.

[0029] As a preference, write corresponding prompt words for the multimodal planning model:

[0030] Create an intelligent interaction framework to capture user intent and integrate multi-dimensional information for deep semantic analysis input. The input content includes the visual scene of the environment, historical behavior records, and user needs.

[0031] Based on the input content, natural language processing and computer vision technology are used to generate multiple subtasks. The feasibility and priority of each subtask are evaluated using reinforcement learning strategies, and an execution chain is constructed. Finally, the execution chain with the highest probability of execution is selected for output. Output: behavior, subtask;

[0032] A few examples include:

[0033] When the user's demand is: help me get the book closest to me on the sofa;

[0034] The subtasks of multimodal planning model generation include:

[0035] 1. Use the depth camera to locate the sofa and books and calculate the relative distance to the user;

[0036] 2. Filter out the book closest to the user;

[0037] 3. Plan the path to get books and avoid obstacles;

[0038] 4. Move to the book and grab it;

[0039] 5. Return to the user along the original route and hand the book to the user;

[0040] Execution chain and corresponding executable probability:

[0041] 1->2->3->4->5, 75%;

[0042] 2->1->3->4->5, 35%;

[0043] 3->1->2->4->5, 20%;

[0044] Select 1->2->3->4->5, output: behavior, subtask.

[0045] As a preference, write corresponding prompt words for the instruction generation model:

[0046] The input content of the instruction generation model includes: behavior, subtask and past task execution records, and the past task execution records include the success and failure cases of task execution and the corresponding parameter adjustments;

[0047] Combining the input content and real-time environmental feedback, the subtasks are deconstructed into robot control instructions, and safety checks are performed to ensure that the control instructions are accurate and meet the robot's kinematic and dynamic constraints. Among them, the intelligent mapping tool is used to associate the abstract location in the control instruction with the specific position coordinates, and the object recognition tool is used to assist in locating the target object in the control instruction;

[0048] A few examples include:

[0049] The current subtask is: take the blue notebook on the coffee table in the living room to the bookshelf in the study. The subtask is deconstructed into the control instructions of the robot:

[0050] 1. Navigation: Navigate to the coffee table in the living room

[0051] 2. Object Detection: Detecting the Blue Notebook

[0052] 3. Grab: Grab the blue notebook

[0053] 4. Navigation: Navigate to the study bookshelf

[0054] 5. Place: Place on the bookshelf.

[0055] The beneficial effects of the present invention are that compared with the traditional robot control method, which requires setting specific trigger logic and control model for each task, the multimodal large model combined with the instruction generation model shows stronger adaptability and generalization when facing different tasks. The present invention increases the human-computer interaction capability of the robot. Traditional robots only respond to specific tasks and needs, while robots controlled by large models can not only complete corresponding tasks, but also communicate with users, provide emotional value and companionship for users, and thus provide more intelligent and personalized services. The present invention can continuously improve the control instruction planning performance of the robot through repeated fine-tuning and optimization, and ultimately achieve efficient and accurate robot control. The present invention realizes efficient model division of labor and collaborative cooperation by using multimodal planning models and instruction generation models to complete the content of their respective fields. The multimodal large model is responsible for processing and understanding user intentions and analyzing multimodal information, while the large language model is responsible for generating specific control instructions. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 Examples of operations for planning pointers;

[0057] Figure 2 Optimize the evaluation process for multimodal planning large models and instruction generation models;

[0058] Figure 3 This is the operating process of the robot in actual application. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0061] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0062] The robot control method based on the multimodal large model of this embodiment includes:

[0063] Step 1: Select the base multimodal large model and large language model, and combine the robot's multimodal fine-tuning dataset and the fine-tuning dataset of the subtask-machine instruction pair to fine-tune the base multimodal large model and large language model to obtain the multimodal planning large model and instruction generation model; the multimodal planning large model is used to deconstruct the subtask sequence according to the user's requirements and historical behavior records, and the instruction generation model is used to generate corresponding control instructions according to the deconstructed subtask sequence;

[0064] Specifically, step 1 includes:

[0065] Step 11: Select appropriate base multimodal large models (e.g. QwenVL, etc.) and large language models (e.g. Qwen2.5, GLM-4, etc.). These models have been pre-trained on large-scale data sets and have rich knowledge and strong language generation capabilities. They can avoid the need for a large amount of computing resources and data resources to pre-train a large model from scratch, and the effect is difficult to exceed the current excellent open source model solutions. Therefore, base models with better performance and better model scalability are selected.

[0066] Step 12: Prepare a dataset suitable for fine-tuning the multimodal large model and a dataset for fine-tuning the large language model. When preparing a fine-tuning dataset, you need to ensure that the fine-tuning dataset can cover most of the tasks encountered in the robot's usage scenario. For a multimodal large model, if you choose an image-based multimodal large model, the fine-tuning dataset should contain a variety of information such as images and text descriptions (for example:<image,input,output> , where the output includes behavior and corresponding results, such as subtasks or answers. For large language models, the dataset should include the subtasks decomposed from the target task and the corresponding machine instructions (for example:<subtasks,commands> ), and can also be modified according to actual conditions.

[0067] Step 13: Fine-tune the base multimodal large model and the large language model to obtain the multimodal planning large model and the instruction generation model, and detect whether they can output correctly on their respective tasks. The choice of fine-tuning method is crucial to the final performance of the model. Common fine-tuning methods include instruction fine-tuning, full-scale fine-tuning, and parameter efficient fine-tuning. Instruction fine-tuning can improve the performance of the model on specific tasks by providing clear instructions or examples for specific tasks. Parameter efficient fine-tuning is efficient fine-tuning through low-rank matrix decomposition, reducing computing and storage costs. During the fine-tuning process, the performance of the model on the target task should be evaluated regularly to ensure that it can effectively respond to user instructions and perform corresponding tasks. That is, whether effective behavior can be made for the user's current reaction. In this embodiment, low-rank matrices A and B are added to the base multimodal large model and the large language model, and the low-rank matrices A and B are fine-tuned and updated through the robot's multimodal fine-tuning data set and the fine-tuning data set of the subtask-machine instruction pair, and the weights of the multimodal planning large model and the instruction generation model are obtained as ΔW+AB. By introducing low-rank matrices A and B, the cost of training ΔW is greatly reduced. In this way, for the input x, the final fine-tuned model output is: h=(ΔW+AB)x.

[0068] Step 2: Write corresponding prompt words for the multimodal planning model and the instruction generation model;

[0069] Step 2 writes corresponding prompts for the multimodal planning model and instruction generation model by using chain of thoughts (CoT) and few-shot prompting.

[0070] Step 21: Write the corresponding prompt word for the multimodal planning model pIn order for the multimodal planning model to effectively process and plan multimodal information, the input and output characteristics of the model need to be considered when writing prompts. For example, if the model's task is to identify user intent based on the input image and user needs and deconstruct it into reasonable subtasks, better results can be achieved by introducing thought chains and a small number of prompt methods. The prompts should clarify the current model requirements, specify the model output format, follow the thought chain method, gradually guide the model to analyze user input, and provide the model with a small number of examples to standardize the model output. p The reference example consists of three parts: INPUT, LIMIT, and EXAMPLE:

[0071] INPUT: Create an intelligent interaction framework, accurately capture user intent, and integrate multi-dimensional information for deep semantic analysis input. The input content may include:

[0072] 1. Visual scene of the environment: detailed environmental image analysis information, including object position, color, shape and spatial relationship;

[0073] 2. Historical behavior records: Complete historical conversation context, including topic trends, user preferences, etc.

[0074] 3. User input: user needs.

[0075] Based on the input content, natural language processing and computer vision technology are used to generate multiple subtasks, gain in-depth insights into users’ real needs, and plan action plans and steps.

[0076] LIMIT: Strictly follow the output specifications, first combine all information to generate multiple subtasks, use reinforcement learning strategies to evaluate the feasibility and priority of each subtask, then build an execution chain, and finally select the execution chain with the highest probability of execution to output <behavior, subtask> in a specific format.

[0077] EXAMPLE: For example, when the user demand is "Help me get the book closest to me on the sofa", the behavior and plan generated after analysis by the multimodal planning model are -> "Behavior: ACT, subtasks: 1. Use the depth camera to locate the sofa and the book, and calculate the relative distance to the user 2. Filter out the book closest to the user 3. Plan the book-fetching path and avoid obstacles 4. Move to the book and grab the book 5. Return to the user along the original route and hand the book to the user; execution chain and corresponding executable probability: 1->2->3->4->5 (75%), 2->1->3->4->5 (35%), 3->1->2->4->5 (20%); Finally, select the execution chain with the highest executable probability to output <behavior, subtask>.

[0078] Step 22: Write the corresponding prompt word Prompt for the instruction generation modelc For the instruction generation model, the design of prompt words should focus on clearly expressing the task objectives and the expected output format. For example, the task is to let the model generate robot control instructions based on the given continuous subtasks. c The reference example consists of four parts: INPUT, LIMIT, TOOL, and EXAMPLE;

[0079] INPUT: Build an intelligent task decomposition and instruction generation system. The input content includes:

[0080] 1. Behavior: specific actions;

[0081] 2. Past task execution records: covering successful and failed cases of task execution and corresponding parameter adjustments;

[0082] 3. Current subtask: complex task to be decomposed;

[0083] Based on the input content and real-time environmental feedback, the subtasks are deconstructed into detailed machine instructions that can be directly executed by the robot.

[0084] LIMIT: The output of the limited instruction generation model strictly follows a specific format. It combines the input content and real-time environmental feedback to deconstruct the subtasks into robot control instructions and perform safety checks to ensure that the control instructions are accurate and comply with the robot's kinematic and dynamic limitations.

[0085] TOOL: You can use the smart mapping tool to associate abstract locations with specific coordinates, such as "living room coffee table" →<navigation_target x=5.67,y=3.43,θ=2.04> , you can also use object recognition tools to help locate the target object, such as "identify the blue notebook on the desktop" →<object_detection target=blue_notebook> wait.

[0086] EXAMPLE: For example, according to the current subtask "take the blue notebook on the coffee table in the living room to the bookshelf in the study", use auxiliary tools such as mapping tools to give deconstructed machine control instructions such as:

[0087] 1. Navigation: Navigate to the coffee table in the living room

[0088] 2. Object Detection: Detecting the Blue Notebook

[0089] 3. Grab: Grab the blue notebook

[0090] 4. Navigation: Navigate to the study bookshelf

[0091] 5. Place: Place on the bookshelf.

[0092] Step 3: Control the robot through the multimodal planning model, instruction generation model and planning pointer, including:

[0093] Step 31: Build a planning instruction queue and planning pointer.

[0094] The planning pointer queue is used to store the corresponding control instructions generated by the instruction generation model in the order of subtasks, and the planning pointer is constructed to effectively manage these instructions to achieve the following effects: Figure 1 As shown. The planning pointer points to the control instruction currently being executed in the planning instruction queue. The control instructions that have been executed normally are moved into the historical behavior record for storage. If the user issues a new request during the execution process, the planning pointer will point to the newly input control instruction, and the control instruction corresponding to the user's previous request will be discarded, that is; the previously unfinished instructions will be directly discarded because they are no longer managed by the pointer, and will not be stored in the behavior history. In addition, if there is a problem with the instructions planned by the large model that causes the robot to execute abnormally, the planning pointer will control the stop of all instructions for this round of control and wait for the large model to re-plan. This mechanism ensures the consistency and adaptability of the robot's behavior, while avoiding the continuous interference of erroneous instructions on the robot's behavior.

[0095] Step 32: Deploy and configure the multimodal planning model and the instruction generation model. The deployment and configuration phase requires ensuring that the multimodal planning model and the instruction generation model can run stably in the robot's hardware and software environment. This includes optimizing the model to adapt to the robot's computing resource constraints, and configuring the interface between the model and the robot control system to ensure that the instructions output by the model can be accurately understood and executed by the robot. In addition, program monitoring and logging functions need to be set up to monitor the performance and behavior of the model in real time during operation, and to promptly identify and resolve any problems that arise.

[0096] Step 33: Check whether the robot can be controlled normally through the multimodal planning model and the instruction generation model to ensure that the robot can operate normally. After the model is deployed, testing is required to verify whether the model can control the robot normally. This includes testing the robot's information acquisition ability, behavior planning ability, and execution ability under different environmental and task conditions. For example, the robot can be set in a predefined task scenario, and the tester gives instructions to let the robot perform tasks in these scenarios to observe whether it can accurately obtain environmental information, reasonably plan behavior, and implement it.

[0097] Step 4: Judge the planning control through human-computer interaction. If the execution effect is good, the content of this plan will be stored in the excellent data set. If the execution effect is poor, the content of this plan will be discarded.

[0098] The evaluation is carried out through human-computer interaction. The better content in the evaluation process is stored in the new data set, otherwise it is discarded. During the human-computer interaction process, the tester will evaluate the results of the tasks performed by the robot. If the execution effect is good, it means that the planning and instruction generation of the model are effective. Therefore, the content of this planning, including the input scene image and description, the deconstructed subtask sequence, and the generated control instructions, should be stored in the excellent data set. This excellent data set will serve as a data resource for further fine-tuning of the model. On the contrary, if the execution effect is not good this time, it means that the model has not reached the ideal state, such as insufficient understanding of the specific scene or incorrect control instructions generated. At this time, the content of this planning will be discarded to avoid using erroneous or inefficient planning data for further training of the model.

[0099] Step 5: Repeat step 4 until the number of executions reaches the set value. If the success rate reaches 99% or more, proceed to step 6. Otherwise, use the excellent data set to fine-tune and optimize the multimodal planning model and instruction generation model, and proceed to step 4.

[0100] If the planning success rate in this round of evaluation reaches 99% or above, it is considered successful. Otherwise, the two large models are re-tuned using new data. After the evaluation is completed, it is necessary to count the proportion of successful planning in this round. In this implementation method, the success rate in this round of planning is calculated by the formula, where: S is the success rate, t is the total number of planning times, c is the number of correct executions, and p is the number of correct executions that achieve the desired effect:

[0101]

[0102] If the success rate reaches 99% or above, it means that the multimodal planning model and the instruction generation model can cooperate accurately in most cases, and reasonably plan and control the behavior of the robot. Therefore, it can be considered that the model has achieved a good performance level, and this round of optimization can be regarded as a success. However, if the success rate is lower than 99%, it means that there is still room for improvement in the model. At this time, the two large models should be re-fine-tuned and optimized using the new data set collected in step 41. This new data set contains cases where the model performs well in actual applications, and can provide the model with learning samples that are closer to actual needs, which helps the model better adapt to various task scenarios, thereby improving the success rate in the next round of evaluation. The effect is as follows: Figure 2 shown.

[0103] Step 6: Use the planning pointer, the fine-tuned optimized multimodal planning model and the instruction generation model to control the robot.

[0104] Step 6 is the specific application of the multimodal planning model and the instruction generation model, further including:

[0105] Build a judgment module: embed a judgment module in the operation process of the multimodal planning model and the instruction generation model. This module is responsible for real-time monitoring of the instructions output by the model and comparing and analyzing the actual execution results of the robot. The judgment module can judge whether the instructions are executed correctly based on multiple information sources such as preset rules, the confidence output of the model, and the sensor feedback of the robot. For example, if the instruction generated by the model requires the robot to move to a specific position, but the robot's position sensor feedback shows that it has not reached the expected position, the judgment module will determine that the instruction is executed incorrectly.

[0106] Establish error feedback: Once the judgment module identifies an instruction error, it will immediately send a signal to the multimodal planning model, instruction generation model, and planning pointer. This signal will trigger two response behaviors: First, notify the multimodal planning model and instruction generation model that there is a problem with the current plan and that new instructions need to be reconsidered and generated; second, tell the planning pointer to pause the current instruction execution sequence and prepare to receive new planning instructions. After receiving the error signal, the multimodal planning model and instruction generation model will review the input information and previous planning logic, and generate a new instruction sequence in combination with the current environmental feedback. The planning pointer will then point to the new instruction sequence and manage the execution of the instructions to ensure that the robot can act correctly according to the updated instructions.

[0107] Continuous optimization and learning. In addition to immediate replanning responses, the error detection and feedback mechanism will also promote continuous learning and optimization of the model. Each replanning process triggered by an error can be regarded as a learning case. By analyzing the cause of the error and the correct solution, the multimodal planning model and the instruction generation model are adjusted; the adjusted multimodal planning model and instruction generation model are used to replan the user's requirements and output new control instructions. In the long run, this will help the model better adapt to various complex and dynamic environmental conditions and improve its stability and reliability in practical applications. Specific implementation effects are as follows: Figure 3 shown.

[0108] This implementation method uses a multimodal large model to focus on user needs and environmental conditions, analyze user intentions and plan reasonable sub-tasks, and then uses the instruction generation model to convert them into machine instructions executable by the robot, so that the robot can make better human-computer interaction and more reasonable behavioral logic.

[0109] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be used in other described embodiments.

Claims

1. A robot control method based on a multimodal large model, characterized in that: include: S1. Select the base multimodal large model and the large language model, and combine the robot's multimodal fine-tuning dataset and the fine-tuning dataset of the subtask-machine instruction pair to fine-tune the base multimodal large model and the large language model to obtain the multimodal planning large model and the instruction generation model; the multimodal planning large model is used to deconstruct the subtask sequence according to the user's requirements and historical behavior records, and the instruction generation model is used to generate corresponding control instructions according to the deconstructed subtask sequence; S2, controlling the robot through a multimodal planning model, an instruction generation model, and a planning pointer; Among them, the planning pointer queue is used to store the corresponding control instructions generated by the instruction generation model in the order of subtasks. The planning pointer points to the control instruction currently being executed in the planning instruction queue. The control instruction that has been executed normally is moved into the historical behavior record for storage. If the user issues a new request during the execution process, the planning pointer will point to the newly input control instruction, and the control instruction corresponding to the user's previous request will be discarded; S3. Evaluate the planning control through human-computer interaction. If the execution effect is good, the content of this plan will be stored in the excellent data set. If the execution effect is poor, the content of this plan will be discarded. S4, repeatedly execute S3 until the number of executions reaches the set value. If the success rate reaches more than 99%, then go to S5. Otherwise, use the excellent data set to fine-tune and optimize the multimodal planning model and instruction generation model, and then go to S3. S5. Use the planning pointer, the fine-tuned optimized multimodal planning model and the instruction generation model to control the robot.

2. The robot control method based on a multimodal large model according to claim 1, characterized in that: The S5 includes: S51, controlling the robot through the planning pointer, the fine-tuned optimized multimodal planning model, the instruction generation model and the planning pointer; S52, judging whether the control instruction is executed correctly, if not, sending a signal to the multimodal planning model, the instruction generation model and the planning pointer; After receiving the signal, the multimodal planning model and the instruction generation model re-plan and output new control instructions; After receiving the signal, the planning pointer stops the currently executing control instruction, discards the control instruction corresponding to the user's current request, and waits for a new control instruction.

3. The robot control method based on a multimodal large model according to claim 2, characterized in that: In S5, after receiving the signal, the multi-modal planning model and the instruction generation model re-plan and output a new control instruction: Adjust the multimodal planning model and instruction generation model by analyzing the causes of errors and the correct solutions; The adjusted multi-modal planning model and instruction generation model are used to re-plan the user's requirements and output new control instructions.

4. The robot control method based on a multimodal large model according to claim 1, characterized in that: In S1, fine-tune the base multimodal large model and the large language model: Low-rank matrices A and B are added to the base multimodal large model and the large language model. The low-rank matrices A and B are fine-tuned and updated through the robot's multimodal fine-tuning dataset and the fine-tuning dataset of the subtask-machine instruction pairs, and the weights of the multimodal planning large model and the instruction generation model are obtained as ΔW+AB.

5. The robot control method based on multimodal large model according to claim 1, characterized in that: The method also includes: writing corresponding prompt words for the multimodal planning model and the instruction generation model by using thought chaining and a method based on a small number of examples.

6. The robot control method based on multimodal large model according to claim 5, characterized in that: Write corresponding prompt words for the multimodal planning model: Create an intelligent interaction framework to capture user intent and integrate multi-dimensional information for deep semantic analysis input. The input content includes the visual scene of the environment, historical behavior records, and user needs. Based on the input content, natural language processing and computer vision technology are used to generate multiple subtasks. The feasibility and priority of each subtask are evaluated using reinforcement learning strategies, and an execution chain is constructed. Finally, the execution chain with the highest probability of execution is selected for output. Output: behavior, subtask; A few examples include: When the user's demand is: help me get the book closest to me on the sofa; The subtasks of multimodal planning model generation include:

1. Use the depth camera to locate the sofa and books and calculate the relative distance to the user; 2. Filter out the book closest to the user; 3. Plan the path to get books and avoid obstacles; 4. Move to the book and grab it; 5. Return to the user along the original route and hand the book to the user; Execution chain and corresponding executable probability: 1->2->3->4->5，75％； 2->1->3->4->5，35％； 3->1->2->4->5，20％； Select 1->2->3->4->5, output: behavior, subtask.

7. The robot control method based on multimodal large model according to claim 5, characterized in that: Write the corresponding prompt words for the instruction generation model: The input content of the instruction generation model includes: behavior, subtask and past task execution records, and the past task execution records include the success and failure cases of task execution and the corresponding parameter adjustments; Combining the input content and real-time environmental feedback, the subtasks are deconstructed into robot control instructions, and safety checks are performed to ensure that the control instructions are accurate and meet the robot's kinematic and dynamic constraints. Among them, the intelligent mapping tool is used to associate the abstract location in the control instruction with the specific position coordinates, and the object recognition tool is used to assist in locating the target object in the control instruction; A few examples include: The current subtask is: take the blue notebook on the coffee table in the living room to the bookshelf in the study. The subtask is deconstructed into the control instructions of the robot:

1. Navigation: Navigate to the coffee table in the living room 2. Object Detection: Detecting the Blue Notebook 3. Grab: Grab the blue notebook 4. Navigation: Navigate to the study bookshelf 5. Place: Place on the bookshelf.

8. A computer-readable storage device storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the robot control method based on a multimodal large model as claimed in any one of claims 1 to 7 are implemented.

9. A robot control device based on a multimodal large model, comprising a storage device, a processor, and a computer program stored in the storage device and executable on the processor, characterized in that: The processor executes the computer program to implement the steps of the robot control method based on the multimodal large model as claimed in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the robot control method based on a multi-modal large model as claimed in any one of claims 1 to 7 are implemented.

Images