A Method and System for Generating Manipulation Trajectories of Embossed Robots Based on Multimodal Large Models

Abstract

This invention discloses a method and system for generating embodied robot operation trajectories based on a multimodal large model, belonging to the field of artificial intelligence technology. The method includes: acquiring visual, linguistic, and robot input data, preprocessing them, and mapping them to a high-dimensional space to obtain visual, linguistic, and robot input vectors; inputting these input vectors into an original multimodal large model and an expert model for training, obtaining a pre-trained multimodal large model and a trained expert model; using a generative large model to generate call data for the trained expert model, forming an instruction fine-tuning dataset, which is then input into the pre-trained multimodal large model for training, obtaining a final trained multimodal large model; and generating the robot operation trajectory based on the robot operation data and the final trained multimodal large model. This application effectively solves the problems of low generalization, low precision, and difficulty in adapting to changing environments and task requirements in existing robot operations.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and system for generating the operational trajectory of an embodied robot based on a multimodal large model. Background Technology

[0002] In recent years, multimodal large models have been widely used in the field of embodied robotics, especially demonstrating significant potential in robot task planning and execution. However, existing technologies mainly employ two strategies: one is direct text planning, and the other is direct action output. While direct text planning leverages the powerful reasoning capabilities of large models to generate high-level task plans, it often lacks applicability in specific real-world scenarios. For example, when the instruction is "tidy up the table," text planning might generate a high-level task plan like "organize the items." However, the robot cannot determine the specific operational steps from this general instruction, such as how to classify items, the order in which to perform them, or which tools to use. This leaves the robot lacking targeted operational guidance and struggling to handle the details of the task. On the other hand, while direct action output can generate specific operational instructions, it largely sacrifices the reasoning capabilities of the large model, resulting in generated operations that lack generalization and are difficult to adapt to changing environments and task requirements.

[0003] Therefore, in order to address the shortcomings of these two extreme strategies, there is an urgent need for a method that can strike a balance between the reasoning ability of large models and the execution of specific operations, so as to retain the generalization ability of large models in environmental understanding and task planning, while also providing sufficiently detailed guidance for downstream models to support robots in performing fine-grained operations. Summary of the Invention

[0004] In order to solve the problems existing in the prior art, the present invention provides the following technical solution.

[0005] The first aspect of this invention provides a method for generating the operational trajectory of an embodied robot based on a multimodal large model, comprising:

[0006] Visual input data, language input data, and robot input data are acquired and preprocessed. The preprocessed visual input data, language input data, and robot input data are aligned and mapped to high-dimensional space to obtain visual input vector, language input vector, and robot input vector, respectively.

[0007] The visual input vector, language input vector, and robot input vector are respectively input into the original multimodal large model and the original expert model for training, to obtain a pre-trained multimodal large model and a trained expert model;

[0008] Generative large models are used to generate call data for trained expert models, forming a standardized instruction fine-tuning dataset:

[0009] The instruction-tuned dataset is input into the pre-trained multimodal large model for training, to obtain the final trained multimodal large model;

[0010] Based on robot operation data and the final trained multimodal large model, the robot operation trajectory is generated.

[0011] Preferably, the preprocessing of the acquired visual input data, language input data, and robot input data includes cleaning the acquired visual input data, language input data, and robot input data.

[0012] Preferably, aligning the preprocessed visual input data, language input data, and robot input data and mapping them to a high-dimensional space includes: aligning the preprocessed visual input data, language input data, and robot input data and mapping them to a high-dimensional space using a multilayer perceptron to obtain visual input vectors, language input vectors, and robot input vectors.

[0013] Preferably, the visual input data includes image data, point cloud data, and video data; the language input data includes language command data for the robot; and the robot input data includes robot body parameters and sensor parameters.

[0014] Preferably, the original multimodal large model is an LLaVA model.

[0015] Preferably, the generation of robot operation trajectory includes generating robot visual trajectory, robot operation trajectory coordinates, and robot operation instruction planning.

[0016] Preferably, the robot visualization trajectory and robot operation trajectory coordinates are generated and output by a trained expert model, and the robot operation instruction planning is generated and output by a finally trained multimodal large model.

[0017] A second aspect of the present invention provides a system for generating the operational trajectory of an embodied robot based on a multimodal large model, comprising:

[0018] The multimodal data acquisition and mapping module is used to acquire visual input data, language input data and robot input data and preprocess them. After aligning the preprocessed visual input data, language input data and robot input data, they are mapped to high-dimensional space to obtain visual input vector, language input vector and robot input vector respectively.

[0019] The first model training module is used to input the visual input vector, language input vector and robot input vector into the original multimodal large model and the original expert model for training, respectively, to obtain the initially trained multimodal large model and the trained expert model;

[0020] The expert model call data generation module is used to generate call data for the trained expert model using a generative large model, and form a standardized instruction fine-tuning dataset.

[0021] The second model training module is used to input the instruction fine-tuning dataset into the initially trained multimodal large model for training, so as to obtain the final trained multimodal large model.

[0022] The operation trajectory generation module is used to generate robot operation trajectories based on robot operation data and the final trained multimodal large model.

[0023] A third aspect of the present invention provides a memory that stores multiple instructions for implementing the method for generating the maneuver trajectory of a embodied robot based on a multimodal large model as described in the first aspect.

[0024] A fourth aspect of the present invention provides an electronic device, including a processor and a memory connected to the processor, the memory storing a plurality of instructions which can be loaded and executed by the processor to enable the processor to perform the method for generating embodied robot maneuver trajectories based on a multimodal large model as described in the first aspect.

[0025] The beneficial effects of this invention are as follows: The method and system for generating embodied robot operation trajectories based on a multimodal large model provided by this invention trains the multimodal large model using multimodal data such as vision, language, and robot data. This enables the trained multimodal large model to simultaneously perform text planning and operation trajectory generation for robot operations, effectively solving the problems of low generalization, low precision, and difficulty in adapting to changing environments and task requirements in existing technologies for embodied robot operations. This provides solid technical support for the operation of embodied robots in changing environments. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating the method for generating the operational trajectory of an embodied robot based on a multimodal large model as described in this invention.

[0027] Figure 2 This is a functional structure diagram of the embodied robot operation trajectory generation system based on a multimodal large model as described in this invention. Detailed Implementation

[0028] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0029] The method provided by this invention can be implemented in a terminal environment that may include one or more of the following components: a processor, a memory, and a display screen. The memory stores at least one instruction, which is loaded and executed by the processor to implement the method described in the following embodiments.

[0030] A processor may include one or more processing cores. The processor uses various interfaces and lines to connect various parts of the terminal, and performs various functions and processes data by running or executing instructions, programs, code sets or instruction sets stored in memory, and by calling data stored in memory.

[0031] Memory can include random access memory (RAM) or read-only memory (ROM). Memory can be used to store instructions, programs, code, code sets, or instructions.

[0032] The display screen is used to show the user interface of each application.

[0033] In addition, those skilled in the art will understand that the structure of the terminal described above does not constitute a limitation on the terminal. The terminal may include more or fewer components, or combine certain components, or have different component arrangements. For example, the terminal may also include radio frequency circuits, input units, sensors, audio circuits, power supplies, and other components, which will not be described in detail here.

[0034] Example 1

[0035] like Figure 1 As shown, this embodiment of the invention provides a method for generating the operational trajectory of an embodied robot based on a multimodal large model, including:

[0036] S101, acquire visual input data, language input data and robot input data and preprocess them, align the preprocessed visual input data, language input data and robot input data and map them to high-dimensional space respectively to obtain visual input vector, language input vector and robot input vector;

[0037] S102, the visual input vector, language input vector and robot input vector are respectively input into the original multimodal large model and the original expert model for training, so as to obtain the pre-trained multimodal large model and the trained expert model;

[0038] S103 utilizes generative large models to generate call data for trained expert models and forms a standardized instruction fine-tuning dataset.

[0039] S104, Input the instruction fine-tuning dataset into the pre-trained multimodal large model for training, and obtain the final trained multimodal large model;

[0040] S105 generates the robot's operation trajectory based on the robot's operation data and the final trained multimodal large model.

[0041] Specifically, in step S101, the visual input data includes image data, point cloud data, and video data; the language input data includes language command data for the robot, such as the language command given to the robot, "Please pour the water from the teapot into the cup, being careful not to knock over anything around you"; the robot input data includes robot body parameters (robot model, robotic arm type, etc.), sensor parameters, etc. Preprocessing the acquired visual input data, language input data, and robot input data includes cleaning the acquired visual input data, language input data, and robot input data to ensure the quality of the acquired multimodal data. The preprocessed visual input data, language input data, and robot input data are aligned and mapped to a high-dimensional space to obtain visual input vectors, language input vectors, and robot input vectors. This involves aligning the preprocessed visual input data, language input data, and robot input data, and then mapping them to a high-dimensional space using a multilayer perceptron to obtain visual input vectors, language input vectors, and robot input vectors that meet the dimensionality requirements of multimodal large-scale models. It should be noted that, for example, visual input data obtained through a visual encoder is mapped through a visual mapping layer to obtain visual input vectors, language input data is mapped through a text mapping layer to obtain language input vectors, and robot input data is mapped through an ontology mapping layer to obtain robot input vectors. In this step, by fusing these multimodal input data (visual input data, language input data, and robot input data), the reasoning ability, scene perception ability, and generalization ability in the robotics field of multimodal large-scale models can be comprehensively improved.

[0042] In step S102, the original multimodal large model can be the LLaVA model, which is a multimodal large model designed to integrate and understand various data types, including text, images, and videos. The original expert model can be the DINOv2 model, SAM model, etc. It should be noted that expert model is a term in the field of artificial intelligence. In this application, the expert model generates the robot's visual trajectory and robot operation trajectory coordinates based on the text planning (i.e., robot operation instruction planning) output by the multimodal large model, thereby achieving the goal of simultaneously outputting text planning and operation trajectory.

[0043] In step S103, the generative large model can be GPT-4. The robot input vector is input into the original multimodal large model for training, enabling the model to generate a realistic and reliable operable area based on the current robot body. This ensures that the model has a more accurate understanding of its own capabilities and limitations, thereby improving the executability and accuracy of the operation.

[0044] In step S104, the instruction fine-tuning dataset obtained in step S103 is input into the pre-trained multimodal large model for further training. This is mainly used to enhance the ability of the finally trained multimodal large model to actively invoke the trained expert model when facing the task of generating object operation trajectories.

[0045] The robot operation data in step S105 has the same data format as the visual input data, language input data, and robot input data in step S101. It can be used as test data to test the finally trained multimodal large model. In addition, the robot operation data can also be used as an evaluation index during testing to judge the generation effect of the operation trajectory of the finally trained multimodal large model. It should be noted that a very accurate 3D operation trajectory can be obtained through detector tracking and environmental depth information estimation. The robot operation data is generated based on the 3D operation trajectory and serves as the real value in the model training process. Generating the robot operation trajectory includes generating the robot visual trajectory, robot operation trajectory coordinates, and robot operation command planning. The robot visual trajectory and robot operation trajectory coordinates are generated and output by the trained expert model, and the robot operation command planning is generated and output by the finally trained multimodal large model.

[0046] This application trains a large multimodal model and simultaneously generates text-based plans for the robot's maneuvers and corresponding maneuver trajectories. This allows for both high-level, generalized guidance from the text-based plans and low-level, fine-grained guidance from the maneuver trajectories. Through this method, the application achieves the following effects:

[0047] (1) Improve operational accuracy: By using multimodal data to enhance the understanding of the scene and target objects, more accurate operation trajectories can be generated, which can improve the success rate of robots in detailed tasks (such as assembly, grasping, etc.).

[0048] (2) Improve operational efficiency: By integrating multimodal data (such as vision, language, robot, etc.) and the powerful learning capabilities of large models, the robot's operational trajectory can be generated more efficiently, reducing trajectory planning time and improving the robot's reaction speed in actual tasks.

[0049] (3) Enhanced intelligent decision-making ability: Based on multimodal data, task scenarios can be analyzed from multiple perspectives such as vision, language and robots, enabling robots to have a higher environmental understanding ability, thereby improving their intelligent decision-making level in complex environments;

[0050] (4) Strong adaptability: Through large model learning and fusion of multimodal data, it can better cope with diverse tasks and environmental changes, and has strong adaptability, thereby improving the task execution capability of the embodied robot. It has a wide range of application prospects in many industries, such as intelligent manufacturing, home services, medical assistance and other fields.

[0051] (5) It can reduce the difficulty of programming and control: Multimodal large models reduce the dependence on traditional robot engineers for programming and control by automatically learning and generating operation trajectories, enabling non-professionals to perform simple robot control through natural language commands and other means.

[0052] (6) Improve human-computer interaction experience: By combining multimodal inputs such as language and vision, robots can understand human intentions more naturally, thereby achieving a smoother interactive experience, which enables robots to better collaborate with humans;

[0053] (7) Task automation and generalization capabilities: The multimodal large model has powerful task transfer and generalization capabilities. It can apply the generated trajectory to new tasks by learning from existing data, thereby reducing the need for retraining and debugging.

[0054] Example 2

[0055] like Figure 2 As shown, another aspect of the present invention also includes a functional module architecture that is completely consistent with the aforementioned method flow. That is, the embodiments of the present invention also provide a system for generating the operational trajectory of an embodied robot based on a multimodal large model, including:

[0056] The multimodal data acquisition and mapping module 201 is used to acquire visual input data, language input data and robot input data and preprocess them. After aligning the preprocessed visual input data, language input data and robot input data, they are mapped to a high-dimensional space to obtain visual input vector, language input vector and robot input vector respectively.

[0057] The first model training module 202 is used to input the visual input vector, language input vector and robot input vector into the original multimodal large model and the original expert model for training, respectively, to obtain the pre-trained multimodal large model and the trained expert model;

[0058] The expert model call data generation module 203 is used to generate call data for the trained expert model using a generative large model, and form a standardized instruction fine-tuning dataset.

[0059] The second model training module 204 is used to input the instruction fine-tuning dataset into the initially trained multimodal large model for training, so as to obtain the final trained multimodal large model.

[0060] The operation trajectory generation module 205 is used to generate the robot operation trajectory based on the robot operation data and the finally trained multimodal large model.

[0061] This system can be implemented using the embodied robot operation trajectory generation method based on a multimodal large model provided in Embodiment 1 above. For the specific implementation method, please refer to the description in Embodiment 1, which will not be repeated here.

[0062] The present invention also provides a memory that stores multiple instructions for implementing the method for generating the maneuver trajectory of a embodied robot based on a multimodal large model as described in Embodiment 1.

[0063] The present invention also provides an electronic device, including a processor and a memory connected to the processor, the memory storing a plurality of instructions which can be loaded and executed by the processor to enable the processor to execute the method for generating the embodied robot operation trajectory based on a multimodal large model as described in Embodiment 1.

[0064] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if these modifications and modifications of the invention fall within the scope of the claims and their equivalents, the invention is also intended to include these modifications and modifications.

Claims

1. A method for generating the operational trajectory of an embodied robot based on a multimodal large model, characterized in that, include: Visual input data, language input data, and robot input data are acquired and preprocessed. The preprocessed visual input data, language input data, and robot input data are aligned and mapped to high-dimensional space to obtain visual input vector, language input vector, and robot input vector, respectively. The visual input vector, language input vector, and robot input vector are respectively input into the original multimodal large model and the original expert model for training, to obtain a pre-trained multimodal large model and a trained expert model; Generative large models are used to generate call data for trained expert models and form a standardized instruction fine-tuning dataset. The instruction-tuned dataset is input into the pre-trained multimodal large model for training, to obtain the final trained multimodal large model; Based on robot operation data and the final trained multimodal large model, the robot operation trajectory is generated; Preprocessing of the acquired visual input data, language input data, and robot input data includes cleaning the acquired visual input data, language input data, and robot input data; The step of aligning the preprocessed visual input data, language input data, and robot input data and mapping them to a high-dimensional space includes: aligning the preprocessed visual input data, language input data, and robot input data and mapping them to a high-dimensional space using a multilayer perceptron; The visual input data includes image data, point cloud data, and video data; the language input data includes language commands to the robot; and the robot input data includes robot body parameters and sensor parameters.

2. The method for generating the operational trajectory of an embodied robot based on a multimodal large model as described in claim 1, characterized in that, The original multimodal large model is the LLaVA model.

3. The method for generating the operational trajectory of an embodied robot based on a multimodal large model as described in claim 1, characterized in that, The generated robot operation trajectory includes generating a visual robot trajectory, robot operation trajectory coordinates, and robot operation command planning.

4. The method for generating the operational trajectory of an embodied robot based on a multimodal large model as described in claim 3, characterized in that, The robot's visualized trajectory and robot operation trajectory coordinates are generated and output by a trained expert model, and the robot operation instruction planning is generated and output by a finally trained multimodal large model.

5. A system for generating the operational trajectory of an embodied robot based on a multimodal large model, characterized in that, include: The multimodal data acquisition and mapping module is used to acquire visual input data, language input data and robot input data and preprocess them. After aligning the preprocessed visual input data, language input data and robot input data, they are mapped to high-dimensional space to obtain visual input vector, language input vector and robot input vector respectively. The first model training module is used to input the visual input vector, language input vector and robot input vector into the original multimodal large model and the original expert model for training, respectively, to obtain the initially trained multimodal large model and the trained expert model; The expert model call data generation module is used to generate call data for the trained expert model using a generative large model, and form a standardized instruction fine-tuning dataset. The second model training module is used to input the instruction fine-tuning dataset into the initially trained multimodal large model for training, so as to obtain the final trained multimodal large model. The operation trajectory generation module is used to generate robot operation trajectories based on robot operation data and the finally trained multimodal large model; Preprocessing of the acquired visual input data, language input data, and robot input data includes cleaning the acquired visual input data, language input data, and robot input data; The step of aligning the preprocessed visual input data, language input data, and robot input data and mapping them to a high-dimensional space includes: aligning the preprocessed visual input data, language input data, and robot input data and mapping them to a high-dimensional space using a multilayer perceptron; The visual input data includes image data, point cloud data, and video data; the language input data includes language commands to the robot; and the robot input data includes robot body parameters and sensor parameters.

6. A memory, characterized in that, The system stores multiple instructions for implementing the method for generating the maneuver trajectory of a embodied robot based on a multimodal large model as described in any one of claims 1-4.

7. An electronic device, characterized in that, The system includes a processor and a memory connected to the processor, the memory storing multiple instructions that can be loaded and executed by the processor to enable the processor to perform the method for generating the maneuver trajectory of a embodied robot based on a multimodal large model as described in any one of claims 1-4.

Images