Action generation method, system and equipment based on multi-modal large model and medium

Abstract

The invention discloses an action generation method, system and device based on a multi-modal large model and a medium, and relates to the field of artificial intelligence, and the method comprises the steps: obtaining an action type of a current environment input by a user and a multi-modal instruction; performing intention understanding and step splitting on the multi-modal instruction based on the first multi-modal large model to obtain a user control intention in a text form; for any processing step in the user control intention, environment perception is carried out by adopting a second multi-mode large model based on the user control intention and the current environment information; a third multi-mode large model is adopted to generate a next action; and calling a corresponding action interface according to the next action. The multi-mode instruction input by the user can be analyzed in real time, the environment information can be understood, and the action adapting to the current environment is generated, so that efficient and autonomous control of the robot is realized, and high-accuracy generation and real-time interaction of complex actions are realized.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a method, system, device and medium for generating motion based on a multimodal large model. Background Art

[0002] In recent years, large language models have made important achievements in language understanding and generation. Multimodal large models have further expanded the ability of language understanding and generation. By combining visual, language and sensor data, they have improved the ability to analyze and reason about complex scenes, showing great potential in complex information processing. However, related multimodal large models have the following disadvantages.

[0003] 1. Low accuracy in processing complex actions: When processing complex control intentions, the amount of information about basic actions is too low and there is little data containing semantic descriptions of actions, resulting in a large deviation between the generated action sequence and the actual goal, affecting the robot's execution efficiency and accuracy.

[0004] 2. Multimodal data fusion is difficult: In practical applications, how to effectively fuse data from different modalities remains a challenge. Signals from different modalities have different feature processing methods, and it is difficult to fuse multimodal features and effectively integrate them into the action generation and execution process.

[0005] 3. Poor interactivity: Robots can often only perform tasks according to preset programs or instructions, and cannot adjust according to human instructions or environmental changes. This lack of interactive control method limits the application scenarios and flexibility of robots.

[0006] 4. Poor generalization: Robot posture control algorithms are often optimized only for specific tasks or scenarios, and are difficult to adapt to complex and changing environments and needs. When the task or scenario changes, the control action needs to be redesigned or adjusted, which greatly increases the cost and difficulty. Summary of the invention

[0007] The purpose of this application is to provide a method, system, device and medium for generating action based on a multimodal large model, which can improve the generation accuracy of complex actions.

[0008] To achieve the above objectives, this application provides the following solutions: In a first aspect, the present application provides an action generation method based on a multimodal large model, comprising: Get the action type and multimodal instructions of the current environment input by the user; Based on the first multimodal large model, the multimodal instruction is understood and the steps are split to obtain the user control intention in text form; the user control intention includes a plurality of continuous processing steps; For any processing step in the user control intention, current environment information is collected, and based on the user control intention and the current environment information, a second multimodal large model is used to perform environment perception to obtain an environment perception result; Generate a next action using a third multimodal large model according to the executed processing steps, the action type, the processing steps and the environmental perception result; The corresponding action interface is called according to the next step action to execute the processing step.

[0009] In a second aspect, the present application provides an action generation system based on a multimodal large model, comprising: The data acquisition module is used to obtain the action type and multimodal instructions of the current environment input by the user; An intention understanding module, used to understand the intention and split the steps of the multimodal instruction based on the first multimodal large model to obtain the user control intention in text form; the user control intention includes multiple continuous processing steps; An environment perception module, for collecting current environment information for any processing step in the user control intention, and performing environment perception using a second multimodal large model based on the user control intention and the current environment information to obtain an environment perception result; An action generation module, configured to generate a next action using a third multimodal large model according to the executed processing steps, the action type, the processing steps and the environmental perception result; The action execution module is used to call the corresponding action interface according to the next step action to execute the processing step.

[0010] In a third aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned action generation method based on a multimodal large model.

[0011] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned action generation method based on a multimodal large model.

[0012] According to the specific embodiments provided in this application, this application has the following technical effects: The present application provides a method, system, device and medium for generating action based on a multimodal large model, which fully utilizes the advantages of language understanding, task planning and multimodal environmental perception of the multimodal large model, and can parse multimodal instructions input by users in real time, understand environmental information, and generate actions adapted to the current environment, thereby realizing efficient and autonomous control of the robot, and realizing high-accuracy generation and real-time interaction of complex actions. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0014] Figure 1 This is an application environment diagram of an action generation method based on a multimodal large model in one embodiment of the present application.

[0015] Figure 2 An overall flow chart of an action generation method based on a multimodal large model provided in one embodiment of the present application.

[0016] Figure 3 A detailed flowchart of an action generation method based on a multimodal large model provided in one embodiment of the present application.

[0017] Figure 4 A schematic diagram of functional modules of an action generation system based on a multimodal large model provided in one embodiment of the present application.

[0018] Figure 5 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] The potential of large multimodal models in complex information processing is as follows.

[0021] 1. Multimodal pre-trained models: Multimodal pre-trained models have been able to learn large-scale multimodal datasets, including intrinsic semantic connections across modalities, thus making breakthrough progress in multimodal understanding and generation tasks.

[0022] 2. Model architecture: Multimodal large models can understand and integrate information from different modalities. Native multimodal large model architectures represented by "AnyGPT", "Chameleon", "Anole", etc. achieve a more natural and rich human-computer interaction experience, supporting multiple modalities such as text, voice, image, and music.

[0023] 3. Excellent performance: In the OpenCompass multimodal evaluation field and OCRBench test, many large multimodal models performed well, demonstrating advanced performance in tasks such as text recognition and visual question answering.

[0024] 4. Application areas: Multimodal technology has been widely used in autonomous driving, medical assisted diagnosis, smart home, virtual reality games and emotional companionship.

[0025] Action sequence generation technology is an important research direction in the field of artificial intelligence and robotics. It involves selecting and generating action sequences from a series of basic actions to achieve a specific goal. The following introduces some technical terms involved in action sequence generation technology.

[0026] 1. Action representation: Define how to represent the action. Common methods include joint angles, coordinate positions, velocities, and accelerations.

[0027] 2. Basic strategy: The generation of action sequences can be achieved based on a predefined rule base or an optimization algorithm. Rule-based control relies on predefined rules, while optimization algorithms look for the best action sequence.

[0028] 3. Generation methods: With the development of deep learning, action sequence generation algorithms have evolved from physical models (inverse kinematics models) to data-driven methods, such as autoregressive models, variational autoencoders (VAE), normalized flows, generative adversarial networks (GANs), and denoising diffusion probabilistic models (DDPMs).

[0029] 4. Task-driven: including single task action and compound task action. Single task action means that the robot performs a single clear action, while compound task action involves a combination of action targets and action sequences.

[0030] 5. Interactivity: Action sequence generation needs to consider the interaction between the robot and the environment or humans. Interactive actions require the robot to be able to adjust according to environmental feedback or human instructions.

[0031] 6. Autonomy: Action sequences can be either preset or adaptive. Preset actions follow a set procedure, while adaptive actions can adjust according to changes in the environment.

[0032] 7. Real-time: Action sequence generation needs to consider real-time, that is, the robot needs to generate and execute actions within a limited time.

[0033] The purpose of this application is to realize action generation in a multimodal environment. The generated actions provide a clear execution entry for physical robots and open application programming interfaces (APIs), and have good application prospects.

[0034] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0035] The action generation method based on the multimodal large model provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, or it can be integrated on the server 104, or it can be placed on the cloud or other servers. The terminal 102 can send the action type and multimodal instructions of the current environment input by the user to the server 104, and the server 104 generates an action based on the received action type and multimodal instructions of the current environment. The server 104 can feedback the generated action to the terminal 102. In addition, in some embodiments, the action generation method based on the multimodal large model can also be implemented by the server 104 or the terminal 102 alone.

[0036] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.

[0037] In an exemplary embodiment, Figure 2 and Figure 3As shown, a method for generating actions based on a multimodal large model is provided, which can be used for generating actions during browser operation or during robot arm operation. The method is executed by a computer device, specifically, it can be executed by a computer device such as a terminal or a server alone, or it can be executed by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used for explanation, and the steps include the following steps 201 to 205.

[0038] Step 201, obtaining the action type and multimodal instruction of the current environment input by the user.

[0039] In an exemplary embodiment, the user can input commands in multiple modes such as voice, text, gesture, etc. The user defines the action type required by the current environment according to the relevant environment of the action sequence currently desired to be executed. The action type includes basic actions, combined actions, and restricted actions.

[0040] Basic actions are the smallest units that can be executed in the current environment, such as program call interfaces, mechanical movement methods, etc.

[0041] Combined actions are based on basic action types and define complex continuous action processes.

[0042] Restricted actions are inverse constraints, defining rules that basic actions must not be combined.

[0043] Two examples of different action types are provided below, namely a browser operation example and a robotic arm operation example, as shown in Tables 1 to 6.

[0044] Table 1 Examples of basic actions in browser operations

[0045] Table 2 Examples of combined actions in browser operations

[0046] Table 3 Examples of restricted actions in browser operations

[0047] Table 4 Basic action examples in the robot arm operation example

[0048] Table 5 Examples of combined actions in the robot arm operation example

[0049] Table 6 Examples of restricted actions in the robot operation example

[0050] Step 202: Based on the first multimodal large model, the multimodal instruction is understood and the steps are separated to obtain the user control intention in text form. The user control intention includes a plurality of continuous processing steps.

[0051] In an exemplary embodiment, the first multimodal large model is a pre-trained intent recognition agent. The input multimodal instructions are serialized through relevant technical methods such as modal tokenizers, and then the control intent is understood. The user intent is split into steps through the intent recognition agent to obtain the user control intent in text form.

[0052] Table 7 provides examples of prompt words of the first multimodal large model in the step decomposition of the robot arm operation, and Table 8 provides examples of output results of the first multimodal large model for the robot arm operation and the browser operation.

[0053] Table 7 Examples of prompt words for step splitting in a multimodal large model

[0054] Table 8 Example of user input and model output

[0055] Step 203, for any processing step in the user control intention, collect current environmental information, and use a second multimodal large model to perform environmental perception based on the user control intention and the current environmental information to obtain an environmental perception result.

[0056] In an exemplary embodiment, the second multimodal large model is a pre-trained environment perception agent. First, the current environment information such as images and recordings of the surrounding environment is collected, and each modal information is analyzed, extracted and identified, and the relationship between the environment and the current control target is understood in combination with the user's control intention, and the environment perception result is output.

[0057] Table 9 provides examples of prompt words for the second multimodal large model to recognize image information, and Table 10 provides input and output examples of the second multimodal large model in robot arm operation and browser operation.

[0058] Table 9 Examples of prompt words for image information recognition using the second multimodal large model

[0059] Table 10 Input and output examples of the second multimodal large model

[0060] Step 204, using the third multimodal large model to generate the next action according to the executed processing steps, the action type, the processing steps and the environmental perception results.

[0061] In an exemplary embodiment, the third multimodal large model is a pre-trained action generation agent. The action generation agent generates the next action based on the environmental perception results and the user's control intention, and refers to the preset action type. At the same time, it avoids generating restricted actions and tries to generate combined actions.

[0062] Table 11 provides examples of prompt words for the next step of the third multimodal large model generation action, and Table 12 provides examples of output results of the third multimodal large model generation action in robot arm operation and browser operation.

[0063] Table 11 Examples of next action prompt words generated by the third multimodal large model

[0064] Table 12 Example of the third multimodal large model generation action

[0065] In order to ensure the execution accuracy of the action, between step 204 and step 205, it also includes: based on the environmental perception result and the user control intention, using the fourth multimodal large model to check the quality of the next action. If the quality of the next action meets the requirements, step 205 is executed. If the quality of the next action does not meet the requirements, step 204 is executed.

[0066] In an exemplary embodiment, the fourth multimodal large model is a pre-trained action analysis agent. The action analysis agent re-checks the quality of action generation according to the environmental perception results and the user's control intention. If there is a problem, the modification suggestion is returned to the action generation agent so that it can regenerate the action. If there is no problem, it is executed immediately or the user is asked for a second confirmation. Among them, the user can be asked for a second confirmation by a virtual API Agent robot using a pop-up window or command line input, or by a physical robot using physical buttons or fixed voice confirmation instructions.

[0067] Step 205: Call the corresponding action interface according to the next step action to execute the processing step.

[0068] Robotic Arm Example: If the action is init(), the robot will execute the action "Initialize and start the robot API". This action is executed according to the program file pre-stored by the user in the robot.

[0069] Browser example: If the action is navigateToURL(xxx--url), the browser executor will execute the browser handle to jump to the target website and return the jump completion status.

[0070] In order to further ensure the execution accuracy of the action, the action generation method based on the multimodal large model also includes steps 206 and 207.

[0071] Step 206: Perform action checking using the pre-trained action checking agent and environment perception agent.

[0072] Specifically, the current environment information is collected again, and the environment perception is performed through the environment perception agent to obtain the environment perception result after execution. The action checking agent determines whether the user control intention is completed in this action execution based on the environment perception result before execution and the environment perception result after execution, combined with the user control intention, the currently executed action and the standard operating procedure (SOP). If the user control intention is completed, the operation is terminated after saving the action log. If the user control intention is not completed, step 204 is repeated after saving the action log.

[0073] Table 13 provides examples of prompt words used by the action check agent for inspection, and Table 14 provides examples of the output results of the action check agent.

[0074] Table 13 Examples of prompt words for the action check agent to perform inspection

[0075] Table 14 Example of output results of action checking agent

[0076] Step 207, collecting the log information, prompt information generated during the execution of each processing step and the data results generated after the execution of each step, and storing them in a log database.

[0077] This application uses user-preset action types to adapt to various scenarios and provide users with a highly generalized action generation function. Multimodal large model technology is used to extract and identify user intentions, and a complete user intention is analyzed in combination with the environment in which the current executor is located to derive a step-by-step user intention for action generation. Environmental analysis and step-by-step user intention generation actions are used to avoid generating restricted actions, and combined actions are generated first to improve the accuracy of action generation. Check and analyze the surrounding environment before and after the action is executed. After the action is executed, compare the multimodal environmental collection information, environmental analysis results, and user intentions before and after the execution to determine whether the action execution meets the user's intention.

[0078] The beneficial effects of this application include at least the following points.

[0079] 1. Multimodal instructions and environmental perception: Users can input instructions in multiple modes such as voice, text, gestures, etc. The multimodal large model can quickly identify the information contained and give analysis results. Similarly, multimodal environmental data is input to the multimodal large model for environmental analysis, allowing the multimodal large model to understand the objects, obstacles, etc. in the current environment, and understand the relationship between the environment and the current control target. This allows users to get rid of the impact of having to personally measure and calculate environmental variables every time, optimizes the user experience, and makes action generation more intelligent.

[0080] 2. Improved accuracy of complex actions: The multimodal large model generates action sequences step by step, and performs pre-checks and post-checks for each step. Through multi-step generation and pre- and post-checks, the robot can update environmental variables in real time and make plans and decisions for the next step of action generation.

[0081] 3. Real-time interaction and adjustment: During the execution of an action, the robot can continue to accept the user's new control intentions. During the multi-step generation process, the robot can confirm with the user whether to execute the planned action.

[0082] 4. Improved generalization: The concept of action sequence covers a wide range and can be understood as a function call sequence with an environment. Both physical robots and virtual API Agents are controllable objects. If it is a physical robot (such as a robotic arm), the pre-stored action file can be used as the basic action. If it is a virtual API Agent (such as a Web browser), the keyboard and mouse operation and download operation can be used as the basic action. The model then generates a basic action sequence to realize the user's control intention.

[0083] In summary, this application provides an action generation method with high accuracy, strong controllability and good generalization, allowing users to control the physical robot and the virtual API Agent at the same time through this unified action generation method. It solves the challenges of interactivity and generalization in current robot control, and makes full use of the advantages of language understanding, task planning, and multimodal environmental perception of multimodal large models. It can parse user instructions in real time, understand environmental information, and generate action sequences that adapt to the current environment, thereby realizing efficient and autonomous control of the robot. It realizes high-accuracy generation and real-time interaction of complex actions.

[0084] Based on the same inventive concept, the embodiment of the present application also provides an action generation system for implementing the above-mentioned action generation method. The implementation scheme for solving the problem provided by the system is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more action generation system embodiments provided below can refer to the limitations on the action generation method above, and will not be repeated here.

[0085] In an exemplary embodiment, Figure 4 As shown, an action generation system is provided, including: a data acquisition module 401, an intention understanding module 402, an environment perception module 403, an action generation module 404 and an action execution module 405.

[0086] The data acquisition module 401 is used to acquire the action type and multimodal instructions of the current environment input by the user.

[0087] The intention understanding module 402 is used to understand the intention and split the steps of the multimodal instruction based on the first multimodal large model to obtain the user control intention in text form. The user control intention includes multiple continuous processing steps.

[0088] The environmental perception module 403 is used to collect current environmental information for any processing step in the user control intention, and use the second multimodal large model to perform environmental perception based on the user control intention and the current environmental information to obtain an environmental perception result.

[0089] The action generation module 404 is used to generate the next action using the third multimodal large model according to the executed processing steps, the action type, the processing steps and the environmental perception results.

[0090] The action execution module 405 is used to call the corresponding action interface according to the next step action to execute the processing step.

[0091] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 5As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store action types and multimodal instructions. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for generating an action based on a multimodal large model is implemented.

[0092] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0093] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0094] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0095] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0096] In this application, all actions to obtain signals, information or data are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0097] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0098] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0099] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0100] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for generating motion based on a multimodal large model, characterized in that: The action generation method based on the multimodal large model includes: Get the action type and multimodal instructions of the current environment input by the user; Based on the first multimodal large model, the multimodal instruction is understood and the steps are split to obtain the user control intention in text form; the user control intention includes a plurality of continuous processing steps; For any processing step in the user control intention, current environment information is collected, and based on the user control intention and the current environment information, a second multimodal large model is used to perform environment perception to obtain an environment perception result; Generate a next action using a third multimodal large model according to the executed processing steps, the action type, the processing steps and the environmental perception result; The corresponding action interface is called according to the next step action to execute the processing step.

2. The action generation method based on a multimodal large model according to claim 1 is characterized in that: The action types include basic actions, combined actions and restricted actions.

3. The action generation method based on a multimodal large model according to claim 1 is characterized in that: The multimodal instructions include voice, text and gesture.

4. The action generation method based on a multimodal large model according to claim 1 is characterized in that: The first multimodal large model is a pre-trained intention recognition agent, the second multimodal large model is a pre-trained environment perception agent, and the third multimodal large model is a pre-trained action generation agent.

5. The action generation method based on a multimodal large model according to claim 1, characterized in that: Before calling the corresponding action interface according to the next action, the action generation method based on the multimodal large model further includes: Based on the environmental perception result and the user control intention, using a fourth multimodal large model to check the quality of the next action; If the quality of the next action meets the requirements, the corresponding action interface is called according to the next action; If the quality of the next action does not meet the requirement, the next action is regenerated through the third multimodal large model until the quality of the next action meets the requirement.

6. The method for generating motion based on a multimodal large model according to claim 5, characterized in that: The fourth multimodal large model is a pre-trained motion analysis agent.

7. The action generation method based on a multimodal large model according to claim 1, characterized in that: The action generation method based on the multimodal large model also includes: Collect the log information, prompt information generated during the execution of each processing step, and the data results generated after the execution of each processing step, and store them in the log database.

8. An action generation system based on a multimodal large model, applied to the action generation method based on a multimodal large model according to any one of claims 1 to 7, characterized in that: The action generation system based on the multimodal large model includes: The data acquisition module is used to obtain the action type and multimodal instructions of the current environment input by the user; An intention understanding module, used to understand the intention and split the steps of the multimodal instruction based on the first multimodal large model to obtain the user control intention in text form; the user control intention includes multiple continuous processing steps; An environment perception module, for collecting current environment information for any processing step in the user control intention, and performing environment perception using a second multimodal large model based on the user control intention and the current environment information to obtain an environment perception result; An action generation module, configured to generate a next action using a third multimodal large model according to the executed processing steps, the action type, the processing steps and the environmental perception result; The action execution module is used to call the corresponding action interface according to the next step action to execute the processing step.

9. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the action generation method based on a multimodal large model as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for generating an action based on a multimodal large model described in any one of claims 1 to 7 is implemented.

Images