An intention understanding interaction method based on multimodal large model

Abstract

The present invention discloses an intention understanding interaction method based on a multimodal large model, comprising: clarifying the fine-grained tasks required for users to complete their goals in the current interaction scenario; performing fine-grained task classification and multimodal related subtask decomposition; collecting training data for fine-grained tasks and data sets for multimodal tasks and related subtasks; training the multimodal large model using the training data and data sets; and utilizing the trained multimodal large model to perform interactive control and process user input. The present invention is particularly suitable for scenarios requiring complex human-computer interaction, such as intelligent assistants and virtual assistants. It can significantly improve the system's perception of environmental changes and its ability to respond to user needs, reduce reliance on the collection of multimodal task data sets, and provide important reference and support for the development of human-computer interaction technology.

Description

Technical Field

[0001] The present invention belongs to the technical fields of artificial intelligence (AI) and human-computer interaction (HCI), and in particular relates to an intention understanding interaction method based on a multimodal large model, which aims to achieve accurate recognition and feedback of complex user instructions by processing multiple data modalities (such as text, images, voice, etc.). Background Art

[0002] With the rapid development of artificial intelligence (AI), the research and application of large multimodal models have become a key technology in intelligent interactive systems. The core of multimodal tasks lies in improving the system's ability to understand user intent by integrating information from multiple modalities, such as text, images, and speech. This approach can more accurately identify user intent, thereby improving the quality and efficiency of human-computer interaction. However, despite the great potential of large multimodal models, current technological development still faces challenges of high complexity and cost.

[0003] First, the complexity of multimodal tasks significantly increases the difficulty for models to understand user intent. Compared to traditional unimodal tasks, multimodal tasks require processing diverse data types from multiple modalities, making user intent identification more complex. This can require multiple simple task steps to achieve the user's goal. A major challenge in multimodal tasks is how to effectively split and plan tasks based on user input intent to accurately achieve the user's goal.

[0004] Secondly, the difficulty and high cost of data collection for multimodal tasks are another obstacle to their promotion and application. To achieve high-quality multimodal model training, a large amount of labeled data is often required. However, obtaining large-scale multimodal data and accurately labeling it is a very time-consuming and expensive task. Unlike single-modal data, the collection of multimodal data requires the coordination of multiple devices or systems, such as collecting video, audio and text information at the same time. This not only increases the complexity of hardware facilities, but also involves multiple technical links such as data synchronization, format conversion, and storage management. What is more complicated is that data labeling also requires professionals to label data of different modalities separately to ensure the consistency and accuracy of each modality. Therefore, while reducing the difficulty and cost of multimodal data collection, how to maintain the effectiveness of multimodal tasks is another major challenge for multimodal tasks. Summary of the Invention

[0005] The purpose of the present invention is to address the deficiencies of the existing technology and propose an intention understanding interaction method based on a multimodal large model.

[0006] To achieve the above objectives, the present invention provides an intention understanding interaction method based on a multimodal large model, comprising the following steps:

[0007] Task extraction: Identify the fine-grained tasks required for users to complete their goals in the current interaction scenario; these fine-grained tasks include target-finding tasks, small talk tasks, and exclusive knowledge question-answering tasks;

[0008] Fine-grained task classification and multimodal-related subtask decomposition: Fine-grained tasks are classified according to their type into fine-grained tasks that require multimodal information processing and fine-grained tasks that do not. Fine-grained tasks that require multimodal information processing are defined as multimodal tasks, which are then analyzed and decomposed into multiple related subtasks, including character selection and image description.

[0009] Fine-grained task data collection: Collecting training data for the fine-grained tasks, including single-task annotated data, multi-task annotated data, general data, and corresponding prompt templates;

[0010] Multimodal and related subtask data preparation: Collect the dataset of the multimodal task and the dataset of the related subtask. The training data in the dataset includes image-text pairs, general data, and corresponding prompt templates.

[0011] Training the task planning capability of a large multimodal model: combining the single-task annotated data, multi-task annotated data, and general data with the prompt template to train a large multimodal model;

[0012] Training the multimodal task execution capabilities of the multimodal large model: The dataset of the multimodal task is mixed with the datasets of related subtasks, and the multimodal large model is trained again. The multimodal large model is used to process user input and image information of the corresponding environment, and then determine user intent to achieve user goals. If there are multiple multimodal tasks, the corresponding data are mixed for joint training;

[0013] Model interaction control: Use the trained multimodal large model to process user input: Use the multimodal large model to plan tasks for user input, and call the corresponding modules in sequence according to the planned tasks; when multimodal information processing is required, use the multimodal large model to combine user input with images and audio to achieve the goal.

[0014] Furthermore, the classification according to the type of the fine-grained task includes: dividing fine-grained tasks that require multiple modal information into multimodal understanding interaction tasks, dividing fine-grained tasks that can be solved by only a single modal information into single-modal tasks, and dividing fine-grained tasks that require question and answer based on specific application scenarios into application-specific question and answer tasks. The application-specific question and answer tasks require the use of RAG technology to read relevant documents and then answer them.

[0015] Furthermore, collecting the training data of the fine-grained task includes:

[0016] Training data collection: Collect or generate conversation data generated in the current application scenario and use the conversation data as training data;

[0017] Fine-grained task label generation: Generate clear intent labels for the training data of task planning in the multimodal large model, that is, fine-grained task annotation data, to ensure that the multimodal large model can accurately generate precisely planned task sequences for users.

[0018] Furthermore, the collecting of the dataset of the multimodal task and the collecting of the dataset of the related subtasks at the same time include:

[0019] Label generation for multimodal data and related subtasks: In datasets processing multimodal tasks, labels are generated based on user input and captured images. Each label has a fixed format and includes a description of the selected object and the coordinates of the upper left and lower right corners of the box. In training data processing related subtasks, labels are generated based on user input and images. The label format varies depending on the task.

[0020] Prompt template design: Prompt templates are designed for task planning and multimodal tasks respectively to guide the multimodal large model to accurately plan logical task sequences, accurately identify user intentions, and complete interaction with users.

[0021] Furthermore, the training of the multimodal large model includes:

[0022] Using training data and prompt templates, the multimodal large model is fine-tuned in a supervised manner to obtain a multimodal large model that is used to accurately plan task sequences based on user input, process multimodal information, accurately identify user intentions, and complete interactions.

[0023] Furthermore, the processing of user input using the trained multimodal large model includes:

[0024] Task planning: Based on user input, a multimodal large model is used to plan tasks based on user intent and output a task sequence to ensure that the task sequence is logical and meets user needs;

[0025] Task processing strategy: Understand user intent, select based on the existing fine-grained task list, sort the selected fine-grained tasks according to logical relationships, and output the final fine-grained task sequence; based on the fine-grained task sequence, determine whether each fine-grained task requires multimodal information to achieve the goal, and assign fine-grained tasks that do not require multimodal information to the corresponding module for processing; to ensure that the user's goals are ultimately achieved and improve user experience and response quality.

[0026] The present invention achieves the following beneficial effects: It performs fine-grained task division based on user input during user conversations, facilitating accurate understanding of user intent and achieving user goals. It also enhances the effectiveness of multimodal tasks through the collection of multimodal subtask data, reducing the difficulty and cost of data collection for multimodal tasks. Compared to traditional human-computer interaction systems, this invention can better handle complex multimodal application scenarios, provide a more intelligent conversational experience, and enhance the accuracy and practicality of the interactive system. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a flow chart of the method of the present invention;

[0028] Figure 2 It is the multimodal large model of the present invention that intends to understand the interactive control diagram. DETAILED DESCRIPTION

[0029] The method proposed in the present invention is further described in detail below with reference to the accompanying drawings and specific examples.

[0030] like Figure 1 As shown, the present invention proposes an intention understanding interaction method based on a multimodal large model. This method can, according to the user's input, first plan a fine-grained task sequence that is logical and can meet the user's needs based on a large model that has been fine-tuned and has the ability to plan tasks; then, the multimodal intention understanding interaction control framework is used to determine whether the fine-grained task requires multimodal information. If necessary, the multimodal task is handed over to the large model that has been fine-tuned and has the ability to handle multimodal tasks for execution, otherwise it is handed over to the corresponding functional module for execution. In addition, the interaction framework is also responsible for determining whether it is necessary to use RAG technology for professional Q&A during task execution, so as to achieve more accurate responses. The whole process is specifically divided into the following steps:

[0031] Step 1: Task extraction

[0032] In a specific embodiment of the present invention, the first step is to perform task extraction to identify the fine-grained tasks that may be required in user interactions. First, it is necessary to consider the possible conversations that may occur in the interaction, analyze which fine-grained task combinations are needed based on these conversations to achieve the goal, and finally collect all the required fine-grained tasks for subsequent data preparation and model training. The following are specific implementation steps and examples, taking the laboratory assistant application as an example:

[0033] Step 1.1: Clear Mission

[0034] In this step, we first identify the conversational data that may occur during the interaction between the lab assistant and the user. Then, by analyzing this data, we can determine the combination of fine-grained tasks required to achieve the user's goal. These fine-grained tasks include, but are not limited to, finding the target, greeting, and introducing the lab.

[0035] Example: A lab assistant's interactions might involve fine-grained tasks such as:

[0036] Find your target

[0037] ·say hello

[0038] ·Introduction to the laboratory

[0039] Pick up

[0040] Step 1.2: Task Type Classification

[0041] In this step, it's important to clearly define which tasks are multimodal and which are application-specific question-answering tasks. Tasks that require information from multiple modalities to achieve their objectives and achieve better results are considered multimodal tasks, while tasks that can be achieved with information from only a single modality are considered unimodal tasks. Application-specific question-answering tasks vary depending on the application scenario, and users may ask questions specific to that scenario.

[0042] In the application scenario of laboratory assistant, finding the target in the fine-grained task is defined as a multimodal task, because it requires the user input and the corresponding environment image information to determine the target the user wants to find. The user input is as follows:

[0043] User: Please come over to the guest and say hello.

[0044] In this conversation, the user's needs are broken down into two fine-grained tasks: finding the target and greeting. The target-finding task requires accurate locating the target based on user input and the corresponding image of the environment to meet the user's needs. The greeting task, on the other hand, only requires a single modal piece of information to achieve its purpose.

[0045] Question answering tasks that require RAG technology to search for relevant documents and answer questions belong to specific application domains. In the laboratory assistant application scenario, for example:

[0046] User: Please introduce the situation of your laboratory.

[0047] Step 2: Data Preparation

[0048] In this embodiment of the present invention, step 2 is intended to prepare for the two training phases of the multimodal large model. The data preparation phase includes data collection, data labeling, and template design, as well as providing the model with necessary information such as dialogue labels, fine-grained task lists, and prompt templates. The following are the specific steps for data preparation:

[0049] Step 2.1: Training data collection

[0050] First, we need to collect data on user interactions in the lab assistant scenario. This data should cover all possible scenarios. We can simulate a portion of these conversations, then use a large model to generate content with similar semantic expressions based on this data. Finally, we manually filter the data to identify realistic scenarios and construct training, test, and validation sets. Here's an example:

[0051] User: Please go up to the guest, greet him, and introduce yourself.

[0052] Step 2.2: Fine-grained task label generation

[0053] In this step, for each user data point, the user's intent must be understood and then fine-grained task division is performed. These labels will be used to train the multimodal large model's task planning capabilities. The fine-grained tasks in the lab assistant application scenario include those defined in step 1, such as finding the target, greeting, and introducing. For the user example shown in step 2.1, the corresponding labels are "find the target#greeting#introduction." Label generation begins with preliminary labeling using the large model, followed by manual correction.

[0054] Step 2.3: Generate multimodal data and related subtask labels

[0055] This step labels the multimodal data and its related subtasks. In the lab assistant, the multimodal task is to select the target pointed to by the user. The model makes a judgment based on the user input and the relevant environmental image information, and finally outputs the position of the target on the image. Multimodal data requires the unification of text, images, and labels. The text data is obtained through step 2.1, the image data is obtained by artificially simulating the interactive scene and taking photos, and the corresponding labels are manually obtained using the image annotation tool. The label data of the relevant subtasks are obtained by simple processing of the public dataset.

[0056] Step 2.4: Tip Template Design

[0057] To guide large multimodal models to better complete task planning and execution, we can design prompt templates suitable for different tasks. These templates help large multimodal models more accurately understand the user's goals and effectively combine multimodal data for reasoning and decision-making. Through appropriate prompt design, the model can better utilize its capabilities in complex task scenarios and enhance its performance in human-computer interaction. This not only improves the model's efficiency in processing multimodal information but also ensures its accuracy and flexibility in real-world tasks.

[0058] In the application scenario of a laboratory assistant, the prompt template for task planning can be:

[0059]

[0060]

[0061]

[0062] The prompt template for the multimodal box selection target task can be:

[0063]

[0064]

[0065] Step 3: Model training

[0066] In this embodiment of the present invention, step 3 aims to train the multimodal large model twice, with two different tasks: fine-grained task planning and multimodal target selection. The following are the specific model training steps, taking the laboratory assistant scenario as an example:

[0067] Step 3.1: Task planning multimodal large model training

[0068] In this sub-step, the present invention will train a large multimodal model that can understand user input and then plan a fine-grained task sequence that meets user needs. This model will achieve the purpose of accurate planning through supervised fine-tuning. For the laboratory assistant scenario, an open source large-scale pre-trained multimodal model (such as Qwen-VL) is used. Then, using the fine-grained task planning label dataset and the corresponding prompt template, the multimodal large model is fine-tuned in a supervised manner.

[0069] Step 3.2: Multimodal box selection target large model training

[0070] In this sub-step, the present invention will train a large multimodal model so that it can understand user semantics and image information at the same time, combine the two for reasoning, and finally select the target to meet user needs. Similarly, an open source large-scale pre-trained multimodal model (such as Qwen-VL) is used, and the multimodal task data and related subtask data and corresponding prompt templates are used to perform supervised fine-tuning on the model to ensure that the target can be accurately selected. When there are multiple multimodal tasks, the data of multiple tasks can be mixed for joint training.

[0071] Step 4: Interactive Control

[0072] In the embodiment of the present invention, step 4 is a key stage of interactive control, such as Figure 2 As shown in the figure, fine-grained task planning is performed based on user input, and then the tasks are executed one by one according to the task sequence. At the same time, different tasks are sent to different modules for processing based on whether they require multimodal information and whether they belong to exclusive domain questions and answers, so as to achieve smooth user interaction.

[0073] Step 4.1: Mission Planning

[0074] The interactive framework will first combine the user's input with the task planning prompt template and submit it to the multimodal large model for semantic understanding. It will then select the required tasks based on the fine-grained task list, output a fine-grained task sequence in a logical execution order, and finally traverse and execute according to the fine-grained task sequence to ultimately achieve the user's goal.

[0075] Step 4.2: Task processing strategy

[0076] In this sub-step, the interactive framework processes the fine-grained task sequence generated in step 4.1 for the lab assistant's application scenario. First, the system determines whether each task is multimodal. If so, the multimodal task module is called, and the user input and captured environmental image information are obtained, which are then passed to the multimodal large model for processing. The multimodal large model outputs the target's position information based on the input, and then calls the target tracking module, allowing the lab assistant to track the target based on the target's position. If the task is not multimodal, the system further distinguishes whether the task is a domain-specific question-and-answer task. If it is a domain-specific question-and-answer task, the system uses RAG technology to retrieve relevant documents and provide professional answers. If it is not a domain-specific task, the corresponding unimodal processing module is used for answering and processing. This process design ensures the efficient processing of multimodal and unimodal tasks and the precise allocation of task types.

[0077] After the above four steps, intention understanding interaction based on a multimodal large model can be achieved.

[0078] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only.

[0079] It will be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A multimodal large model-based intention understanding interaction method, characterized by: The following steps are involved: Identify the fine-grained tasks required for users to complete their goals in the current interaction scenario; these fine-grained tasks include target-finding tasks, small talk tasks, and exclusive knowledge question-answering tasks; Classify the fine-grained tasks according to their types into fine-grained tasks that require multimodal information processing and fine-grained tasks that do not require multimodal information processing; The fine-grained task requiring multimodal information processing is defined as a multimodal task, and the multimodal task is analyzed to decompose multiple related subtasks, wherein the related subtasks include selecting a person and describing an image; Collecting training data for the fine-grained tasks, the training data including single-task annotated data, multi-task annotated data, general data, and corresponding prompt templates; Collecting a dataset for the multimodal task and a dataset for the related subtasks, wherein the training data in the dataset includes image-text pairs, general data, and corresponding prompt templates; The single-task annotated data, the multi-task annotated data and the general data are combined with corresponding prompt templates to train a multimodal large model; The dataset of the multimodal task and the datasets of related subtasks are mixed and the multimodal large model is trained again. The multimodal large model is used to process the user input and the image information of the corresponding environment, and then determine the user's intention to achieve the user's goal. If there are multiple multimodal tasks, the training data corresponding to each multimodal task is integrated into a unified dataset, and then these tasks are jointly trained. Use the trained multimodal large model to process user input: Use the multimodal large model to plan tasks for user input, and call the corresponding modules in sequence according to the planned tasks; when multimodal information processing is required, use the multimodal large model to combine user input with images and audio to achieve the goal.

2. The intention understanding interaction method based on a multimodal large model according to claim 1 is characterized in that: The classification according to the type of the fine-grained task includes: classifying fine-grained tasks that require multiple modal information into multimodal understanding interaction tasks, classifying fine-grained tasks that can be solved with only a single modal information into single-modal tasks, and classifying fine-grained tasks that require question and answer based on specific application scenarios into application-specific question and answer tasks. The application-specific question and answer tasks require the use of RAG technology to read relevant documents and then answer them.

3. The intention understanding interaction method based on a multimodal large model according to claim 1 is characterized in that: The collecting of training data for the fine-grained task includes: Collect or generate conversation data generated in the current application scenario and use the conversation data as training data; Generate clear intent labels for the training data of task planning in the multimodal large model, that is, fine-grained task annotation data, to ensure that the multimodal large model can accurately generate precisely planned task sequences for users.

4. The intention understanding interaction method based on a multimodal large model according to claim 1 is characterized in that: The collecting of the dataset of the multimodal task and the dataset of the related subtasks simultaneously includes: In the dataset for multimodal tasks, labels are generated based on user input and captured images. Each label has a fixed format and includes a description of the selected object and the coordinates of the upper left and lower right corners of the box. In the training data for related subtasks, labels are generated based on user input and images, and the label format varies depending on the task. Prompt templates are designed for task planning and multimodal tasks respectively to guide the multimodal large model to accurately plan logical task sequences, identify user intentions, and complete interaction with users.

5. The intention understanding interaction method based on a multimodal large model according to claim 1 is characterized in that: The training of the multimodal large model includes: Using training data and prompt templates, the multimodal large model is fine-tuned in a supervised manner to obtain a multimodal large model for accurately planning task sequences based on user input, processing multimodal information, accurately identifying user intentions, and completing interactions.

6. The intention understanding interaction method based on a multimodal large model according to claim 1 is characterized in that: The method of processing user input using the trained multimodal large model includes: Based on user input, a multimodal large model is used to plan tasks based on user intent and output a task sequence to ensure that the task sequence is logical and meets user needs; Understand the user's intention, select according to the existing fine-grained task list, sort the selected fine-grained tasks according to logical relationships, and output the final fine-grained task sequence; based on the fine-grained task sequence, determine whether each fine-grained task requires multimodal information to achieve the goal, and assign fine-grained tasks that do not require multimodal information to the corresponding module for processing.

Images