Multi-modal large model-based intent understanding interaction method

Abstract

The invention discloses an intention understanding interaction method based on a multi-modal large model. The intention understanding interaction method comprises the following steps: determining a fine-grained task required by a user to complete a target in a current interaction scene; performing fine-grained task classification and multi-modal related sub-task decomposition; collecting training data of fine-grained tasks and data sets of multi-modal tasks and related subtasks; training a multi-modal large model through the training data and the data set; and performing interaction control by using the trained multi-modal large model, and processing user input. The method is particularly suitable for scenes needing complex man-machine interaction, such as intelligent assistants and virtual assistants, the sensing ability of the system to environment changes and the response ability to user requirements can be remarkably improved, dependence on multi-modal task data set collection is reduced, and important reference and support are provided for development of the man-machine interaction technology.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence (AI) and human-computer interaction (HCI), and in particular relates to an intention understanding interaction method based on a multimodal large model, which aims to realize accurate recognition and feedback of user complex instructions by processing multiple data modalities (such as text, image, voice, etc.). Background Art

[0002] With the rapid development of artificial intelligence technology, the research and application of multimodal large models have gradually become key technologies in intelligent interactive systems. The core of multimodal tasks is to improve the system's ability to understand user intent by integrating information sources of multiple modalities, such as text, images, and voice. This approach can more accurately identify user intent and thus improve the quality and efficiency of human-computer interaction. However, despite the great potential of multimodal large models, current technological development still faces challenges of high complexity and cost.

[0003] First, the complexity of multimodal tasks significantly increases the difficulty for models to understand user intent. Compared with traditional unimodal tasks, multimodal tasks need to process different types of data from multiple modalities, which makes user intent recognition more complicated and may require multiple simple task steps to achieve the user's goal. How to accurately achieve the user's goal through effective task splitting and planning based on the user's input intent is a major challenge for multimodal tasks.

[0004] Secondly, the difficulty and high cost of data collection for multimodal tasks are another obstacle to their promotion and application. To achieve high-quality multimodal model training, a large amount of labeled data is often required. However, obtaining large-scale multimodal data and accurately labeling it is a very time-consuming and expensive task. Unlike unimodal data, the collection of multimodal data requires the coordination of multiple devices or systems, such as collecting video, audio and text information at the same time, which not only increases the complexity of hardware facilities, but also involves multiple technical links such as data synchronization, format conversion, and storage management. What's more complicated is that data labeling also requires professionals to label data of different modalities separately to ensure the consistency and accuracy of each modal data. Therefore, while reducing the difficulty and cost of multimodal data collection, how to maintain the effect of multimodal tasks is another major challenge for multimodal tasks. Summary of the invention

[0005] The purpose of the present invention is to address the deficiencies of the prior art and to propose an intention understanding interaction method based on a multimodal large model.

[0006] To achieve the above object, the present invention provides an intention understanding interaction method based on a multimodal large model, comprising the following steps:

[0007] Task extraction: clarify the fine-grained tasks required for users to complete their goals in the current interaction scenario; the fine-grained tasks include target-finding tasks, small talk tasks, and exclusive knowledge question-and-answer tasks;

[0008] Fine-grained task classification and multimodal related subtask decomposition: Classify the fine-grained tasks according to their types into fine-grained tasks that require multimodal information processing and fine-grained tasks that do not require multimodal information processing; define the fine-grained tasks that require multimodal information processing as multimodal tasks, analyze the multimodal tasks, and decompose them into multiple related subtasks, including character selection and image description;

[0009] Fine-grained task data collection: collecting training data of the fine-grained tasks, the training data including single-task annotation data, multi-task annotation data, general data and corresponding prompt templates;

[0010] Multimodal and related subtask data preparation: Collect the data set of the multimodal task and the data set of the related subtask, the training data in the data set includes image-text pairs, general data and corresponding prompt templates;

[0011] Task planning capability for training a large multimodal model: combining the single-task annotated data, multi-task annotated data and general data with the prompt template to train a large multimodal model;

[0012] Training the multimodal task execution capability of the multimodal large model: Mix the data set of the multimodal task and the data set of related subtasks, and train the multimodal large model again. The multimodal large model is used to process the user's input and the image information of the corresponding environment, and then judge the user's intention to achieve the user's goal; if there are multiple multimodal tasks, mix the corresponding data for joint training;

[0013] Model interaction control: Use the trained multimodal large model to process user input: Use the multimodal large model to plan tasks for user input, and call the corresponding modules in sequence according to the planned tasks; when multimodal information processing is required, use the multimodal large model to combine user input with images and audio to achieve the goal.

[0014] Furthermore, the classification according to the type of the fine-grained task includes: classifying fine-grained tasks that require multiple modal information into multi-modal understanding interaction tasks, classifying fine-grained tasks that can be solved with only a single modal information into single-modal tasks, and classifying fine-grained tasks that require question and answer based on specific application scenarios into application-specific question and answer tasks. The application-specific question and answer tasks require the use of RAG technology to read relevant documents before answering.

[0015] Furthermore, the collecting of training data for the fine-grained task includes:

[0016] Training data collection: collecting or generating conversation data generated in the current application scenario, and using the conversation data as training data;

[0017] Fine-grained task label generation: Generate clear intent labels for the training data of task planning in the multimodal large model, that is, fine-grained task annotation data, to ensure that the multimodal large model can accurately generate precise task sequences for users.

[0018] Furthermore, the collecting of the data set of the multimodal task and the collecting of the data set of the related subtasks at the same time include:

[0019] Multimodal data and related subtask label generation: In the data set for processing multimodal tasks, corresponding labels are generated based on user input and captured images. The format of each label is fixed, and the label includes a description of the selected object and the coordinate positions of the upper left corner and lower right corner of the box. In the training data for processing related subtasks, corresponding labels are generated based on user input and images. The label format changes with the task.

[0020] Prompt template design: Prompt templates are designed for task planning and multimodal tasks respectively to guide the multimodal large model to accurately plan a logical task sequence, accurately identify user intentions, and complete interaction with users.

[0021] Furthermore, the training of the multimodal large model includes:

[0022] The multimodal large model is fine-tuned in a supervised manner using the training data and the prompt template, and the multimodal large model is trained to accurately plan a task sequence according to user input, process multimodal information, accurately identify user intent and complete interaction.

[0023] Furthermore, the processing of user input using the trained multimodal large model includes:

[0024] Task planning: Based on user input, use a multimodal large model to plan tasks for user intent and output a task sequence to ensure that the task sequence is logical and meets user needs;

[0025] Task processing strategy: Understand user intentions, select based on the existing fine-grained task list, sort the selected fine-grained tasks according to logical relationships, and output the final fine-grained task sequence; determine whether each fine-grained task requires multimodal information to achieve the goal based on the fine-grained task sequence, and assign fine-grained tasks that do not require multimodal information to the corresponding modules for processing; to ensure that the user's goals are ultimately achieved and to improve user experience and response quality.

[0026] The beneficial effects of the present invention are as follows: in user dialogues, the present invention performs fine-grained task division based on user input, facilitates accurate understanding of user intent and achievement of user goals, and at the same time enhances the effect of multimodal actual tasks through multimodal related subtask data, reducing the difficulty and cost of data collection for multimodal tasks. Compared with traditional human-computer interaction systems, the present invention can better cope with complex multimodal application scenarios, provide a more intelligent dialogue experience, and improve the accuracy and practicality of the interactive system. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a flow chart of the method of the present invention;

[0028] Figure 2 It is the multimodal large model of the present invention that intends to understand the interactive control diagram. DETAILED DESCRIPTION

[0029] The method proposed by the present invention is further described in detail below with reference to the accompanying drawings and specific examples.

[0030] like Figure 1 As shown, the present invention proposes an intention understanding interaction method based on a multimodal large model. This method can first plan a fine-grained task sequence that is logical and can meet user needs based on a large model that has been fine-tuned and has task planning capabilities according to user input; then, the multimodal intention understanding interaction control framework is used to determine whether the fine-grained task requires multimodal information. If necessary, the multimodal task is handed over to the large model that has been fine-tuned and has multimodal task processing capabilities for execution, otherwise it is handed over to the corresponding functional module for execution. In addition, the interaction framework is also responsible for determining whether it is necessary to use RAG technology for professional Q&A during task execution, so as to achieve more accurate responses. The whole process is specifically divided into the following steps:

[0031] Step 1: Task extraction

[0032] In a specific implementation of the present invention, the first step is to extract tasks to clarify the fine-grained tasks that may be required in user interaction. First, it is necessary to consider the possible dialogues that may occur in the interaction, analyze which fine-grained task combinations are needed to achieve the goal based on these dialogues, and finally collect all the required fine-grained tasks for subsequent data preparation and model training. The following are specific implementation steps and examples, taking the laboratory assistant application as an example:

[0033] Step 1.1: Clear mission

[0034] In this step, we first need to identify the conversation data that may appear in the interaction between the lab assistant and the user, and then analyze this data to find out the combination of fine-grained tasks required to achieve the user's goal. These fine-grained tasks include but are not limited to finding the target, greeting, introducing the lab, etc.

[0035] Example: A lab assistant's interactions might involve fine-grained tasks such as:

[0036] Find your target

[0037] ·say hello

[0038] ·Introduction to the laboratory

[0039] Pick up

[0040] Step 1.2: Task type classification

[0041] In this step, it is necessary to clarify which are multimodal tasks and which are application-specific question-answering tasks. Tasks that require multiple modal information to achieve the goal and achieve better results are multimodal tasks, while tasks that can achieve the goal by relying on only a single modal information are unimodal tasks. Application-specific question-answering tasks vary with the application scenario, and users may ask questions specific to the relevant scenario.

[0042] In the application scenario of laboratory assistant, the fine-grained task of finding the target is defined as a multimodal task, because it requires the user input and the corresponding environment image information to determine the target that the user wants to find. The user input is as follows:

[0043] User: Please come over to the guest and say hello.

[0044] In this conversation, the user's needs are broken down into two fine-grained tasks: finding the target and saying hello. The task of finding the target requires accurate finding of the target based on the user input and the corresponding environment image information to meet the user's needs. The task of saying hello only requires a single modal information to achieve the goal.

[0045] Question-answering tasks that require the use of RAG technology to search for relevant documents for answers belong to application-specific question-answering tasks, such as in the laboratory assistant application scenario:

[0046] User: Please introduce the situation of your laboratory.

[0047] Step 2: Data Preparation

[0048] In the embodiment of the present invention, step 2 is intended to prepare for the two trainings of the multimodal large model. The data preparation stage includes data collection, data labeling and template design, as well as providing the model with necessary information, such as dialogue labels, fine-grained task lists and prompt templates. The following are the specific steps of data preparation:

[0049] Step 2.1: Training data collection

[0050] First, we need to collect data on user interactions in the lab assistant scenario. This data should include possible situations. We can first obtain a portion of these conversations through simulation, then use a large model to generate content with similar semantic expressions based on this data, and finally manually filter out data that conforms to reality to build training sets, test sets, and validation sets. The following is an example:

[0051] User: Please go up to the guest, greet him, and introduce yourself.

[0052] Step 2.2: Fine-grained task label generation

[0053] In this step, for each user data, it is necessary to understand the user's intention and then divide the task into fine-grained tasks. These labels will be used to train the task planning capabilities of the multimodal large model. The fine-grained tasks in the application scenario of laboratory assistant include those defined in step 1, such as finding the target, saying hello, and introducing. For the user example shown in step 2.1, the corresponding labels are finding the target#saying hello#introducing. For the generation of labels, the large model is used for preliminary labeling, and then manual correction is performed.

[0054] Step 2.3: Multimodal data and related subtask label generation

[0055] This step annotates the multimodal data and its related subtasks. In the lab assistant, the multimodal task is to select the target pointed to by the user. The model makes a judgment based on the user input and related environmental image information, and finally outputs the position of the target on the image. Multimodal data requires the unification of text, images, and labels. The text data is obtained through step 2.1, the image data is obtained by artificially simulating the interactive scene and taking photos, and the corresponding labels are obtained manually through the image annotation tool. For the relevant subtask label data, it is obtained by simple processing of the public data set.

[0056] Step 2.4: Tip Template Design

[0057] In order to guide the multimodal large model to better complete task planning and the execution of multimodal tasks, prompt templates suitable for different tasks can be designed. The role of these templates is to help the multimodal large model understand the user's goals more accurately and effectively combine multiple modal data for reasoning and decision-making. Through reasonable prompt design, the model can better exert its capabilities in complex task scenarios and enhance its performance in human-computer interaction. This not only improves the model's processing efficiency of multimodal information, but also ensures its accuracy and flexibility in actual tasks.

[0058] In the application scenario of laboratory assistant, the prompt template for task planning can be:

[0059]

[0060]

[0061]

[0062] The prompt template for the multi-modal box selection target task can be:

[0063]

[0064]

[0065] Step 3: Model training

[0066] In the embodiment of the present invention, step 3 aims to train the multimodal large model twice, and the two tasks are different, namely, the fine-grained task planning task and the multimodal box selection target task. The following are the specific model training steps, taking the laboratory assistant scenario as an example:

[0067] Step 3.1: Task planning multimodal large model training

[0068] In this sub-step, the present invention will train a large multimodal model that can understand user input and then plan a fine-grained task sequence that meets user needs. This model will achieve the purpose of accurate planning through supervised fine-tuning. For the laboratory assistant scenario, an open source large pre-trained multimodal model (such as Qwen-VL) is used. Then, the multimodal large model is supervised fine-tuned using a fine-grained task planning label dataset and the corresponding prompt template.

[0069] Step 3.2: Multimodal box selection target large model training

[0070] In this sub-step, the present invention will train a large multimodal model so that it can understand user semantics and image information at the same time, combine the two for reasoning, and finally select the target to meet user needs. Also use an open source large-scale pre-trained multimodal model (such as Qwen-VL), use multimodal task data and related subtask data and corresponding prompt templates, and perform supervised fine-tuning on the model to ensure that the target can be accurately selected. When there are multiple multimodal tasks, the data of multiple tasks can be mixed for joint training.

[0071] Step 4: Interactive Control

[0072] In the embodiment of the present invention, step 4 is a key stage of interactive control, such as Figure 2 As shown in the figure, fine-grained task planning is performed based on user input, and then the tasks are executed one by one according to the task sequence. At the same time, different tasks are sent to different modules for processing based on whether they require multimodal information and whether they belong to exclusive domain questions and answers, so as to achieve smooth user interaction.

[0073] Step 4.1: Mission planning

[0074] The interactive framework will first combine the user's input with the prompt template of the task planning and submit it to the multimodal large model for semantic understanding, select the required tasks according to the fine-grained task list, and then output the fine-grained task sequence in the logical execution order. Finally, it traverses and executes according to the fine-grained task sequence to ultimately achieve the user's goal.

[0075] Step 4.2: Task processing strategy

[0076] In this sub-step, for the application scenario of the laboratory assistant, the interactive framework will process according to the fine-grained task sequence generated in step 4.1. First, the system will determine whether each task is a multimodal task. If it is a multimodal task, the multimodal task module will be called, and the user input and the relevant environmental image information taken will be obtained, and passed to the multimodal large model for processing. The multimodal large model will output the position information of the input target, and then call the target tracking module to enable the laboratory assistant to track according to the target position. If the task does not belong to a multimodal task, the system will further distinguish whether the task is a question-and-answer task in a dedicated field. If it is a question-and-answer task in a dedicated field, the system will retrieve relevant documents through RAG technology and provide professional answers; if it is not a task in a dedicated field, it will be handed over to the corresponding single-modal processing module for answering and processing. This process design ensures the efficient processing of multimodal and single-modal tasks and the precise allocation of task types.

[0077] After the above four steps, intention understanding interaction based on a multimodal large model can be achieved.

[0078] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only.

[0079] It should be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A multimodal large model-based intention understanding interaction method, characterized in that: The following steps are involved: Identify the fine-grained tasks required for users to complete their goals in the current interaction scenario; the fine-grained tasks include target-finding tasks, small talk tasks, and exclusive knowledge question-and-answer tasks; Classifying the fine-grained tasks according to their types into fine-grained tasks that require multimodal information processing and fine-grained tasks that do not require multimodal information processing; The fine-grained task that needs to process multimodal information is defined as a multimodal task, and the multimodal task is analyzed to decompose a plurality of related subtasks, wherein the related subtasks include selecting a person and describing an image; Collecting training data of the fine-grained task, the training data including single-task annotated data, multi-task annotated data, general data and corresponding prompt templates; Collecting a data set of the multimodal task and a data set of the related subtasks, wherein the training data in the data set includes image-text pairs, general data, and corresponding prompt templates; The single-task annotated data, the multi-task annotated data and the general data are combined with corresponding prompt templates to train a multimodal large model; The data set of the multimodal task and the data set of related subtasks are mixed, and the multimodal large model is trained again. The multimodal large model is used to process the user's input and the image information of the corresponding environment, and then determine the user's intention to achieve the user's goal; if there are multiple multimodal tasks, the training data corresponding to each multimodal task is integrated into a unified data set, and then these tasks are jointly trained; Use the trained multimodal large model to process user input: perform task planning for user input through the multimodal large model, and call the corresponding modules in sequence according to the planned tasks; when multimodal information processing is required, use the multimodal large model to combine user input with images and audio to achieve the goal.

2. The intention understanding interaction method based on a multimodal large model according to claim 1 is characterized in that: The classification according to the type of the fine-grained task includes: classifying fine-grained tasks that require multiple modal information into multi-modal understanding interaction tasks, classifying fine-grained tasks that can be solved with only a single modal information into single-modal tasks, and classifying fine-grained tasks that require question and answer based on specific application scenarios into application-specific question and answer tasks. The application-specific question and answer tasks require the use of RAG technology to read relevant documents before answering.

3. The intention understanding interaction method based on a multimodal large model according to claim 1 is characterized in that: The collecting of training data for the fine-grained task includes: Collect or generate conversation data generated in the current application scenario, and use the conversation data as training data; Generate clear intent labels for the training data of task planning in the multimodal large model, that is, fine-grained task annotation data, to ensure that the multimodal large model can accurately generate and plan precise task sequences for users.

4. The intention understanding interaction method based on a multimodal large model according to claim 1 is characterized in that: The collecting of the data set of the multimodal task and the collecting of the data set of the related subtasks at the same time include: In the dataset for processing multimodal tasks, corresponding labels are generated based on user input and captured images. The format of each label is fixed and includes a description of the selected object and the coordinate positions of the upper left corner and the lower right corner of the box. In the training data for processing related subtasks, corresponding labels are generated based on user input and images. The label format varies with the task. Prompt templates are designed for task planning and multimodal tasks respectively to guide the multimodal large model to accurately plan logical task sequences, identify user intentions, and complete interaction with users.

5. The intention understanding interaction method based on a multimodal large model according to claim 1 is characterized in that: The training of the multimodal large model comprises: The multimodal large model is fine-tuned in a supervised manner using the training data and the prompt template, and the multimodal large model is trained to accurately plan task sequences according to user input, process multimodal information, accurately identify user intentions and complete interactions.

6. The intention understanding interaction method based on a multimodal large model according to claim 1 is characterized in that: The method of processing user input using the trained multimodal large model includes: Based on user input, a multimodal large model is used to plan tasks based on user intent and output a task sequence to ensure that the task sequence is logical and meets user needs; Understand the user's intention, select according to the existing fine-grained task list, sort the selected fine-grained tasks according to logical relationships, and output the final fine-grained task sequence; determine whether each fine-grained task requires multimodal information to achieve the goal based on the fine-grained task sequence, and assign fine-grained tasks that do not require multimodal information to the corresponding modules for processing.

Images