Multi-modal multi-task intelligent interaction method, device, electronic device and storage medium

Abstract

The present invention relates to a multimodal multitask intelligent interaction method, device, electronic device and storage medium. The method comprises: obtaining multimodal data, and preprocessing and extracting features of the multimodal data to construct a multimodal knowledge base. Receive input multimodal information, and identify the current user and usage scenario based on the multimodal information to obtain multiple tasks corresponding to the current user and the usage scenario. Use multiple tasks as input to the Transformer multitask model to form a corresponding multi-head model of the multiple tasks, and fuse the features of different tasks to establish a multimodal understanding framework to output recognition results corresponding to the multiple tasks. Select a corresponding interaction strategy based on the recognition result, generate corresponding interaction content according to the interaction strategy, and visualize the interaction content. The method introduces more interaction methods, is close to the habits and needs of users, and can complete interaction tasks more quickly and accurately.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction technology, and in particular to a multi-modal multi-task intelligent interaction method, device, electronic device and storage medium. Background Art

[0002] With the continuous development of science and technology, people's requirements for intelligent interaction are getting higher and higher. Traditional single-modal intelligent interaction can no longer meet people's needs. There are mainly the following problems:

[0003] (1) Limited interaction capabilities and accuracy issues. Due to technical limitations, single-modal intelligent interaction may have difficulty understanding the user's intentions and context, resulting in low accuracy in the interaction content. For example, communication between humans and machines is more easily affected by factors such as the environment and noise, making it difficult to identify the user's intentions.

[0004] (2) The interactive content is mostly basic and mechanical dialogue. Most intelligent interactive systems can only provide services within a relatively limited task domain, so they can only conduct simple question-and-answer dialogues for specific issues and tasks.

[0005] (3) Low intelligence. For example, most intelligent interactions can only accept text and voice input, which limits the way users interact with the system to a certain extent, making it difficult for the interactive system to learn users' personalized needs and preferences, and thus unable to provide personalized services for different users.

[0006] Therefore, traditional intelligent interaction methods are limited to interacting with users through text or voice, which makes human-computer interaction efficiency low in different interaction scenarios, system interaction is easily affected by the environment, interaction accuracy is low, and it is difficult to meet the interaction needs in multimodal interaction scenarios. Summary of the invention

[0007] Based on this, it is necessary to provide a multimodal multi-task intelligent interaction method, device, electronic device and storage medium that can improve human-computer interaction efficiency and interaction accuracy in a variety of interaction scenarios in response to the above technical problems.

[0008] The present invention provides a multi-modal multi-task intelligent interaction method, the method comprising:

[0009] Acquiring multimodal data, and performing preprocessing and feature extraction on the multimodal data to construct a multimodal knowledge base;

[0010] Receiving input multimodal information, and identifying a current user and a usage scenario based on the multimodal information to obtain a plurality of tasks corresponding to the current user and the usage scenario;

[0011] The multiple tasks are used as inputs of the Transformer multi-task model to form the multiple tasks into corresponding multi-head models, and the features of different tasks are fused to establish a multimodal understanding framework to output recognition results corresponding to the multiple tasks;

[0012] Selecting a corresponding interaction strategy based on the recognition result, generating corresponding interaction content according to the interaction strategy and visually displaying the interaction content;

[0013] Among them, the multimodal data includes text data, audio data, video data and image data, the preprocessing includes data cleaning and clustering algorithm processing, the multimodal information includes text information, audio information, video information and image information actively or passively input by the user, the multiple tasks include interaction intention recognition tasks, emotion recognition tasks and entity recognition tasks, the interaction strategies include task-based interaction, knowledge-based interaction and open domain chat interaction, and the interaction content includes text, pictures and voice.

[0014] In one embodiment, the acquiring of multimodal data, and preprocessing and feature extraction of the multimodal data to construct a multimodal knowledge base includes:

[0015] Performing corpus cleaning on the text data, deleting useless data and duplicate data in the text data through clustering annotation, and removing garbled characters and redundant symbols in the text data to obtain preprocessed text data; and

[0016] Illegal images in the image data are removed through clustering annotation, and the image data are labeled and annotated, and the image data are adjusted to a uniform size and resolution to obtain preprocessed image data.

[0017] In one embodiment, the step of acquiring multimodal data, and preprocessing and extracting features from the multimodal data to construct a multimodal knowledge base further includes:

[0018] Performing video segmentation on the video data to segment the video data into a plurality of video segments, and extracting a plurality of video frames from the video segments to perform feature extraction on the video frames; and

[0019] The audio data is denoised, and the audio data is segmented into multiple audio segments according to a set rule, so as to extract features from the multiple audio segments.

[0020] In one embodiment, the step of acquiring multimodal data, and preprocessing and extracting features from the multimodal data to construct a multimodal knowledge base further includes:

[0021] Performing feature extraction on the preprocessed multimodal data by using a convolutional neural network to extract attribute features and feature vectors in the multimodal data, wherein the attribute features and feature vectors are used to perform similarity analysis and data classification on the multimodal data;

[0022] The multimodal data is stored in the multimodal knowledge base including a relational database and a graph database according to the similarity analysis results and data classification results of the attribute features and feature vectors, and indexes between different modal data are established.

[0023] In one embodiment, the receiving of input multimodal information and identifying a current user and a usage scenario based on the multimodal information to obtain a plurality of tasks corresponding to the current user and the usage scenario includes:

[0024] Based on the multimodal information input by different users and different scenarios, a user portrait knowledge base and a scene portrait knowledge base are established as candidate sets, and a portrait matching unit is used to generate a corresponding input representation according to the multimodal information input by the current user;

[0025] The matching score corresponding to the multimodal information input by the current user is calculated through the set retrieval matching strategy, and the user portrait matching vector and scene portrait matching vector with the highest score in the candidate set are fed back.

[0026] In one embodiment, the multiple tasks are used as inputs of a Transformer multi-task model to form the multiple tasks into corresponding multi-head models, and information fusion is performed on features of different tasks to establish a multimodal understanding framework to output recognition results corresponding to the multiple tasks, including:

[0027] Introducing prompt dialog learning in the decoding layer of the Transformer multi-task model to guide the Transformer multi-task model to generate label results corresponding to different tasks;

[0028] The multimodal understanding framework is called to perform recognition processing on different tasks to generate recognition results corresponding to the different tasks, wherein the recognition results have the label results corresponding to each task, and the label results are used to determine the interaction strategies corresponding to the different tasks.

[0029] In one embodiment, selecting a corresponding interaction strategy based on the recognition result, generating corresponding interaction content according to the interaction strategy, and visually displaying the interaction content includes:

[0030] Acquire a user intention corresponding to the current user based on the recognition result, and feed back corresponding first interaction content in response to the user intention;

[0031] It is determined by monitoring whether the first interactive content meets the user needs of the current user, and when the first interactive content does not meet the user needs, the second interactive content is fed back to determine the user intention of the current user.

[0032] The present invention also provides a multi-modal multi-task intelligent interaction device, the device comprising:

[0033] A knowledge base construction module, used for acquiring multimodal data, and performing preprocessing and feature extraction on the multimodal data to construct a multimodal knowledge base;

[0034] An information identification module, used to receive input multimodal information, and identify a current user and a usage scenario based on the multimodal information to obtain a plurality of tasks corresponding to the current user and the usage scenario;

[0035] A task identification module is used to use the multiple tasks as inputs of the Transformer multi-task model to form the multiple tasks into corresponding multi-head models, and to fuse information of features of different tasks to establish a multimodal understanding framework to output identification results corresponding to the multiple tasks;

[0036] A content interaction module, configured to select a corresponding interaction strategy based on the recognition result, generate corresponding interaction content according to the interaction strategy, and visually display the interaction content;

[0037] Among them, the multimodal data includes text data, audio data, video data and image data, the preprocessing includes data cleaning and clustering algorithm processing, the multimodal information includes text information, audio information, video information and image information actively or passively input by the user, the multiple tasks include interaction intention recognition tasks, emotion recognition tasks and entity recognition tasks, the interaction strategies include task-based interaction, knowledge-based interaction and open domain chat interaction, and the interaction content includes text, pictures and voice.

[0038] The present invention also provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the multi-modal multi-task intelligent interaction method as described above is implemented.

[0039] The present invention also provides a computer storage medium storing a computer program, wherein when the computer program is executed by a processor, the multi-modal multi-task intelligent interaction method as described above is implemented.

[0040] The present invention also provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, the multi-modal multi-task intelligent interaction method as described above is implemented.

[0041] The above-mentioned multimodal multi-task intelligent interaction method, device, electronic device and storage medium construct a multimodal knowledge base by acquiring multimodal data, preprocessing and extracting features of the multimodal data. Subsequently, the input multimodal information is received, and the current user and the usage scenario are identified based on the multimodal information to obtain multiple tasks corresponding to the current user and the usage scenario. Multiple tasks are used as the input of the Transformer multi-task model, and the multiple tasks are composed of corresponding multi-head models, and the features of different tasks are information-fused to establish a multimodal understanding framework to output the recognition results corresponding to multiple tasks. Finally, the corresponding interaction strategy is selected based on the recognition result, and the corresponding interaction content is generated according to the interaction strategy and the interaction content is visualized. Multimodal intelligent interaction can introduce more interaction methods, such as images, gestures, touch, facial expressions, eyes, etc., which are closer to the habits and needs of users, while allowing users to complete interaction tasks more quickly and accurately, improving interaction efficiency and user satisfaction. At the same time, multimodal intelligent interaction can also improve the reliability and robustness of interaction. In some noisy environments or inconvenient voice input, users can interact through other methods, avoiding the limitations of existing voice interaction methods and adapting to different interaction scenarios and environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 One of the flow charts of the multi-modal multi-task intelligent interaction method provided by the present invention;

[0044] Figure 2 A schematic diagram of a multimodal knowledge base construction process of a multimodal multitask intelligent interaction method in a specific embodiment of the present invention;

[0045] Figure 3 A schematic diagram of a multimodal information recognition process of a multimodal multitask intelligent interaction method in a specific embodiment of the present invention;

[0046] Figure 4 A schematic diagram of a multi-task learning process of a multi-modal multi-task intelligent interaction method in a specific embodiment of the present invention;

[0047] Figure 5 The second flowchart of the multi-modal multi-task intelligent interaction method provided by the present invention;

[0048] Figure 6 The third flowchart of the multi-modal multi-task intelligent interaction method provided by the present invention;

[0049] Figure 7 The fourth flowchart of the multi-modal multi-task intelligent interaction method provided by the present invention;

[0050] Figure 8 The fifth flowchart of the multi-modal multi-task intelligent interaction method provided by the present invention;

[0051] Fig. 9 The sixth flowchart of the multi-modal multi-task intelligent interaction method provided by the present invention;

[0052] Fig.10 The seventh flowchart of the multi-modal multi-task intelligent interaction method provided by the present invention;

[0053] Fig.11 A schematic diagram of the structure of a multi-modal multi-task intelligent interaction device provided by the present invention;

[0054] Fig.12 This is a diagram of the internal structure of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] Combine the following Figure 1-Figure 12 The present invention describes a multi-modal multi-task intelligent interaction method, device, electronic device and storage medium.

[0057] like Figure 1 As shown, in one embodiment, a multi-modal multi-task intelligent interaction method includes the following steps:

[0058] Step S110, acquiring multimodal data, and performing preprocessing and feature extraction on the multimodal data to construct a multimodal knowledge base.

[0059] Specifically, the server obtains multimodal data such as text data, audio data, video data, and image data, and performs data cleaning and clustering algorithm preprocessing and feature extraction on the multimodal data to complete the construction of the multimodal knowledge base.

[0060] Step S120, receiving input multimodal information, and identifying the current user and the usage scenario based on the multimodal information to obtain multiple tasks corresponding to the current user and the usage scenario.

[0061] Specifically, the server receives multimodal information such as text information, audio information, video information, and image information actively or passively input by the user, and identifies the current user and usage scenario based on the received multimodal information to obtain multiple tasks corresponding to the current user and usage scenario as input of the Transformer multi-task model.

[0062] Step S130, taking multiple tasks as input of the Transformer multi-task model, so as to combine the multiple tasks into a corresponding multi-head model, and fuse the features of different tasks to establish a multimodal understanding framework, so as to output the recognition results corresponding to the multiple tasks.

[0063] Specifically, the server takes multiple tasks such as interaction intention recognition tasks, emotion recognition tasks, and entity recognition tasks as the input of the Transformer multi-task model, combines the multiple tasks into a multi-head model, and fuses the features of different tasks to establish a multimodal understanding framework to output the recognition results corresponding to multiple tasks.

[0064] Step S140, selecting a corresponding interaction strategy based on the recognition result, generating corresponding interaction content according to the interaction strategy and visually displaying the interaction content.

[0065] Specifically, the server selects a corresponding interaction strategy based on the recognition result obtained in step S130, generates corresponding interaction content according to the selected interaction strategy, and visually displays the interaction content to the current user.

[0066] Among them, the interaction strategies include task-based interaction, knowledge-based interaction and open-domain chat interaction, and the interaction content includes text, pictures and voice.

[0067] The above multimodal multitask intelligent interaction method constructs a multimodal knowledge base by acquiring multimodal data, preprocessing and extracting features of the multimodal data. Subsequently, the input multimodal information is received, and the current user and the usage scenario are identified based on the multimodal information to obtain multiple tasks corresponding to the current user and the usage scenario. Multiple tasks are used as the input of the Transformer multitask model, and the multiple tasks are composed of corresponding multi-head models. The features of different tasks are fused to establish a multimodal understanding framework to output the recognition results corresponding to multiple tasks. Finally, the corresponding interaction strategy is selected based on the recognition results, and the corresponding interaction content is generated according to the interaction strategy and the interaction content is visualized. Multimodal intelligent interaction can introduce more interaction methods, such as images, gestures, touch, facial expressions, eyes, etc., which are closer to the habits and needs of users, while allowing users to complete interaction tasks more quickly and accurately, improving interaction efficiency and user satisfaction. At the same time, multimodal intelligent interaction can also improve the reliability and robustness of interaction. In some noisy environments or inconvenient voice input, users can interact through other methods, avoiding the limitations of existing voice interaction methods and adapting to different interaction scenarios and environments.

[0068] See also Figure 2 As shown, in a specific embodiment, the multimodal multitask intelligent interaction method provided by the present invention constructs a rich and heterogeneous knowledge base, and through multimodal learning, various types of modal data are fused from the vector space to realize cross-modal information fusion analysis and associated retrieval, realize automatic classification of knowledge, and actively discover new knowledge through data association relationships. In the process of knowledge base construction, it is necessary to collect data first. The multimodal knowledge base needs to contain multiple types of data such as images, videos, and texts. Therefore, it is necessary to collect data from various sources, such as network pictures, videos, and text materials. Subsequently, the collected multimodal data is preprocessed. To build a multimodal knowledge base, different types of data need to be preprocessed differently. The clustering algorithm is used to automatically cluster related materials and automatically maintain them to ensure the accuracy, consistency, and comparability of the data, thereby providing a better basis for subsequent analysis and application. The specific operation is to clean and de-duplicate the collected data of different modes to ensure the accuracy and reliability of the data.

[0069] In this embodiment, text data cleaning includes corpus cleaning (clustering and annotation to delete irrelevant data and duplicate data, and remove garbled and redundant symbol data), word segmentation (jieba (an open source Chinese word segmentation tool), LTP (Chinese tool set), etc.), and part-of-speech tagging. Image data cleaning mainly requires image inspection, clustering and annotation to remove illegal images (watermarks, noise, blur, etc.), annotating text labels for images, mapping image text expressions, and adjusting images to uniform size and resolution for better comparison and analysis. Video data cleaning mainly requires video segmentation, that is, segmenting the video and cutting it into a series of short clips for subsequent processing and analysis. It is also necessary to perform frame extraction, that is, extract a certain number of frames from the video for feature extraction and analysis. Audio data mainly requires denoising, that is, denoising the audio data to reduce the interference of noise on subsequent processing. It is also necessary to perform sound segmentation, that is, segment the audio data according to certain rules for further processing and analysis.

[0070] After data preprocessing, different features need to be extracted for different types of data. For example, images can be extracted through algorithms such as convolutional neural networks to extract attribute features such as color, texture, and shape, as well as feature vectors of images, for similarity analysis and classification. Text can be extracted through attribute features such as vocabulary and grammar, as well as semantic feature vectors of text, for similarity analysis and classification. Video data can be extracted through algorithms based on deep learning to extract feature vectors of video frames for similarity analysis and classification. Audio data can be extracted through algorithms based on deep learning to extract feature vectors of audio for similarity analysis and classification. After that, the processed data is stored in the knowledge base, and a rich knowledge base system is stored in two ways: relational database or graph database. Relational databases include FAQ question and answer libraries, table question and answer libraries, etc., and graph databases are multimodal knowledge graphs constructed in business fields.

[0071] In this embodiment, an index is established for the data in the knowledge base, wherein a multimodal relationship association is established for different modal knowledge information, and cross-modal association training can be performed based on the CLIP model (multimodal large model), so that different types of data are connected and associated, and multimodal relationships are established. Cross-modal retrieval is implemented using CLIP, and query operations can be conveniently performed. In the subsequent data operation process, a user interface is designed, and convenient tools are provided. The existing knowledge base is optimized through user interaction records, manual service records, etc., and users are supported to add, delete, modify, and query in real time. It has convenient knowledge entry, editing, and review functions, and supports rich media methods such as text, pictures, hyperlinks, videos, emoticons, interactive pages, and interface calls. Knowledge points can be grouped and managed according to business, and the data in the knowledge base is continuously updated, effectively ensuring the practicality and reliability of the knowledge base.

[0072] Combination Figure 3 As shown in the figure, in the process of inputting multimodal information, the input includes two parts: passive input and active input. The passive input data includes multiple modal data such as text, voice, picture and video input by the user, and the active input includes the user face, user portrait and scene portrait information after scene recognition. According to the needs of different groups and different scenarios, the industry user portrait knowledge base and scene portrait knowledge base are established as candidate sets. Then, the user attribute information such as gender, age, face and the environmental attribute information of the scene are used as the query through the portrait matching module to generate personalized input representation, so as to calculate the matching score through the retrieval matching strategy, and return the highest score that best matches the user, that is, the user portrait matching vector (u) and the scene matching vector (g). Finally, the input is entered into the subsequent interaction strategy identification module to capture the personalized interaction mode.

[0073] See also Figure 4 As shown in the figure, entity represents entity, intent represents intention, sentiment represents emotion library, Decoder represents decoder, Encoder represents encoder, and concatenate is the concatenation function in deep learning. In the multi-task learning process, the following steps are specifically included:

[0074] 1) Collect multimodal data: Collect multimodal data such as user voice, video, gestures, etc. through cameras, microphones, sensors and other devices. The multimodal data collected in this way is mainly passive input data. In addition, by matching user portraits and scene portrait databases, user portrait data and scene portrait data can be output as active recognition input data.

[0075] 2) Preprocess the multimodal input data collected in 1), such as denoising audio data, converting speech into text, segmenting text data, extracting frames from videos, etc.

[0076] 3) Multimodal feature extraction: Extract features from preprocessed multimodal data for subsequent modeling. For example, for speech data, extract MFCC acoustic features (spectrum, Mel cepstral coefficients), for text data, use BERT (autoencoder language model) to extract semantic features, and for image data, use MAE (mean absolute error) to extract convolutional neural network features.

[0077] 4) Multi-task learning: Learn multiple tasks (such as interaction intent recognition, sentiment judgment, entity recognition, etc.) as a whole so that multiple tasks can be trained simultaneously and features can be shared. Based on the Transformer multi-task model, multiple tasks are combined into a multi-head model, and the Attention Mask is designed to share some parameters to achieve inductive transfer and improve the generalization performance of the model. Multi-task learning introduces Prompt Learning in the decoding layer to guide the model to generate corresponding task label results. The multi-task learning process can fuse information from features of different tasks, so that cross-modal data expressing similar semantics can be mapped into the same space, thereby obtaining a more comprehensive representation of user input information features, improving the recognition accuracy of the model, and realizing a multi-modal understanding framework with both representation and generation capabilities.

[0078] 5) Output recognition results: Output the label results corresponding to user intention recognition, emotion recognition, and entity recognition. The label results are used for subsequent interactive behavior decision-making.

[0079] Afterwards, a comprehensive judgment of the interactive decision is made based on the label results corresponding to the output of user intention recognition, emotion recognition, and entity recognition. The specific steps are as follows:

[0080] 1) Respond to user intent: Provide corresponding answers, suggestions or actions based on user intent.

[0081] 2) Check the feedback and whether it meets the user's needs. If it does, trigger the user's exit condition. Otherwise, enter 3).

[0082] 3) Ask for further information: If you do not fully understand the user's intent, you need to proactively ask the user for further information to confirm the intent.

[0083] 4) Optional: Provide relevant information feedback. Based on user intent, proactively provide useful information, such as related materials, links, or services.

[0084] 5) Exit condition judgment. Based on user feedback, the system determines whether the user's needs are met or the exit condition is triggered. If not, the process from 1) to 4) is repeated until the exit is completed.

[0085] In this embodiment, through personalized feedback output, different forms of interaction modes can be provided according to the user's usage scenario and emotions. For example, in the office, more suitable texts and cards are selected for interaction. In the leisure environment, highly perceived voice and video multimodal interactions are recommended. In terms of interactive content, personalized expressions are generated according to the current context, user portrait, and emotions of the interaction. For example, if the user is depressed or angry, the expression speed and content style are adjusted to better meet the needs of the user. The recognizer sends prompt information or recommended information according to the usage scenario, time, etc., and intelligently decides the triggering time. For example, combined with the weather, adjustments are made when asking about travel methods, and reminders are given for safety, guiding users to perform specific operations, such as prompting users to enter specified information, reminding users to click specific buttons or links, etc. In addition, higher interactivity and feedback mechanisms, such as incentives and feedback, can also be provided to enhance user experience and participation. During the interaction process, the user's intentions and operation preferences are continuously recorded so that the user's intentions and needs can be more accurately identified in the future, and more intelligent services can be provided.

[0086] like Figure 5 As shown, in one embodiment, the multimodal multitask intelligent interaction method provided by the present invention obtains multimodal data, and preprocesses and extracts features of the multimodal data to construct a multimodal knowledge base, specifically comprising the following steps:

[0087] Step S111 , corpus cleaning is performed on the text data, useless data and duplicate data in the text data are deleted through clustering annotation, and garbled characters and redundant symbols in the text data are removed to obtain preprocessed text data.

[0088] Step S112, removing illegal images from the image data through clustering annotation, labeling the image data, adjusting the image data to a uniform size and resolution, and obtaining pre-processed image data.

[0089] like Figure 6 As shown, in one embodiment, the multimodal multitask intelligent interaction method provided by the present invention obtains multimodal data, and preprocesses and extracts features of the multimodal data to construct a multimodal knowledge base, and specifically includes the following steps:

[0090] Step S113, performing video segmentation on the video data, dividing the video data into a plurality of video segments, and extracting a plurality of video frames from the video segments to perform feature extraction on the video frames.

[0091] Step S114, denoising the audio data, and dividing the audio data into multiple audio segments according to a set rule, so as to extract features from the multiple audio segments.

[0092] like Figure 7As shown, in one embodiment, the multimodal multitask intelligent interaction method provided by the present invention obtains multimodal data, and preprocesses and extracts features of the multimodal data to construct a multimodal knowledge base, and specifically includes the following steps:

[0093] Step S115, performing feature extraction on the preprocessed multimodal data through a convolutional neural network to extract attribute features and feature vectors in the multimodal data, and the attribute features and feature vectors are used to perform similarity analysis and data classification on the multimodal data.

[0094] Step S116, storing the multimodal data in a multimodal knowledge base including a relational database and a graph database according to the similarity analysis results of the attribute features and feature vectors and the data classification results, and establishing an index between different modal data.

[0095] like Figure 8 As shown, in one embodiment, the multimodal multitasking intelligent interaction method provided by the present invention receives input multimodal information, and identifies the current user and the usage scenario based on the multimodal information to obtain multiple tasks corresponding to the current user and the usage scenario, specifically including the following steps:

[0096] Step S122, based on the multimodal information input by different users and different scenarios, a user portrait knowledge base and a scene portrait knowledge base are established as candidate sets, and a portrait matching unit is used to generate a corresponding input representation according to the multimodal information input by the current user.

[0097] Step S124, calculate the matching score corresponding to the multimodal information input by the current user through the set retrieval matching strategy, and feed back the user portrait matching vector and scene portrait matching vector with the highest score in the candidate set.

[0098] like Fig. 9 As shown, in one embodiment, the multimodal multitask intelligent interaction method provided by the present invention takes multiple tasks as input of the Transformer multitask model to form the multiple tasks into corresponding multi-head models, and fuses the features of different tasks to establish a multimodal understanding framework to output recognition results corresponding to the multiple tasks, specifically including the following steps:

[0099] Step S132, introducing prompt dialog learning in the decoding layer of the Transformer multi-task model to guide the Transformer multi-task model to generate label results corresponding to different tasks.

[0100] Step S134, calling the multimodal understanding framework to perform recognition processing on different tasks to generate recognition results corresponding to different tasks, wherein the recognition results have label results corresponding to each task, and the label results are used to determine the interaction strategies corresponding to different tasks.

[0101] like Fig.10 As shown, in one embodiment, the multi-modal multi-task intelligent interaction method provided by the present invention selects a corresponding interaction strategy based on the recognition result, generates corresponding interaction content according to the interaction strategy, and visualizes the interaction content, specifically including the following steps:

[0102] Step S142: acquiring the user intention corresponding to the current user based on the recognition result, and feeding back the corresponding first interaction content in response to the user intention.

[0103] Step S144, determining by monitoring whether the first interactive content meets the user needs of the current user, and when the first interactive content does not meet the user needs, feeding back the second interactive content to determine the user intention of the current user.

[0104] The multimodal multitask intelligent interaction device provided by the present invention is described below. The multimodal multitask intelligent interaction device described below and the multimodal multitask intelligent interaction method described above can be referred to each other.

[0105] like Fig.11 As shown, in one embodiment, a multi-modal multi-task intelligent interaction device includes a knowledge base construction module 1110 , an information identification module 1120 , a task identification module 1130 and a content interaction module 1140 .

[0106] The knowledge base construction module 1110 is used to obtain multimodal data, and perform preprocessing and feature extraction on the multimodal data to construct a multimodal knowledge base.

[0107] The information identification module 1120 is used to receive input multimodal information and identify the current user and the usage scenario based on the multimodal information to obtain multiple tasks corresponding to the current user and the usage scenario.

[0108] The task identification module 1130 is used to take multiple tasks as the input of the Transformer multi-task model to form a corresponding multi-head model for the multiple tasks, and to fuse the features of different tasks to establish a multimodal understanding framework to output the recognition results corresponding to the multiple tasks.

[0109] The content interaction module 1140 is used to select a corresponding interaction strategy based on the recognition result, generate corresponding interaction content according to the interaction strategy, and visualize the interaction content.

[0110] Among them, multimodal data includes text data, audio data, video data and image data, preprocessing includes data cleaning and clustering algorithm processing, multimodal information includes text information, audio information, video information and image information actively or passively input by users, multiple tasks include interaction intention recognition tasks, emotion recognition tasks and entity recognition tasks, interaction strategies include task-based interaction, knowledge-based interaction and open domain chat interaction, and the interaction content includes text, pictures and voice.

[0111] In this embodiment, in the multi-modal multi-task intelligent interaction device provided by the present invention, the knowledge base construction module 1110 is specifically used for:

[0112] The text data is cleaned, useless and duplicate data in the text data is deleted through clustering annotation, and garbled characters and redundant symbols in the text data are removed to obtain the preprocessed text data.

[0113] Illegal images in the image data are removed through clustering annotation, and the image data is labeled and annotated, and the image data is adjusted to a uniform size and resolution to obtain preprocessed image data.

[0114] In this embodiment, in the multi-modal multi-task intelligent interaction device provided by the present invention, the knowledge base construction module 1110 is further used for:

[0115] The video data is segmented into multiple video segments, and multiple video frames are extracted from the video segments to extract features of the video frames.

[0116] The audio data is denoised and segmented into multiple audio segments according to set rules to extract features from the multiple audio segments.

[0117] In this embodiment, in the multi-modal multi-task intelligent interaction device provided by the present invention, the knowledge base construction module 1110 is further used for:

[0118] The preprocessed multimodal data is subjected to feature extraction through a convolutional neural network to extract attribute features and feature vectors in the multimodal data. The attribute features and feature vectors are used to perform similarity analysis and data classification on the multimodal data.

[0119] The multimodal data are stored in a multimodal knowledge base including a relational database and a graph database according to the similarity analysis results of attribute features and feature vectors and the data classification results, and indexes between different modal data are established.

[0120] In this embodiment, in the multi-modal multi-task intelligent interaction device provided by the present invention, the information identification module 1120 is specifically used for:

[0121] Based on the multimodal information input from different users and different scenarios, a user portrait knowledge base and a scene portrait knowledge base are established as candidate sets, and a portrait matching unit is used to generate corresponding input representations according to the multimodal information input by the current user.

[0122] The matching score corresponding to the multimodal information input by the current user is calculated through the set retrieval matching strategy, and the user portrait matching vector and scene portrait matching vector with the highest score in the candidate set are fed back.

[0123] In this embodiment, in the multi-modal multi-task intelligent interaction device provided by the present invention, the task identification module 1130 is specifically used for:

[0124] Prompt dialog learning is introduced in the decoding layer of the Transformer multi-task model to guide the Transformer multi-task model to generate label results corresponding to different tasks.

[0125] The multimodal understanding framework is called to perform recognition processing on different tasks to generate recognition results corresponding to different tasks. The recognition results contain label results corresponding to each task, and the label results are used to determine the interaction strategies corresponding to different tasks.

[0126] In this embodiment, the multi-modal multi-task intelligent interaction device provided by the present invention, the content interaction module 1140 is specifically used for:

[0127] The user intention corresponding to the current user is obtained based on the recognition result, and the corresponding first interaction content is fed back in response to the user intention.

[0128] It is determined by monitoring whether the first interactive content meets the user needs of the current user, and when the first interactive content does not meet the user needs, the second interactive content is fed back to determine the user intention of the current user.

[0129] Fig.12 An example of a physical structure diagram of an electronic device is shown. The electronic device may be a smart terminal, and its internal structure diagram may be as follows: Fig.12 As shown. The electronic device includes a processor, a memory and a network interface connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a multi-modal multi-task intelligent interaction method is implemented, and the method includes:

[0130] Acquire multimodal data, preprocess and extract features from the multimodal data to build a multimodal knowledge base;

[0131] Receive input multimodal information, and identify the current user and the usage scenario based on the multimodal information to obtain multiple tasks corresponding to the current user and the usage scenario;

[0132] Take multiple tasks as input to the Transformer multi-task model to combine multiple tasks into corresponding multi-head models, fuse the features of different tasks, and establish a multimodal understanding framework to output recognition results corresponding to multiple tasks;

[0133] Select the corresponding interaction strategy based on the recognition result, generate the corresponding interaction content according to the interaction strategy and visualize the interaction content;

[0134] Among them, multimodal data includes text data, audio data, video data and image data, preprocessing includes data cleaning and clustering algorithm processing, multimodal information includes text information, audio information, video information and image information actively or passively input by users, multiple tasks include interaction intention recognition tasks, emotion recognition tasks and entity recognition tasks, interaction strategies include task-based interaction, knowledge-based interaction and open domain chat interaction, and the interaction content includes text, pictures and voice.

[0135] Those skilled in the art will understand that Fig.12 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present invention, and does not constitute a limitation on the electronic device to which the scheme of the present invention is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0136] On the other hand, the present invention further provides a computer storage medium storing a computer program, which implements a multi-modal multi-task intelligent interaction method when executed by a processor, the method comprising:

[0137] Acquire multimodal data, preprocess and extract features from the multimodal data to build a multimodal knowledge base;

[0138] Receive input multimodal information, and identify the current user and the usage scenario based on the multimodal information to obtain multiple tasks corresponding to the current user and the usage scenario;

[0139] Take multiple tasks as input to the Transformer multi-task model to combine multiple tasks into corresponding multi-head models, fuse the features of different tasks, and establish a multimodal understanding framework to output recognition results corresponding to multiple tasks;

[0140] Select the corresponding interaction strategy based on the recognition result, generate the corresponding interaction content according to the interaction strategy and visualize the interaction content;

[0141] Among them, multimodal data includes text data, audio data, video data and image data, preprocessing includes data cleaning and clustering algorithm processing, multimodal information includes text information, audio information, video information and image information actively or passively input by users, multiple tasks include interaction intention recognition tasks, emotion recognition tasks and entity recognition tasks, interaction strategies include task-based interaction, knowledge-based interaction and open domain chat interaction, and the interaction content includes text, pictures and voice.

[0142] In another aspect, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and when the processor executes the computer instructions, a multi-modal multi-task intelligent interaction method is implemented, the method comprising:

[0143] Acquire multimodal data, preprocess and extract features from the multimodal data to build a multimodal knowledge base;

[0144] Receive input multimodal information, and identify the current user and the usage scenario based on the multimodal information to obtain multiple tasks corresponding to the current user and the usage scenario;

[0145] Take multiple tasks as input to the Transformer multi-task model to combine multiple tasks into corresponding multi-head models, fuse the features of different tasks, and establish a multimodal understanding framework to output recognition results corresponding to multiple tasks;

[0146] Select the corresponding interaction strategy based on the recognition result, generate the corresponding interaction content according to the interaction strategy and visualize the interaction content;

[0147] Among them, multimodal data includes text data, audio data, video data and image data, preprocessing includes data cleaning and clustering algorithm processing, multimodal information includes text information, audio information, video information and image information actively or passively input by users, multiple tasks include interaction intention recognition tasks, emotion recognition tasks and entity recognition tasks, interaction strategies include task-based interaction, knowledge-based interaction and open domain chat interaction, and the interaction content includes text, pictures and voice.

[0148] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory.

[0149] By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0150] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0151] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A multi-modal multi-task intelligent interaction method, characterized in that: The method comprises: Acquiring multimodal data, and performing preprocessing and feature extraction on the multimodal data to construct a multimodal knowledge base; Receiving input multimodal information, and identifying a current user and a usage scenario based on the multimodal information to obtain a plurality of tasks corresponding to the current user and the usage scenario; The multiple tasks are used as inputs of the Transformer multi-task model to form the multiple tasks into corresponding multi-head models, and the features of different tasks are fused to establish a multimodal understanding framework to output recognition results corresponding to the multiple tasks; Selecting a corresponding interaction strategy based on the recognition result, generating corresponding interaction content according to the interaction strategy and visually displaying the interaction content; The multimodal data includes text data, audio data, video data and image data; the preprocessing includes data cleaning and clustering algorithm processing; the multimodal information includes text information, audio information, video information and image information actively or passively input by the user; the multiple tasks include interaction intention recognition tasks, emotion recognition tasks and entity recognition tasks; the interaction strategies include task-based interaction, knowledge-based interaction and open domain chat interaction; the interaction content includes text, pictures and voice; The step of acquiring multimodal data and performing preprocessing and feature extraction on the multimodal data to construct a multimodal knowledge base includes: Performing corpus cleaning on the text data, deleting useless data and duplicate data in the text data through clustering annotation, and removing garbled characters and redundant symbols in the text data to obtain preprocessed text data; and Removing illegal images from the image data through clustering annotation, labeling the image data, adjusting the image data to a uniform size and resolution, and obtaining preprocessed image data; The step of acquiring multimodal data, and preprocessing and extracting features of the multimodal data to construct a multimodal knowledge base further includes: Performing video segmentation on the video data to segment the video data into a plurality of video segments, and extracting a plurality of video frames from the video segments to perform feature extraction on the video frames; and Performing denoising on the audio data, and dividing the audio data into multiple audio segments according to a set rule, so as to extract features from the multiple audio segments; The step of acquiring multimodal data, and preprocessing and extracting features of the multimodal data to construct a multimodal knowledge base further includes: Performing feature extraction on the preprocessed multimodal data by using a convolutional neural network to extract attribute features and feature vectors in the multimodal data, wherein the attribute features and feature vectors are used to perform similarity analysis and data classification on the multimodal data; The multimodal data is stored in the multimodal knowledge base including a relational database and a graph database according to the similarity analysis results and data classification results of the attribute features and feature vectors, and indexes between different modal data are established; The receiving of input multimodal information and identifying a current user and a usage scenario based on the multimodal information to obtain a plurality of tasks corresponding to the current user and the usage scenario includes: Based on the multimodal information input by different users and different scenarios, a user portrait knowledge base and a scene portrait knowledge base are established as candidate sets, and a portrait matching unit is used to generate a corresponding input representation according to the multimodal information input by the current user; Calculate the matching score corresponding to the multimodal information input by the current user through the set retrieval matching strategy, and feed back the user portrait matching vector and scene portrait matching vector with the highest score in the candidate set; The method of using the multiple tasks as the input of the Transformer multi-task model to form the multiple tasks into a corresponding multi-head model, and fusing the features of different tasks to establish a multimodal understanding framework to output the recognition results corresponding to the multiple tasks includes: Introducing prompt dialog learning in the decoding layer of the Transformer multi-task model to guide the Transformer multi-task model to generate label results corresponding to different tasks; The multimodal understanding framework is called to perform recognition processing on different tasks to generate recognition results corresponding to the different tasks, wherein the recognition results have the label results corresponding to each task, and the label results are used to determine the interaction strategies corresponding to the different tasks.

2. The multi-modal multi-task intelligent interaction method according to claim 1, characterized in that: The selecting a corresponding interaction strategy based on the recognition result, generating corresponding interaction content according to the interaction strategy and visually displaying the interaction content includes: Acquire a user intention corresponding to the current user based on the recognition result, and feed back corresponding first interaction content in response to the user intention; It is determined by monitoring whether the first interactive content meets the user needs of the current user, and when the first interactive content does not meet the user needs, the second interactive content is fed back to determine the user intention of the current user.

3. A multi-modal multi-task intelligent interaction device, characterized in that: For implementing the method according to claim 1, the device comprises: A knowledge base construction module, used for acquiring multimodal data, and performing preprocessing and feature extraction on the multimodal data to construct a multimodal knowledge base; An information identification module, used to receive input multimodal information, and identify a current user and a usage scenario based on the multimodal information to obtain a plurality of tasks corresponding to the current user and the usage scenario; A task identification module is used to use the multiple tasks as inputs of the Transformer multi-task model to form the multiple tasks into corresponding multi-head models, and to fuse information of features of different tasks to establish a multimodal understanding framework to output identification results corresponding to the multiple tasks; A content interaction module, configured to select a corresponding interaction strategy based on the recognition result, generate corresponding interaction content according to the interaction strategy, and visually display the interaction content; Among them, the multimodal data includes text data, audio data, video data and image data, the preprocessing includes data cleaning and clustering algorithm processing, the multimodal information includes text information, audio information, video information and image information actively or passively input by the user, the multiple tasks include interaction intention recognition tasks, emotion recognition tasks and entity recognition tasks, the interaction strategies include task-based interaction, knowledge-based interaction and open domain chat interaction, and the interaction content includes text, pictures and voice.

4. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 2 are implemented.

5. A computer storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 2 are implemented.

Images