Dialogue intention understanding and interpretation text generation method based on deep learning

Abstract

The invention provides a dialogue intention understanding and interpretation text generation method based on deep learning, which comprises a training process and a prediction process, and is characterized in that in the training process, firstly, sample data is acquired and preprocessed, then feature vectors are extracted through a pre-trained multi-mode encoder, then the feature vectors are aligned through projection, and a prediction result is obtained; and finally, training a generation model through a self-criticality learning method, optimizing an output text based on a classification category, and obtaining an interpretation generation text closely related to multi-modal data such as a scene and the like. In the prediction process, after a multi-modal sample is obtained and preprocessed, data is embedded into a unified vector space through a pre-trained multi-modal encoder, then a vector is input into a trained explainable multi-modal intention recognition model, an intention category is obtained through a classification part of the model, and a prediction result is obtained. The generation section obtains an intention interpretation of natural language generation. According to the invention, the interpretation text is generated while intention classification is realized.

Description

Technical Field

[0001] The present invention relates to computer vision and natural language processing technology, and in particular to a method for understanding conversation intentions and generating interpretation text based on deep learning. Background Art

[0002] Intent recognition, also known as intent classification, is a natural language understanding method that aims to understand the user's intent or purpose in a conversation, so that systems such as virtual assistants or chatbots can appropriately perform certain actions or provide relevant responses. The mainstream intent recognition models in the early days mainly focused on the text modality. In real-world scenarios, understanding the intent of a conversation requires the use of information from multiple modalities (text, visual, and auditory). For example, when people understand the intent of others, body language, facial expressions, and voice intonation significantly affect the interpretation of intent. Multimodal intent recognition aims to discern potential intent or purpose from text, visual, or auditory input, integrating multiple perspectives such as facial expressions and voice intonation attributes. For example, the sentence "I won't listen to Jonah's lecture anymore" expresses the intention to stop Jonah from speaking when the tone is flat and the expression is neutral, conveying a negative intent. However, if it is conveyed with a changing tone, relaxed body language, and exaggerated facial expressions, it may be interpreted as a joke, conveying a positive intent, which is exactly the intent recognition result we need.

[0003] In 2022, Tsinghua University introduced a new task called Multimodal Intent Recognition (MIR), which combines text, visual, and audio data. In addition, a MIR benchmark dataset was launched to support research on this task. Although existing MIR methods can achieve satisfactory intent classification results, they often lack the basis for explaining the intent classification decision process, are not effective in utilizing visual and auditory modalities, and cannot simultaneously classify and output reasonable text explanations. Summary of the invention

[0004] In order to solve the problem in the prior art that it is impossible to generate natural language text interpretations while classifying multimodal intents, the present invention proposes an explainable multimodal intent recognition (EMIR), which constructs an optimal multimodal semantic space by utilizing modal bias-aware multimodal feature fusion, and generates interpretation texts based on prototype prompts and through a multi-task paradigm.

[0005] A method for understanding conversation intention and generating interpretation text based on deep learning, the method comprising a training process and a prediction process, the training process comprising:

[0006] Step A1: Obtain the conversation video, audio, and corresponding transcribed text and perform preprocessing. The preprocessing includes converting the text into tokens, cropping the audio and video, and tokenizing them, and obtaining category labels that annotate the conversation intent.

[0007] Step A2: construct an interpretable multimodal intent recognition model, which includes a multimodal encoder model, an intent classification model, and an explanation text generation model;

[0008] Step A3: Embed the conversation video, audio, and text into a unified vector space through the pre-trained multimodal encoder model to obtain the vector representation of each token;

[0009] Step A4: Use the classification loss of the intent classification model to train the classification weight of the intent classification model, and obtain the intent classification result through the intent classification model;

[0010] Step A5: When training the intent classification model, the self-critical learning method is used to optimize the explanation text generation model, by calculating the similarity between the intent explanation text and the vector representation, and training the weight of the explanation text generation model based on the self-critical feedback of the similarity measurement to increase the similarity between the explanation text and the vector representation;

[0011] The prediction process includes:

[0012] Step B1: Obtain sample data and perform preprocessing, the sample data includes conversation video, audio and corresponding transcribed text, cut the video and audio into segments, and tokenize the text and the cut video and audio segments;

[0013] Step B2: Embed the conversation video, audio, and text into a unified vector space through a pre-trained multimodal encoder;

[0014] Step B3: Input the embedding vector into the trained interpretable multimodal intent recognition model. The classification part of the model obtains the intent category, and the generation part obtains the intent explanation generated by natural language.

[0015] Furthermore, in step A3, the embedding vector is extracted by a pre-trained multimodal encoder, specifically including:

[0016] Step A31: Input the video audio clip and text into the encoder to obtain the embedding vector:

[0017] v i , v t , v a =ImageBind encoder (I j , T j , A j );

[0018] Where v represents the embedding vector of each modality, i represents the image modality, t represents the text modality, a represents the audio modality, I, T, A represent the original modality input, and j represents the token number;

[0019] Furthermore, in step A4, the intent classification model in the interpretable multimodal intent recognition model is used to perform intent classification, which specifically includes:

[0020] Step A41: Align the embedded vectors through a projection operation:

[0021]

[0022] Among them, v′ i , v′ a represents the aligned embedding vector, g represents the alignment weight, and the operation Represents vector element-wise multiplication operation;

[0023] Step A42: Input the aligned embedding vector into the pre-trained intent classification model to obtain the classification vector:

[0024] h mask =LLaMA([v′ i , v t , v′ a , p in ])

[0025] Among them, LLaMA is the transformer of the intent classification model, h mask is the classification vector output by the model, p in is the soft hint vector for classification, which can be trained;

[0026] Step A43: Input the classification vector into the fully connected neural network and obtain the intent classification result through the classification loss:

[0027]

[0028] Among them, P(s|p in ) is the output intent category probability distribution, This is the process of obtaining this probability distribution through the soft prompt vector and the intent classification model M. M is a neural network that can be classified, such as a fully connected network, a resnet convolutional network, etc.

[0029] Furthermore, in step A5, the self-critical learning method is used to optimize the explanation text generation model, specifically including:

[0030] Step A51: concatenate the classification vector h and the trainable prompt vector Z to obtain the input vector group t of the explanatory text generation model. The trainable prompt vector Z is randomly initialized and updated during the training process:

[0031]

[0032] Among them, argmax is the vocabulary selection operation of the output text, n is the maximum length of the generated text, and θ is the hidden state representation of the autoregressive text generation process;

[0033] Step A52: Use the trainable hint vector T and embeddings t of modalities other than text e Projected to the hidden layer of the interpretation text generation model, we get v″ t ,v″ t is the new hidden state of the middle layer of the model, and the trainable prompt vector T is initialized randomly and trained with the model:

[0034] v″t=Proj adapter (ImageBind encoder ([T,t e ]));

[0035] Step A53: Use existing multimodal generative models such as GPT to input samples and output scene-related explanations tg , explained by the multimodal encoder and intent classification model tg The intention classification result s′ tg Similarly, the explanation text generation model is used to generate explanation s i , the interpretation text s is obtained through the intent classification model i The intention classification result s′ i ;

[0036] Step A54: Use the cosine distance metric between the generated text and the input text to express the similarity between the intended explanation text and the vector representation, measure the self-critical feedback through the similarity, that is, explain the generation loss, and then back-propagate to adjust the weights of the explanation text generation model to explain the generation loss. It is expressed as:

[0037]

[0038] Among them, t gt is the category label that annotates the conversation intent, φ represents the learnable parameters of the model, is the gradient descent algorithm, cos(.) is the cosine function used to calculate the similarity between the generated text and the true interpretation, when s′ i Greater than the target intention score s tg, the model will receive a positive reward, indicating that the generated explanation text has a positive contribution to the intent classification. On the contrary, when s′ i Less than the target intention score s tg When , the reward is negative, it means that the generated explanation text has a negative impact, and the model will be punished and avoid further generating this explanation;

[0039] Step A55: Generate loss using explanation The model is trained with classification loss, which is defined by the cross entropy.

[0040] Furthermore, the explainable multimodal intent recognition model includes a multimodal encoder model, an intent classification model and an explanation text generation model, and the intent classification model and the explanation text generation model share a pre-trained transformer.

[0041] The multimodal encoder model includes vector embedding and adapters. Vector embedding is used to embed the input audio and video clips into the vector space. The adapter is used to add the audio and video modes to the multimodal model through the projection adapter. The n-layer transformer includes word segmentation and encoding, adding position encoding, multi-head self-attention, summation and regularization, and feedforward neural network. Multi-head self-attention uses multiple queries (matrices Q, K, V) to calculate in parallel to select multiple information from the input information. Each attention focuses on different parts of the input information, and then splicing is performed. The feedforward neural network converts the multiple outputs of the multi-head self-attention into the size of a matrix through splicing and matrix multiplication. Summation and regularization convert the input into a square with a mean of 0. The data with a difference of 1 is normalized to prevent the input data from falling into the saturation area of ​​the activation function. The same transformer structure provides both classification results and text explanations, and performs effective and relevant training. The classification structure of the intent classification model is provided by the classifier and softmax to provide label probability distribution. The classifier can be a fully connected network or a convolutional network. The explanation structure of the explanatory text generation model is obtained by the fully connected network and the topk structure of the vector proximity sorting to obtain an approximate vector sorting. The temperature determines which one of the most approximate first n vectors is selected as the generated word. The de-tokenizer converts the vector into text by looking up the table to obtain the text explanation.

[0042] The present invention proposes a method for understanding conversation intention and generating explanation text based on deep learning, which includes a training process and a prediction process. In the training process, sample data is first obtained and preprocessed to obtain multimodal conversation data and labels, and audio and video data are cut into segments; then, feature vectors are extracted through a pre-trained multimodal encoder; then, the feature vectors are aligned through projection and input into the hidden layer of a language model to obtain a classification result; finally, through a self-critical learning method, the explanation generation results of multimodal language models such as GPT4 are used to optimize the output text based on the classification category, and the explanation generation text closely related to multimodal data such as scenes is obtained; in the prediction process, after obtaining multimodal samples and preprocessing, a fixed-parameter encoder is used to obtain feature vectors, and the feature vectors are added to the hidden layer of the trained language model through a trained projector; finally, through the special token output of the language model, classification is obtained through a trained fully connected network, and in addition to this special token, a natural language explanation output closely related to the scene conversation is obtained through a language generation model fine-tuned by self-criticality.

[0043] The beneficial technical effects of the present invention are:

[0044] (1) Based on the self-critical feedback tuning intention explanation generation model of the present invention, the intention expressed by the speaker in the conversation can be classified according to the video, audio and corresponding text, and an accurate natural language explanation of why this intention is expressed can be generated through self-critical training of the intention and the explanation text, thus realizing the combination of intention classification and explanation generation, and generating explanation text at the same time as the intention classification;

[0045] (2) During the training process, the explanation generation results of pre-trained generative models such as GPT can be trained and adapted to different dialogue scenarios and classifications without relying on manual annotation. Self-critical learning can optimize the explanation generation effect based on the classification results and obtain accurate explanations related to the dialogue, solving the problem of mismatch between text explanations and classifications output by existing models.

[0046] (3) The adapter multimodal model used can utilize different large language model bases through training, such as LLaMA, Mistral, etc. At the same time, the interpretable multimodal intent recognition model proposed in the present invention includes a multimodal encoder model, an intent classification model and an interpretation text generation model, which integrates the adaptation, classification and interpretation text generation parts into one large model, and the classification model and the generation model share the transformer. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 It is a flowchart of a training process of a method for understanding conversation intention and generating interpretation text based on deep learning provided by an embodiment of the present invention;

[0049] Figure 2 is a schematic diagram of the structure of an explainable multimodal intent recognition model provided by an embodiment of the present invention;

[0050] Figure 3 It is a flowchart of the prediction process of a method for understanding conversation intention and explaining text generation based on deep learning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] The present invention proposes a method for understanding conversation intention and generating interpretation text based on deep learning, including a training process and a prediction process, such as Figure 1 As shown, the training process includes:

[0053] See also Figure 1 , the training process includes:

[0054] Step S1, obtaining a conversation sample including video and audio and its corresponding text transcription content, and obtaining a category label of the annotated conversation intent;

[0055] Step S2: convert the text into tokens, and cut the voice and video into segments that can be tokenized. The segment length is preferably two seconds. Segments less than two seconds are padded with zeros, and then the cut segments are tokenized.

[0056] Step S3: construct an interpretable multimodal intent recognition model, where the interpretable multimodal intent recognition model includes a multimodal encoder model, an intent classification model, and an explanation text generation model;

[0057] Step S4: embed the conversation video, audio and text into a unified vector space through the pre-trained multimodal encoder model to obtain a vector representation of each token. One token obtains a 768x1 dimension vector, which specifically includes:

[0058] Step S41: Input the video audio clip and text into the encoder to obtain an embedding vector:

[0059] v i , v t , v a =ImageBind encoder (I j , T j , A j );

[0060] Where v represents the embedding vector of each modality, i represents the image modality, t represents the text modality, a represents the audio modality, I, T, A represent the original modality input, and j represents the token number;

[0061] Step S5: using the classification loss of the intent classification model to train the classification weights of the intent classification model to optimize the intent classification result, specifically including:

[0062] Step S51, align the embedded vectors through a projection operation:

[0063]

[0064] Among them, v′ i , v′ a represents the aligned embedding vector, g represents the alignment weight, and the operation Represents vector element-wise multiplication operation;

[0065] Step S52: Input the aligned embedding vector into the pre-trained intent classification model to obtain a classification vector:

[0066] h mask =LLaMA([v′ i , v t , v′ a , p in ])

[0067] Among them, LLaMA is the transformer of the intent classification model, h mask is the classification vector output by the model, p in is the soft hint vector for classification, which can be trained;

[0068] Step S53: Input the classification vector into the fully connected neural network, and obtain the intent classification result through the classification loss:

[0069]

[0070] Among them, P(s|p in ) is the output intent category probability distribution, This is the process of obtaining this probability distribution through the soft prompt vector and the intent classification model M. M is a neural network that can be classified, such as a fully connected network, a resnet convolutional network, etc.

[0071] Step S6: When training the intent classification model, the self-critical learning method is used to optimize the interpretation text generation model, by calculating the similarity between the intent interpretation text and the vector representation, and training the weight of the interpretation text generation model according to the self-critical feedback of the similarity measurement to increase the similarity between the interpretation text and the vector representation, specifically including:

[0072] Step S61: concatenate the classification vector h and the trainable prompt vector Z to obtain an input vector group t of the interpretation text generation model. The trainable prompt vector Z is randomly initialized and updated during the training process:

[0073]

[0074] Among them, argmax is the vocabulary selection operation of the output text, n is the maximum length of the generated text, and θ is the hidden state representation of the autoregressive text generation process.

[0075] Step S62: Use the trainable hint vector T and other modal embeddings t besides text e Projected to the hidden layer of the interpretation text generation model, we get v″ t ,v″ t is the new hidden state of the middle layer of the model, and the trainable prompt vector T is initialized randomly and trained with the model:

[0076] v″ t =Proj adapter (ImageBind encoder ([T,t e ])).

[0077] Step S63: Use existing multimodal generation models such as GPT to input samples and output scene-related explanations tg , explained by the multimodal encoder and intent classification model tg The intention classification result s′ tg Similarly, the explanation text generation model is used to generate explanation s i , the interpretation text s is obtained through the intent classification model i The intention classification result s′ i , the intent classification result here is not the output label, but the probability representation of each intent category.

[0078] Step S64: Use the cosine distance measurement between the generated text and the input text to express the similarity between the intended explanation text and the vector representation, measure the self-critical feedback through the similarity, that is, explain the generation loss, and then back-propagate to adjust the weight of the explanation text generation model, and explain the generation loss. It is expressed as:

[0079]

[0080] Among them, t gt is the category label that annotates the conversation intent, φ represents the learnable parameters of the model, is the gradient descent algorithm, cos(.) is the cosine function used to calculate the similarity between the generated text and the true interpretation, when s′ i Greater than the target intention score s tg , the model will receive a positive reward, indicating that the generated explanation text has a positive contribution to the intent classification. On the contrary, when s′ i Less than the target intention score s tg When , the reward is negative, it means that the generated explanation text has a negative impact, and the model will be penalized and avoid further generating this explanation.

[0081] Step S65: Generate loss using explanation The model is trained with classification loss, which is defined by the cross entropy.

[0082] Figure 2 It is a structural diagram of the explainable multimodal intent recognition model proposed in the present invention. As shown in the figure, the explainable multimodal intent recognition model includes a multimodal encoder model, an intent classification model and an explanation text generation model. The intent classification model and the explanation text generation model share a pre-trained transformer.

[0083] The multimodal encoder model includes vector embedding and adapters. Vector embedding is used to embed the input audio and video clips into the vector space. The adapter is used to add the audio and video modes to the multimodal model through the projection adapter; the n-layer transformer includes word segmentation and encoding, adding position encoding, multi-head self-attention, summation and regularization, and feedforward neural network. Multi-head self-attention uses multiple queries (matrices Q, K, V) to calculate in parallel to select multiple information from the input information. Each attention focuses on different parts of the input information, and then splicing is performed. The feedforward neural network converts the multiple outputs of the multi-head self-attention into the size of a matrix through splicing and matrix multiplication. Summation and regularization convert the input into data with a mean of 0 and a variance of 1. We normalize the data before sending it to the next layer or output structure to prevent the input data from falling into the saturation area of ​​the activation function. The same transformer structure provides both classification results and text explanations, and performs effective and relevant training. The classification structure of the intent classification model is provided by the classifier and softmax to provide label probability distribution. The classifier can be a fully connected network or a convolutional network. The explanation structure of the explanatory text generation model is obtained by the fully connected network and the topk structure of the vector proximity sorting to obtain an approximate vector sorting. The temperature determines which one of the most approximate first n vectors is selected as the generated word. The de-tokenizer converts the vector into text through a table lookup to obtain a text explanation.

[0084] The prediction process of the method for understanding conversation intention and explaining text generation based on deep learning proposed by the present invention is as follows: Figure 3 As shown, the prediction process includes:

[0085] Step C1, data processing, obtaining multimodal samples and performing data processing, the sample data includes conversation video, audio and corresponding transcribed text, cutting the video and audio into segments, and tokenizing the text and the cut video and audio segments;

[0086] Step C2: Use a multimodal encoder with fixed parameters to embed the conversation video, audio, and text into a unified vector space to obtain a feature vector, and load it into the hidden layer of the intent classification model through the trained projector;

[0087] Step C3: obtaining a classification result through the classification structure of the intent classification model, that is, the trained intent classification model;

[0088] Step C4: pass the output of the intent classification model to the explanation text generation model as input, and process it through the explanation text generation model trained by the self-critical learning method to obtain a natural language explanation output text closely related to the scene dialogue.

[0089] In this process, the entire embedding is passed to the generative model to obtain a series of output embeddings, and the closest word embeddings in the dictionary are obtained through the generative text method. These closest word embeddings are interpreted as the output text by looking up the text-token table.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for understanding conversation intention and generating interpretation text based on deep learning, characterized in that: The method includes a training process and a prediction process, and the training process includes: Step A1: Obtain the conversation video, audio, and corresponding transcribed text and perform preprocessing. The preprocessing includes converting the text into tokens, cropping the audio and video, and tokenizing them, and obtaining category labels that mark the conversation intent. Step A2: construct an interpretable multimodal intent recognition model, which includes a multimodal encoder model, an intent classification model, and an explanation text generation model; Step A3: Embed the conversation video, audio, and text into a unified vector space through the pre-trained multimodal encoder model to obtain the vector representation of each token; Step A4: Use the classification loss of the intent classification model to train the classification weight of the intent classification model, and obtain the intent classification result through the intent classification model; Step A5: When training the intent classification model, the self-critical learning method is used to optimize the explanation text generation model, by calculating the similarity between the intent explanation text and the vector representation, and training the weight of the explanation text generation model based on the self-critical feedback of the similarity measurement to increase the similarity between the explanation text and the vector representation; The prediction process includes: Step B1: Obtain sample data and perform preprocessing, the sample data includes conversation video, audio and corresponding transcribed text, cut the video and audio into segments, and tokenize the text and the cut video and audio segments; Step B2: Embed the conversation video, audio, and text into a unified vector space through a pre-trained multimodal encoder; Step B3: Input the embedding vector into the trained interpretable multimodal intent recognition model. The classification part of the model obtains the intent category, and the generation part obtains the intent explanation generated by natural language.

2. The method according to claim 1, characterized in that In step A3, the embedding vector is extracted through the pre-trained multimodal encoder, which specifically includes: Step A31: Input the video audio clip and text into the encoder to obtain the embedding vector: v i ,v t ,v a =ImageBind encoder (I j ,T j ,A j ); Among them, v represents the embedding vector of each modality, i represents the image modality, t represents the text modality, a represents the audio modality, I, T, A represent the original modality input, and j represents the token number.

3. The method according to claim 1, characterized in that In step A4, the intent classification model in the interpretable multimodal intent recognition model is used to classify the intent, which specifically includes: Step A41: Align the embedded vectors through a projection operation: Among them, v′ i , v′ a represents the aligned embedding vector, g represents the alignment weight, and the operation Represents vector element-wise multiplication operation; Step A42: Input the aligned embedding vector into the pre-trained intent classification model to obtain the classification vector: h mask =LLaMA([v i ′,v t ,v′ a ,p in ]) Among them, LLaMA is the transformer of the intent classification model, h mask is the classification vector output by the model, p in is the soft hint vector used for classification; Step A43: Input the classification vector into the fully connected neural network and obtain the intent classification result through the classification loss: Among them, P(s|p in ) is the output intent category probability distribution, It is the process of obtaining this probability distribution through the soft prompt vector and the intent classification model M.

4. The method according to claim 1, characterized in that: In step A5, the self-critical learning method is used to optimize the explanation text generation model, which specifically includes: Step A51: concatenate the classification vector h and the trainable prompt vector Z to obtain the input vector group t of the explanatory text generation model. The trainable prompt vector Z is randomly initialized and updated during the training process: Among them, argmax is the vocabulary selection operation of the output text, n is the maximum length of the generated text, and θ is the hidden state representation of the autoregressive text generation process; Step A52: Use the trainable hint vector T and embeddings t of modalities other than text e Projected to the hidden layer of the interpretation text generation model, we get v″ t ,v″ t is the new hidden state of the middle layer of the model, and the trainable prompt vector T is initialized randomly and trained with the model: v″ t =Proj adapter (ImageBind encoder ([T,t e ])); Step A53: Use the existing GPT generation model, input the sample, and output the scene-related explanation s tg , explained by the multimodal encoder and intent classification model tg The intention classification result s′ tg Similarly, the explanation text generation model is used to generate explanation s i , the interpretation text s is obtained through the intent classification model i The intention classification result s′ i ; Step A54: Use the cosine distance metric between the generated text and the input text to express the similarity between the intended explanation text and the vector representation, measure the self-critical feedback through the similarity, that is, explain the generation loss, and then back-propagate to adjust the weights of the explanation text generation model to explain the generation loss. It is expressed as: Among them, t gt is the category label that annotates the conversation intent, φ represents the learnable parameters of the model, is the gradient descent algorithm, cos(.) is the cosine function used to calculate the similarity between the generated text and the true interpretation, when s′ i Greater than the target intention score s tg , the model will receive a positive reward, indicating that the generated explanation text has a positive contribution to the intent classification. On the contrary, when s′ i Less than the target intention score s tg When , the reward is negative, it means that the generated explanation text has a negative impact, and the model will be punished and avoid further generating this explanation; Step A55: Generate loss using explanation The model is trained with classification loss, which is defined by the cross entropy.

5. The method according to claim 1, characterized in that The intent classification model and the interpretation text generation model in the explainable multimodal intent recognition model share the pre-trained transformer. The multimodal encoder model includes vector embedding and adapters. Vector embedding is used to embed the input audio and video clips into the vector space. The adapter is used to add the audio and video modalities to the multimodal model through the projection adapter. The transformer includes word segmentation and encoding, adding position encoding, multi-head self-attention, summing and regularization, and feedforward neural network. Multi-head self-attention uses multiple queries to calculate in parallel to select multiple information from the input information. Each attention focuses on different parts of the input information, and then splicing is performed. The feedforward neural network converts the multiple outputs of the multi-head self-attention into the size of a matrix through splicing and matrix multiplication. Summing and regularization convert the input into data with a mean of 0 and a variance of 1, and normalizes the data. The classification structure of the intent classification model is provided with label probability distribution by the classifier and the softmax function. The explanation structure of the explanatory text generation model is obtained by the approximate vector sorting of the fully connected network and the topk structure of vector proximity sorting. The temperature determines to select one of the most approximate top n vectors as the generated word. The de-tokenizer converts the vector into text by looking up the table to obtain the text explanation.

Images