A method for understanding conversation intentions and generating interpretation text based on deep learning

Abstract

The present invention provides a method for understanding conversational intent and generating explanatory text based on deep learning. The method includes a training process and a prediction process. In the training process, sample data is first acquired and preprocessed. Next, feature vectors are extracted using a pretrained multimodal encoder. The feature vectors are then aligned by projection and input into the hidden layer of a language model to obtain a classification result. Finally, a generative model is trained using a self-critical learning method. The output text is optimized based on the classification category, resulting in an explanatory text that is closely related to multimodal data such as the scene. In the prediction process, after acquiring multimodal samples and preprocessing them, the data is embedded into a unified vector space using a pretrained multimodal encoder. The embedded vectors are then input into a trained, interpretable multimodal intent recognition model. The classification part of the model obtains the intent category, and the generation part obtains the intent explanation generated in natural language. The present invention achieves the simultaneous generation of explanatory text while intent classification is performed.

Description

Technical Field

[0001] The present invention relates to computer vision and natural language processing technologies, and in particular to a method for understanding conversation intent and generating interpretation text based on deep learning. Background Art

[0002] Intent recognition, also known as intent classification, is the primary goal of natural language understanding (NLU) to understand the user's intent or purpose in a conversation, enabling systems such as virtual assistants or chatbots to appropriately perform certain actions or provide relevant responses. Early mainstream intent recognition models focused primarily on the text modality. In real-world scenarios, understanding the intent of a conversation requires leveraging information from multiple modalities (text, visual, and auditory). For example, when people interpret others' intentions, body language, facial expressions, and voice intonation significantly influence intent interpretation. Multimodal intent recognition aims to discern underlying intent or purpose from textual, visual, or auditory input, integrating multiple perspectives such as facial expressions and voice intonation. For example, the sentence "I won't listen to Jonah lecture us anymore" conveys a negative intent when delivered with a flat tone and neutral expression, expressing a desire to stop Jonah from speaking. However, if delivered with a varied tone, relaxed body language, and exaggerated facial expressions, it might be interpreted as a joke, conveying a positive intent—precisely the desired intent recognition result.

[0003] In 2022, Tsinghua University introduced a new task called Multimodal Intent Recognition (MIR), which combines text, visual, and audio data. In addition, a MIR benchmark dataset was launched to support research on this task. Although existing MIR methods can achieve satisfactory intent classification results, they often lack the basis for explaining the intent classification decision process, are not effective in utilizing visual and auditory modalities, and cannot simultaneously classify and output reasonable textual explanations. Summary of the Invention

[0004] In order to solve the problem in the existing technology that it is impossible to generate natural language text explanations while classifying multimodal intents, the present invention proposes an explainable multimodal intent recognition (EMIR), which constructs an optimal multimodal semantic space by utilizing multimodal feature fusion with modal bias awareness, and generates explanatory text based on prototype prompts and a multi-task paradigm.

[0005] A method for understanding conversation intent and generating interpretation text based on deep learning, comprising a training process and a prediction process. The training process includes:

[0006] Step A1: Obtain the conversation video, audio, and corresponding transcribed text and perform preprocessing. This includes converting the text into tokens, cropping and tokenizing the audio and video, and obtaining category labels that indicate the conversation's intent.

[0007] Step A2: Build an interpretable multimodal intent recognition model. The interpretable multimodal intent recognition model includes a multimodal encoder model, an intent classification model, and an explanation text generation model.

[0008] Step A3: Embed the conversation video, audio, and text into a unified vector space through the pre-trained multimodal encoder model to obtain the vector representation of each token;

[0009] Step A4: Use the intent classification loss of the intent classification model to train the classification weights of the intent classification model, and obtain the intent classification result through the intent classification model;

[0010] Step A5: When training the intent classification model, use a self-critical learning method to optimize the interpretation text generation model. This method calculates the similarity between the intent interpretation text and its vector representation, and uses self-critical feedback from the similarity measure to train the weights of the interpretation text generation model to increase the similarity between the interpretation text and its vector representation.

[0011] The forecasting process includes:

[0012] Step B1: Obtain and preprocess sample data, including conversation video, audio, and corresponding transcribed text. Crop the video and audio into segments, and tokenize the text and the cropped video and audio segments.

[0013] Step B2: Embed the conversation video, audio, and text into a unified vector space through a pre-trained multimodal encoder;

[0014] Step B3: Input the embedding vector into the trained interpretable multimodal intent recognition model. The classification part of the model obtains the intent category, and the generation part obtains the intent explanation generated by natural language.

[0015] Furthermore, in step A3, an embedding vector is extracted through a pre-trained multimodal encoder, specifically including:

[0016] Step A31: Input the video audio clip and text into the encoder to obtain the embedding vector:

[0017] v i , v t , v a =ImageBind encoder (I j , T j , A j );

[0018] Where v represents the embedding vector of each modality, i represents the image modality, t represents the text modality, a represents the audio modality, I, T, A represent the original modality input, and j represents the token number;

[0019] Furthermore, in step A4, intent classification is performed using the intent classification model in the interpretable multimodal intent recognition model, specifically including:

[0020] Step A41: Align the embedded vectors through a projection operation:

[0021]

[0022] Among them, v′ i 、v′ a represents the aligned embedding vector, g represents the alignment weight, and the operation Represents a vector element-wise multiplication operation;

[0023] Step A42: Input the aligned embedding vector into the pre-trained intent classification model to obtain the classification vector:

[0024] h mask =LLaMA([v′ i , v t , v′ a , p in ])

[0025] Among them, LLaMA is the transformer of the intent classification model, h mask is the classification vector output by the model, p in is the soft hint vector for classification, which can be trained;

[0026] Step A43: Input the classification vector into the fully connected neural network and obtain the intent classification result through the classification loss:

[0027]

[0028] Among them, P(s|p in ) is the output intent category probability distribution, This probability distribution is obtained through the soft prompt vector and the intent classification model M. M is a neural network that can be classified, such as a fully connected network and a resnet convolutional network.

[0029] Furthermore, in step A5, a self-critical learning method is used to optimize the interpretation text generation model, specifically including:

[0030] Step A51: Concatenate the classification vector h and the trainable prompt vector Z to obtain the input vector group t of the interpretation text generation model. The trainable prompt vector Z is randomly initialized and updated during the training process:

[0031]

[0032] Among them, argmax is the vocabulary selection operation of the output text, n is the maximum length of the generated text, and θ is the hidden state representation of the autoregressive text generation process;

[0033] Step A52: Use the trainable hint vector T and the embedding t of other modalities besides text e Projected to the hidden layer of the interpretation text generation model, we get v″ t ,v″ t is the new hidden state of the middle layer of the model, and the trainable prompt vector T is initialized randomly and trained with the model:

[0034] v″t=Proj adapter (ImageBind encoder ([T,t e ]));

[0035] Step A53: Use existing multimodal generative models such as GPT to input samples and output scene-related explanations tg , explained by the multimodal encoder and intent classification model tg Intent classification result s′ tg , similarly, using the explanation text generation model to generate explanation s i , get the explanation text s through the intent classification model i Intent classification result s′ i ;

[0036] Step A54: Use the cosine distance metric between the generated text and the input text to express the similarity between the intended explanation text and the vector representation, and use the self-critical feedback of the similarity metric to explain the generation loss, and then backpropagate to adjust the weights of the explanation text generation model to explain the generation loss. Expressed as:

[0037]

[0038] Among them, t gt is the category label that annotates the conversation intent, φ represents the learnable parameters of the model, is a gradient descent algorithm, cos(.) is a cosine function used to calculate the similarity between the generated text and the true explanation, when s′ i Greater than the target intention score s tgWhen , the model will receive a positive reward, indicating that the generated explanation text has a positive contribution to the intent classification. On the contrary, when s′ i Less than the target intention score s tg When , the reward is negative, it means that the generated explanation text has a negative impact, and the model will be punished and avoid generating this explanation further;

[0039] Step A55: Generate loss using explanations The model is trained with classification loss, which is defined by cross entropy.

[0040] Furthermore, the explainable multimodal intent recognition model includes a multimodal encoder model, an intent classification model and an explanation text generation model, and the intent classification model and the explanation text generation model share a pre-trained transformer.

[0041] The multimodal encoder model includes vector embedding and adapters. Vector embedding is used to embed the input audio and video clips into the vector space. The adapter is used to add the audio and video modalities to the multimodal model through the projection adapter; the n-layer transformer includes word segmentation and encoding, adding position encoding, multi-head self-attention, summation and regularization and feedforward neural network. Multi-head self-attention uses multiple queries (matrices Q, K, V) to calculate in parallel to select multiple information from the input information. Each attention focuses on different parts of the input information, and then splicing is performed. The feedforward neural network converts the multiple outputs of the multi-head self-attention into the size of a matrix through splicing and matrix multiplication. Summation and regularization convert the input into a square with a mean of 0. For data with a difference of 1, the data is normalized to prevent the input data from falling into the saturation zone of the activation function. The same transformer structure provides both classification results and text explanations, and performs effective and relevant training. The classification structure of the intent classification model is provided by the classifier and softmax to provide label probability distribution. The classifier can be a fully connected network or a convolutional network. The interpretation structure of the explanatory text generation model is obtained by a fully connected network and a topk structure of vector proximity sorting to obtain approximate vector sorting. The temperature determines which of the top n most approximate vectors is selected as the generated word. The de-tokenizer converts the vector into text through a table lookup to obtain the text explanation.

[0042] The present invention proposes a method for understanding conversation intentions and generating interpretation text based on deep learning, which includes a training process and a prediction process. In the training process, sample data is first obtained and preprocessed to obtain multimodal conversation data and labels, and audio and video data are cropped into segments; then, feature vectors are extracted through a pre-trained multimodal encoder; then, the feature vectors are aligned through projection and input into the hidden layer of a language model to obtain a classification result; finally, through a self-critical learning method, the interpretation generation results of multimodal language models such as GPT4 are utilized to optimize the output text based on the classification category, and an interpretation generation text closely related to multimodal data such as scenes is obtained; in the prediction process, after obtaining multimodal samples and preprocessing, a fixed-parameter encoder is used to obtain feature vectors, and they are added to the hidden layer of the trained language model through a trained projector; finally, classification is obtained through a special token output of the language model through a trained fully connected network. In addition to this special token, a natural language interpretation output closely related to the scene conversation is obtained through a language generation model that has been fine-tuned by self-criticality.

[0043] The beneficial technical effects of the present invention are:

[0044] (1) Based on the self-critical feedback tuning intention interpretation generation model of the present invention, the intention expressed by the speaker in the conversation can be classified according to the video, audio and corresponding text, and an accurate natural language explanation of why the intention is expressed is generated through self-critical training of the intention and the interpretation text. This realizes the combination of intention classification and interpretation generation, and generates interpretation text at the same time as the intention classification.

[0045] (2) During the training process, the explanation generation results of pre-trained generative models such as GPT can be trained and adapted to different dialogue scenarios and classifications without relying on manual annotation. Self-critical learning can optimize the explanation generation effect based on the classification results and obtain accurate explanations related to the dialogue, solving the problem of mismatch between the text explanation output and the classification of the existing model.

[0046] (3) The adapter multimodal model used can utilize different large language model bases through training, such as LLaMA, Mistral, etc. At the same time, the interpretable multimodal intent recognition model proposed in the present invention includes a multimodal encoder model, an intent classification model and an interpretation text generation model, integrating adaptation, classification and interpretation text generation parts into a large model, and the classification model and the generation model share the transformer. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 This is a flowchart of a training process for a method for understanding conversation intent and generating interpretation text based on deep learning, provided by an embodiment of the present invention;

[0049] Figure 2 is a schematic diagram of the structure of an interpretable multimodal intent recognition model provided by an embodiment of the present invention;

[0050] Figure 3 This is a flowchart of the prediction process of a method for understanding conversation intent and generating interpretation text based on deep learning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0052] The present invention proposes a method for understanding conversation intentions and generating interpretation text based on deep learning, which includes a training process and a prediction process. Figure 1 As shown, the training process includes:

[0053] See also Figure 1 , the training process includes:

[0054] Step S1: Obtain a conversation sample containing video and audio and its corresponding text transcription content, and simultaneously obtain the category label of the annotated conversation intent;

[0055] Step S2: Convert the text into tokens and crop the voice and video into segments that can be tokenized. The segment length is preferably two seconds. Segments less than two seconds are padded with zeros, and then the cropped segments are tokenized.

[0056] Step S3: construct an interpretable multimodal intent recognition model, which includes a multimodal encoder model, an intent classification model, and an explanation text generation model;

[0057] Step S4: Use the pre-trained multimodal encoder model to embed the conversation video, audio, and text into a unified vector space to obtain a vector representation of each token. Each token is represented by a 768x1 dimension vector, specifically including:

[0058] Step S41: Input the video, audio clip and text into the encoder to obtain the embedding vector:

[0059] v i , v t , v a =ImageBind encoder (I j , T j , A j );

[0060] Where v represents the embedding vector of each modality, i represents the image modality, t represents the text modality, a represents the audio modality, I, T, A represent the original modality input, and j represents the token number;

[0061] Step S5: Use the intent classification loss of the intent classification model to train the classification weights of the intent classification model to optimize the intent classification results, specifically including:

[0062] Step S51: Align the embedded vectors through a projection operation:

[0063]

[0064] Among them, v′ i 、v′ a represents the aligned embedding vector, g represents the alignment weight, and the operation Represents a vector element-wise multiplication operation;

[0065] Step S52: Input the aligned embedding vector into the pre-trained intent classification model to obtain the classification vector:

[0066] h mask =LLaMA([v′ i , v t , v′ a , p in ])

[0067] Among them, LLaMA is the transformer of the intent classification model, h mask is the classification vector output by the model, p in is the soft hint vector for classification, which can be trained;

[0068] Step S53: Input the classification vector into the fully connected neural network and obtain the intent classification result through the classification loss:

[0069]

[0070] Among them, P(s|p in ) is the output intent category probability distribution, This probability distribution is obtained through the soft hint vector and the intent classification model M. M is a neural network that can be classified, such as a fully connected network or a resnet convolutional network.

[0071] Step S6: When training the intent classification model, use a self-critical learning method to optimize the interpretation text generation model. The similarity between the intent interpretation text and the vector representation is calculated, and the weight of the interpretation text generation model is trained based on the self-critical feedback of the similarity measure to increase the similarity between the interpretation text and the vector representation. Specifically, the following steps are performed:

[0072] Step S61: Concatenate the classification vector h and the trainable prompt vector Z to obtain the input vector group t of the interpretation text generation model. The trainable prompt vector Z is randomly initialized and updated during the training process:

[0073]

[0074] Among them, argmax is the vocabulary selection operation of the output text, n is the maximum length of the generated text, and θ is the hidden state representation of the autoregressive text generation process.

[0075] Step S62: Use the trainable hint vector T and other modal embeddings t except text e Projected to the hidden layer of the interpretation text generation model, we get v″ t ,v″ t is the new hidden state of the middle layer of the model, and the trainable prompt vector T is initialized randomly and trained with the model:

[0076] v″ t =Proj adapter (ImageBind encoder ([T,t e ])).

[0077] Step S63: Using existing multimodal generative models such as GPT, input samples and output scene-related explanations tg , explained by the multimodal encoder and intent classification model tg Intent classification result s′ tg , similarly, using the explanation text generation model to generate explanation s i , get the explanation text s through the intent classification model i Intent classification result s′ i ,The intent classification result here is not the output label, but the probability representation of each intent category.

[0078] Step S64: Use the cosine distance between the generated text and the input text to measure the similarity between the intended interpretation text and the vector representation, measure the self-critical feedback through the similarity, and then adjust the weight of the interpretation text generation model through backpropagation to explain the generation loss. Expressed as:

[0079]

[0080] Among them, t gt is the category label that annotates the conversation intent, φ represents the learnable parameters of the model, is a gradient descent algorithm, cos(.) is a cosine function used to calculate the similarity between the generated text and the true explanation, when s′ i Greater than the target intention score s tg When , the model will receive a positive reward, indicating that the generated explanation text has a positive contribution to the intent classification. On the contrary, when s′ i Less than the target intention score s tg When , the reward is negative, it means that the generated explanation text has a negative impact, and the model will be penalized and avoid generating this explanation further.

[0081] Step S65: Generate loss using explanation The model is trained with classification loss, which is defined by cross entropy.

[0082] Figure 2 This is a structural diagram of the explainable multimodal intent recognition model proposed in the present invention. As shown in the figure, the explainable multimodal intent recognition model includes a multimodal encoder model, an intent classification model and an explanation text generation model. The intent classification model and the explanation text generation model share a pre-trained transformer.

[0083] The multimodal encoder model includes vector embedding and adapters. Vector embedding is used to embed the input audio and video clips into the vector space. The adapter is used to add the audio and video modalities to the multimodal model through the projection adapter; the n-layer transformer includes word segmentation and encoding, adding position encoding, multi-head self-attention, summation and regularization and feedforward neural network. Multi-head self-attention uses multiple queries (matrices Q, K, V) to calculate in parallel to select multiple information from the input information. Each attention focuses on different parts of the input information, and then splicing is performed. The feedforward neural network converts the multiple outputs of the multi-head self-attention into the size of a matrix through splicing and matrix multiplication. Summation and regularization convert the input into data with a mean of 0 and a variance of 1. We normalize the data before sending it to the next layer or output structure to prevent the input data from falling into the saturation zone of the activation function. The same transformer structure provides both classification results and text explanations, and performs effective and relevant training. The classification structure of the intent classification model is provided by the classifier and softmax to provide label probability distribution. The classifier can be a fully connected network or a convolutional network. The interpretation structure of the explanatory text generation model is obtained by the fully connected network and the topk structure of vector proximity sorting to obtain approximate vector sorting. The temperature determines which of the top n most approximate vectors is selected as the generated word. The de-tokenizer converts the vector into text through a table lookup to obtain the text explanation.

[0084] The prediction process of the method for understanding conversation intention and explaining text generation based on deep learning proposed by this invention is as follows: Figure 3 As shown, the prediction process includes:

[0085] Step C1: Data processing: Acquire and process multimodal samples. The sample data includes conversation video, audio, and corresponding transcribed text. The video and audio are cut into segments, and the text and the cut video and audio segments are tokenized.

[0086] Step C2: Use a fixed-parameter multimodal encoder to embed the conversation video, audio, and text into a unified vector space to obtain a feature vector, which is then loaded into the hidden layer of the intent classification model through the trained projector;

[0087] Step C3: Obtain a classification result through the classification structure of the intent classification model, that is, the trained intent classification model;

[0088] Step C4: Pass the output of the intent classification model to the interpretation text generation model as input, and process it through the interpretation text generation model trained by the self-critical learning method to obtain a natural language interpretation output text closely related to the scene dialogue.

[0089] In this process, the entire embedding is passed to the generative model to obtain a series of output embeddings, and the word embeddings closest to the embedding in the dictionary are obtained through the generative text method. These closest word embeddings are interpreted by looking up the text-token table.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for understanding conversation intentions and generating interpretation text based on deep learning, characterized in that: The method includes a training process and a prediction process, and the training process includes: Step A1: Obtain the conversation video, audio, and corresponding transcribed text and perform preprocessing. This includes converting the text into tokens, cropping and tokenizing the audio and video, and obtaining category labels that indicate the conversation's intent. Step A2: Construct an interpretable multimodal intent recognition model. The interpretable multimodal intent recognition model includes a multimodal encoder model, an intent classification model, and an explanation text generation model. The intent classification model and the explanation text generation model share a pre-trained transformer. The multimodal encoder model includes vector embedding and an adapter. The vector embedding is used to embed the input audio and video clips into the vector space. The adapter is used to add the audio and video modalities to the multimodal model through the projection adapter. The transformer includes word segmentation and encoding, position encoding, multi-head self-attention, summation and regularization, and a feedforward neural network. The multi-head self-attention uses multiple queries to parallelly calculate multiple selected from the input information. Each attention unit focuses on a different part of the input information, which is then spliced ​​together. The feedforward neural network converts the multiple outputs of the multi-head self-attention unit into a matrix through splicing and matrix multiplication. Aggregation and regularization convert the input into data with a mean of 0 and a variance of 1, normalizing the data. The classification structure of the intent classification model uses the classifier and the softmax function to provide label probability distribution. The interpretation structure of the explanatory text generation model uses a fully connected network and a topk structure of vector proximity sorting to obtain an approximate vector sorting. The temperature determines the selection of the first n most approximate vectors as the generated word. The de-tokenizer converts the vector into text through a table lookup to obtain a text interpretation. Step A3: Embed the conversation video, audio, and text into a unified vector space through the pre-trained multimodal encoder model to obtain the vector representation of each token; Step A4: Use the intent classification loss of the intent classification model to train the classification weights of the intent classification model, and obtain the intent classification result through the intent classification model; Step A5: When training the intent classification model, use a self-critical learning method to optimize the interpretation text generation model. This method calculates the similarity between the intent interpretation text and its vector representation, and uses self-critical feedback from the similarity measure to train the weights of the interpretation text generation model to increase the similarity between the interpretation text and its vector representation. The forecasting process includes: Step B1: Obtain and preprocess sample data, including conversation video, audio, and corresponding transcribed text. Crop the video and audio into segments, and tokenize the text and the cropped video and audio segments. Step B2: Embed the conversation video, audio, and text into a unified vector space through a pre-trained multimodal encoder; Step B3: Input the embedding vector into the trained interpretable multimodal intent recognition model. The classification part of the model obtains the intent category, and the generation part obtains the intent explanation generated by natural language.

2. The method according to claim 1, characterized in that In step A3, the embedding vector is extracted through the pre-trained multimodal encoder, which specifically includes: Step A31: Input the video audio clip and text into the encoder to obtain the embedding vector: v i ,v t ,v a =ImageBind encoder (I j ,T j ,A j ); Where v represents the embedding vector of each modality, i represents the image modality, t represents the text modality, a represents the audio modality, I, T, A represent the original modality input, and j represents the token number.

3. The method according to claim 1, characterized in that In step A4, intent classification is performed using the intent classification model in the interpretable multimodal intent recognition model, specifically including: Step A41: Align the embedded vectors through a projection operation: Among them, v′ i 、v′ a represents the aligned embedding vector, g represents the alignment weight, and the operation Represents a vector element-wise multiplication operation; Step A42: Input the aligned embedding vector into the pre-trained intent classification model to obtain the classification vector: h mask =LLaMA([v′ i ,v t ,v′ a ,p in ]) Among them, LLaMA is the transformer of the intent classification model, h mask is the classification vector output by the model, p in is the soft hint vector used for classification; Step A43: Input the classification vector into the fully connected neural network and obtain the intent classification result through the classification loss: Among them, P(s|p in ) is the output intent category probability distribution, It is the process of obtaining this probability distribution through the soft prompt vector and the intent classification model M.

4. The method according to claim 1, wherein In step A5, a self-critical learning method is used to optimize the interpretation text generation model, specifically including: Step A51: Concatenate the classification vector h and the trainable prompt vector Z to obtain the input vector group t of the interpretation text generation model. The trainable prompt vector Z is randomly initialized and updated during the training process: Among them, argmax is the vocabulary selection operation of the output text, n is the maximum length of the generated text, and θ is the hidden state representation of the autoregressive text generation process; Step A52: Use the trainable hint vector T and the embedding t of other modalities besides text e Projected to the hidden layer of the interpretation text generation model, we get v″ t ,v″ t is the new hidden state of the middle layer of the model, and the trainable prompt vector T is initialized randomly and trained with the model: v″ t =Proj adapter (ImageBind encoder ([T,t e ])); Step A53: Use the existing GPT generation model, input the sample, and output the scene-related explanation s tg , explained by the multimodal encoder and intent classification model tg Intent classification result s′ tg , similarly, using the explanation text generation model to generate explanation s i , get the explanation text s through the intent classification model i Intent classification result s′ i ; Step A54: Use the cosine distance metric between the generated text and the input text to express the similarity between the intended explanation text and the vector representation, and use the self-critical feedback of the similarity metric to explain the generation loss, and then backpropagate to adjust the weights of the explanation text generation model to explain the generation loss. Expressed as: Among them, t gt is the category label that annotates the conversation intent, φ represents the learnable parameters of the model, is a gradient descent algorithm, cos(.) is a cosine function used to calculate the similarity between the generated text and the true explanation, when s′ i Greater than the target intention score s tg When , the model will receive a positive reward, indicating that the generated explanation text has a positive contribution to the intent classification. On the contrary, when s′ i Less than the target intention score s tg When , the reward is negative, it means that the generated explanation text has a negative impact, and the model will be punished and avoid generating this explanation further; Step A55: Generate loss using explanations The model is trained with classification loss, which is defined by cross entropy.

Images