Multimodal intention recognition method and system for uncertain modality loss

Abstract

The present invention provides a multimodal intent recognition method and system for uncertain modality missing, belonging to the field of data processing technology. First, feature extraction is performed through an LSTM encoder; secondly, multimodal feature fusion is performed through an attention-based bidirectional gated multimodal feature fusion method, and the distance between each modality and the text modality is shortened based on the distance constraint strategy of CMD; then, the fused multimodal features are input into the Transformer codec for learning, and at the same time, different attention level prompts are input according to the uncertain modality missing situation, so as to guide the network to pay attention to those missing modalities; finally, the multimodal features are input into the classifier for intent recognition; at the same time, during the training process, the model pre-trained in the full modality scenario is used to guide the model in the classifier, so as to help the model further improve the generalization ability while reducing overfitting.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a multimodal intent recognition method for uncertain modal missing, a multimodal intent recognition system for uncertain modal missing, a computer device, and a computer-readable storage medium. Background Art

[0002] The statements in this section merely provide background art related to the present invention and do not necessarily constitute prior art.

[0003] Multimodal data provides diverse information, and data from different modalities are highly complementary. By combining information from different modalities for analysis and judgment, the model can better understand the meaning and context behind the data, thereby effectively improving model performance and robustness. However, existing multimodal intent recognition models assume that all modal data (text, audio, and visual) are always available during training and testing. In real-world environments, it is impossible to obtain data from all modalities due to some uncontrollable human factors, and multimodal intent recognition may result in missing or incomplete information from certain modalities. For example, Figure 1 As shown in the figure, when the camera is damaged, the information of the visual modality will be lost; the text cannot be obtained due to the privacy protection policy, and the audio noise is too loud due to the excessive environmental noise. When performing multimodal intent recognition, modality loss will have a negative impact on the performance and effect of the model. This is because when a certain modality data is missing, the model may not be able to fully utilize the correlation and complementarity between multiple modalities, resulting in a decrease in the performance of the model. Unfortunately, most of the existing multimodal intent recognition models cannot handle the problem of modality loss. Therefore, how to deal with the problem of modality loss in multimodal intent recognition has become a new challenge.

[0004] In response to the problem of missing modalities, some research works simply discard the missing modalities or use matrix completion methods to estimate the missing modalities, but these methods will lead to a decrease in the overall performance of the model. Because deep learning models have powerful learning capabilities, the research trend in recent years is to use deep neural networks to learn the potential relationship between available modalities. These methods can be roughly divided into two categories: joint learning and generative. Joint learning methods can directly maximize the overall performance of the model on multimodal data through an end-to-end optimization process without optimizing the encoder or decoder of each modality separately. Therefore, in recent years, joint learning methods have been used more to deal with missing modalities, which can better maintain the consistency and overall performance of the model. In the field of multimodal intent recognition, some scholars have conducted corresponding research on missing modalities and achieved certain results. However, existing research can only solve the problem of missing text modalities, but cannot solve the problem of missing uncertain modalities, and can only process multimodal data containing both visual and textual modalities.

[0005] Uncertain modality missing refers to the situation in which one or more modalities are randomly unavailable in samples in multimodal data. In the field of multimodal sentiment analysis, some joint learning-based models have been proposed to solve the problem of uncertain modality missing. These models do not need to retrain the model when different modalities are missing, reducing the complexity of the model. Although current research has achieved certain results, there are still the following shortcomings:

[0006] (1) Existing research shows that text modality plays a major role in multimodal intent recognition. However, existing research treats different modalities as equal in the process of multimodal feature fusion. Therefore, in the fusion process, multimodal features cannot interact deeply and are sensitive to noise;

[0007] (2) The lack of uncertain modalities will lead to inconsistency between the semantics of multimodal samples and those of the complete modality, which in turn leads to inaccurate model recognition. Existing studies have not fully considered this inconsistency, thus reducing the performance and robustness of the model;

[0008] (3) When dealing with the missing uncertain modalities, existing research models may over-rely on the information of existing modalities, thereby making incorrect estimates of the data distribution of the missing modalities, which in turn leads to overfitting. Summary of the invention

[0009] In order to address the shortcomings of the prior art, the present invention provides a multimodal intent recognition method and system for uncertain modality missing. First, feature extraction is performed through an LSTM (Long Short-Term Memory, long short-term memory artificial neural network) encoder; secondly, multimodal feature fusion is performed through an attention-based bidirectional gated multimodal feature fusion method, and the distance between each modality and the text modality is shortened based on the CMD distance constraint strategy; then, the fused multimodal features are input into the Transformer codec for learning, and at the same time, different attention level prompts are input according to the uncertain modality missing situation, so as to guide the network to pay attention to those missing modalities; finally, the multimodal features are input into a classifier for intent recognition. At the same time, during the training process, the model pre-trained in the full modality scenario is used to guide the model in the classifier, so as to help the model further improve the generalization ability while reducing overfitting.

[0010] In order to achieve the above object, the present invention adopts the following technical solution:

[0011] In a first aspect, the present invention provides a multimodal intent recognition method for uncertain modality loss.

[0012] A multimodal intent recognition method for uncertain modality loss includes the following processes:

[0013] Obtain multimodal data with uncertain modal missing, extract features of multimodal data through LSTM encoder, and obtain features of different modalities;

[0014] Minimize the similarity loss between text modality features and visual modality features and audio modality features, and fuse different modality features through attention-based bidirectional gated multimodal feature fusion;

[0015] The fused multimodal features are input into the Transformer model for learning. According to the lack of uncertain modalities, different attention-level cues are added to the keys and values ​​of the multi-head self-attention layer of each Transformer block to guide the Transformer model to pay attention to the missing modalities.

[0016] The features output by the Transformer model are used as the input of the classifier to obtain multimodal intent recognition results.

[0017] In a second aspect, the present invention provides a multimodal intent recognition system for uncertain modality loss.

[0018] A multimodal intention recognition system for uncertain modality loss, comprising:

[0019] The feature extraction unit is configured to: obtain multimodal data with uncertain modality missing, perform feature extraction of the multimodal data through an LSTM encoder, and obtain features of different modalities;

[0020] The feature fusion unit is configured to: minimize the similarity loss between text modality features and visual modality features and audio modality features, and fuse different modality features through an attention-based bidirectional gated multimodal feature fusion method;

[0021] The modality missing prompt unit is configured to: input the fused multimodal features into the Transformer model for learning, and add different attention level prompts to the keys and values ​​of the multi-head self-attention layer of each Transformer block according to the uncertain modality missing situation, so as to guide the Transformer model to pay attention to the missing modality;

[0022] The intent recognition unit is configured to use the features output by the Transformer model as the input of the classifier to obtain a multimodal intent recognition result.

[0023] In a third aspect, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program is suitable for being loaded by a processor and executing the multimodal intent recognition method for uncertain modality missing as described in the first aspect of the present invention.

[0024] In a fourth aspect, the present invention provides a computer device, comprising: a processor and a computer-readable storage medium;

[0025] a processor adapted to execute a computer program;

[0026] A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the multimodal intention recognition method for uncertain modality loss as described in the first aspect of the present invention is implemented.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] The present invention innovatively proposes a multimodal intent recognition model (MIRUMM model) for uncertain modal missing to solve the problem of uncertain modal missing. It adopts an attention-based bidirectional gated multimodal feature fusion module to perform multimodal feature fusion, and based on the constraint strategy of CMD distance, it shortens the distance between audio modality, visual modality and text modality, thereby solving the problem that multimodal feature fusion cannot interact deeply. According to the situation of uncertain modal missing, attention-level prompts are added to Transformer to guide the model to pay attention to these missing modalities. In addition, the present invention proposes a pre-trained model based on complete modality training to supervise the training process of the entire model in the classifier, thereby helping the model to improve the generalization performance. Experimental results on public datasets show that compared with the baseline model, the MIRUMM model proposed in the present invention has improved accuracy, F1 score, precision and recall under different uncertain modal missing rates.

[0029] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0031] Figure 1 A schematic diagram of mode loss provided in the background technology;

[0032] Figure 2 An architectural diagram of a multimodal intent recognition model for uncertain modality loss provided in Example 1 of the present invention;

[0033] Figure 3 A schematic diagram of a multimodal feature fusion module provided in Example 1 of the present invention;

[0034] Figure 4 A schematic diagram of a bidirectional gating mechanism provided in Example 1 of the present invention;

[0035] Figure 5 A schematic diagram of a modality missing prompt provided in Example 1 of the present invention;

[0036] Figure 6 A schematic diagram of multi-task parameter sensitivity analysis provided in Example 1 of the present invention;

[0037] Figure 7 A schematic diagram of sensitivity analysis of pre-trained temperature coefficient parameters provided in Example 1 of the present invention;

[0038] Figure 8A schematic diagram of visualization of different missing rates provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0039] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0040] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0041] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0042] Embodiment 1:

[0043] In the present invention, given a multimodal intention recognition data D containing three modes, t , x a , x v}, where x t , x a , x v Represents text mode, audio mode and visual mode respectively, using to represent the missing modality, where m∈v,a,t. For example, assuming the audio modality is missing, the multimodal representation can be expressed as The present invention studies how to accurately perform multimodal intent recognition based on deep learning technology in the case of uncertain modal missing, wherein uncertain modal missing refers to that the number of missing modalities is uncertain and the types of missing modalities are also uncertain.

[0044] To address the problem of uncertain modality missing, the present invention proposes a multimodal intent recognition model for uncertain modality missing (MIRUMM), such as Figure 2 As shown, an attention-based gated multimodal feature fusion module is used to perform multimodal feature fusion and close the distance between other modalities and the text modality, so as to fully explore the potential associations between the various modalities; in addition, the present invention inputs different attention level prompts in the Transformer according to the lack of uncertain modalities, so as to guide the network to pay attention to those missing modalities. MIRUMM includes two modules, including a pre-trained AGNN model trained with full modality and a main AGNN model. The present invention takes the missing audio modality as an example to illustrate the overall workflow of the MIRUMM model. The workflow of the MIRUMM model is as follows:

[0045] (1) Multimodal data with uncertain modalities missing Input the complete modal data pre-trained model AGNN and obtain the pre-trained AGNN model pair The encoding of the pre-trained AGNN model is consistent with the main AGNN model.

[0046] (2) Multimodal data with uncertain modalities missing Input the main AGNN model. First, the data of each modality is input into the corresponding LSTM encoder to extract modality-specific features; secondly, the multimodal feature fusion module fuses each modality to obtain a joint feature representation and performs similarity learning to narrow the distance between audio, visual and text modalities; then, the Transformer encoder module learns the joint feature representation and adds different modality missing prompt blocks before the Transformer original key K and value V according to the situation of uncertain modality missing; finally, the Transformer output is classified by the classifier; at the same time, the output of the pre-trained model is used to supervise the learning of the entire model in the classifier, thereby improving the generalization ability of MIRUMM.

[0047] The three modalities used in the present invention, text, vision and audio, are all time series information, and LSTM is a RNN variant suitable for processing time series data. It can effectively capture the time series information in text, vision and audio, thus laying the foundation for the subsequent multimodal feature fusion. Therefore, the modality-specific features of each modality are first learned through the LSTM encoder. Specifically, the original feature X is obtained through LSTM. a Extract the utterance-level features E a ; The encoder of the visual modality is the same as the audio, which is also LSTM, reading the original feature X v And output the discourse-level visual feature E v ; The text encoder also uses LSTM to generate the original feature X t Extract the discourse-level text features E t , the process of modal feature extraction is shown in formula (1):

[0048]

[0049] Among them, x m (m∈l, v, a) represents the input features of each modality, (m∈l, v, a) represents the parameters of the LSTM model.

[0050] MIRUMM performs multimodal feature fusion after feature learning. Multimodal feature fusion can increase information richness and reduce data noise and interference, thereby improving the accuracy of intent recognition results. In order to solve the problem that all modalities are treated as equal and cannot interact deeply in the multimodal feature fusion process, the present invention designs a multimodal feature fusion module based on an attention-based bidirectional gated neural network. The overall structure of the multimodal feature fusion module includes self-attention, similarity learning, cross-modal attention and gated bidirectional fusion, such as Figure 3 shown.

[0051] After each modal feature enters the multimodal feature fusion module, self-attention is first used to extract the features of each modality. The self-attention mechanism can consider the global information of the input sequence instead of being limited to the local area. At the same time, using a self-attention module for each modality helps avoid information confusion, as shown in formula (2):

[0052]

[0053] Among them, E m is the characteristic of each mode, m∈v,a,t,Q=W Q E m , K=W K E m and V = W V E m denote the query matrix, key matrix and value matrix respectively, W Q , W K and W V represents the parameters learned during model training, is the dimension of the key matrix.

[0054] This paper designs a method for learning inter-modal feature similarity, aiming to alleviate the modality differences that occur in the process of modality loss prediction and improve the robustness of multi-modal joint representation, such as Figure 3 As shown. Since the text modality plays a major role in multimodal intent recognition, the present invention performs similarity learning on text features-audio features and text features-visual features, and makes the visual modality and audio modality close to the text modality by minimizing the similarity loss, thereby improving the quality of multimodal feature fusion.

[0055] Specifically, the present invention uses CMD distance metric as similarity loss. CMD distance is a metric used to measure the difference between two distributions. CMD distance evaluates the difference between them by comparing the central moment difference distance of the two distributions. Let X and Y be bounded random samples with probability distributions p and q on the interval [a, b] respectively. The definition of CMD distance metric is shown in formula (3):

[0056]

[0057] Where E(x) = (1 / |X|)∑ x∈X x is the empirical expectation vector of sample x, C k (x) = E((xE(x)) k ) is the vector of the central moments of all k-order samples of the x-coordinate, and the similarity loss is shown in formula (4):

[0058]

[0059] Among them, h1, h2 are the features extracted by self-attention, (h1, h2)∈(t,a),(t,v).

[0060] The present invention uses cross-modal attention to learn related features between different modalities. The main difference between cross-modal attention and self-attention is that cross-modal attention compares the elements between two different modal input sequences to calculate weights. Considering that the audio modality and the visual modality are not as important as the text modality, the present invention does not calculate the cross-modal features from audio to vision and from vision to audio. The present invention uses cross-modal attention to learn related features from text to vision, related features from vision to text, related features from text to audio, and related features from audio to text, as shown in formula (5):

[0061]

[0062] Among them, (h1, h2)∈(t, a), (t, v), (a, t), (v, t), Represents the parameters of the respective cross-modal attention learning.

[0063] The present invention uses bidirectional gating to fuse audio-related features and At the same time, the visual related features are integrated and To get audio features For example, the structure of bidirectional gated fusion is as follows Figure 4 As shown. Bidirectional gated fusion has advantages in processing time series data, especially for tasks that need to consider contextual relationships. The introduction of this mechanism can improve the expressiveness of the model and more effectively capture the key information in the sequence data. The final audio features and visual features are obtained through bidirectional gated fusion, as shown in formula (6) and formula (7):

[0064]

[0065]

[0066] in, Refers to the tensor product, G U Gate functions are used to control the flow of information. and Represents the audio features and visual features calculated by the hidden layer, and the gate function formula is:

[0067]

[0068] Among them, cat represents the concatenation operation, Conv represents the convolution layer, Sig represents the Sigmoid function of the tensor, and the gate function will first concatenate the two input parameters, then perform a convolution operation on the concatenated result, and finally obtain the final output through the Sigmoid function.

[0069] The present invention concatenates the feature vectors of all modalities to obtain a common joint representation H of multiple modalities. all , as shown in formula (9):

[0070]

[0071] Among them, || represents the splicing operation, H t is the text feature extracted from the self-attention in formula (2), is the result of gated fusion of formula (6), is the result of gated fusion of formula (7), H all It is the multimodal features after final fusion, which are used for a series of downstream tasks.

[0072] The model cannot perceive the lack of modality during training, but will be trained in a complete modality manner. Therefore, it is necessary to make the model aware of these modality losses during training. The present invention proposes a Transformer-based modality loss prompt module. The key idea of ​​the method of the present invention is to add different trainable vectors (i.e., prompts) to the keys and values ​​of the multi-head self-attention layer of each Transformer block according to the modality loss situation, and to alleviate the performance degradation caused by modality loss by adding different prompts and training the prompts.

[0073] In the multi-head self-attention layer, the original key, value, and query are calculated as K = H all , V=H all , Q=H all , where H all ∈R l×d is the input sequence of training samples, l is the length, d is the model dimension, and in order to learn the specific information of different modal missing situations, the present invention has 7 trainable d-dimensional vectors as key and value prompts, denoted as P m ∈R h×d, h is the sequence length, d is the model dimension, and m is the case of modality missing, which respectively represent the complete modality input and the cases of 6 different modal missing, such as Figure 5 As shown on the right. The calculation is shown in formula (10) and formula (11):

[0074] K′=concat(P m , K) = concat(P m , H all ) (10);

[0075] V′=concat(P m , V) = concat(P m , H all ) (11);

[0076] Among them, the new V′(K′)∈R (h+l)×d is used to calculate the multi-head self-attention; then, K′, V′ and Q with the modality missing prompts are sent to the multi-head attention mechanism in Transformer for calculation, such as Figure 5 shown.

[0077] The present invention inputs the features obtained by the multimodal feature fusion module into the Transformer codec module, and injects the modality missing prompt into K and V of the multi-head self-attention module according to the modality missing situation. The calculation process is shown in formula (12), formula (13) and formula (14):

[0078] H m =MultiHead(Q,K′,V′)=MultiHead(H all , [H all ||P m ],[H all ||P m ]) (12);

[0079] H m =Layernorm(H all +H m ) (13);

[0080]

[0081] Among them, W e 1 , W e 2 are weight matrices, are all biases, and ReLU represents the activation function. Similarly, the encoder output H m As the input of the decoder, the decoded output H dThe expression of can be expressed as formula (15), formula (16) and formula (17):

[0082] H d =MultiHead(H m , H m , H m ) (15);

[0083] H d =Layernorm(H d +H m ) (16);

[0084]

[0085] in, and are weight matrices, and are biases, and ReLU represents the activation function.

[0086] In order to classify the output, the present invention converts the output feature H d Input the classifier to predict intent and calculate the intent classification loss function. The classification loss calculation is shown in formula (18) and formula (19):

[0087]

[0088]

[0089] Among them, CrossEntropyLoss represents the cross loss function, W1 and W2 are weight matrices, b1 and b2 are biases, and ReLU represents the activation function.

[0090] When uncertain modalities are missing, the model may over-rely on the information of existing modalities, thereby making incorrect estimates of the data distribution of the missing modalities, which in turn leads to overfitting. The present invention designs a supervision module based on a pre-trained model to perform generalization learning. The goal of this module is to allow the model to learn the generalization ability of the pre-trained model trained with complete modalities, and the results obtained will be better than the model that simply fits the missing modal training data. Specifically, MIRUMM uses the pre-training loss in the classifier to shorten the distance between the pre-trained model and the main AGNN model. The output of the pre-trained model contains more information than the label. At the same time, when uncertain modalities occur, the semantic knowledge of the complete modality is passed to the model, thereby effectively improving the generalization ability of MIRUMM. Among them, the pre-trained model and the main AGNN model are two independent models with the same model structure and no shared parameters. The pre-training loss function is shown in formula (20):

[0091]

[0092] Among them, D KL represents the KL divergence, τ represents the temperature coefficient, which measures the difference between two probability distributions. is the output of the pre-trained model, is the output of the main AGNN model.

[0093] The MIRUMM model is a multi-task learning model that includes intent recognition, similarity learning, and pre-training guidance. Each task has a corresponding loss function. The MIRUMM model training requires combining the loss functions of all tasks together for training. The overall training loss is expressed as shown in formula (21):

[0094] L total =λ1L r +λ2L sim +λ3L pre (twenty one);

[0095] Among them, λ1, λ2, λ3 are the weights corresponding to each loss function, L r is the classification loss, L sim is the similarity loss, L pre is the pre-training loss.

[0096] In order to verify the performance of the MIRUMM model, the present invention has carried out sufficient experimental verification on two public data sets. The present invention first introduces the public data sets used in the experiment; secondly, introduces the experimental environment and parameter settings; then, introduces the five baseline models used in the comparative experiment; finally, introduces the details of the comparative experiment, ablation experiment and other related experiments, and conducts an in-depth analysis of the experimental results.

[0097] The present invention has conducted a large number of experiments on the public datasets MInRec and EMOTyDA. Both datasets are multimodal datasets including visual, textual and audio modalities. For the MInRec dataset, the original data of the dataset comes from a TV series. MInRec is currently the first benchmark dataset for real-world multimodal scene intent recognition, containing 2224 samples including textual, visual and audio modalities. MInRec contains 2 coarse-grained classifications and 20 fine-grained intent classifications, and the data is divided into training set, validation set and test set according to 3:1:1. Compared with MInRec, EMOTyDA is a large-scale dataset for dialogue behavior classification, and all data are annotated with 12 common dialogue behavior labels. Limited by the limitation of computing resources, the present invention selects 2500 representative samples based on the EMOTyDA dataset as the experimental dataset, and divides them into training set, validation set and test set according to the ratio of 3:1:1. In order to ensure fairness, all methods are experimented on this experimental dataset.

[0098] All experiments were conducted on a Linux server with an operating system version of Ubuntu 18.04. The server has an Intel(R) Xeon(R) Platinum 8255C, a 2.50GHz CPU, a GeForce RTX 3090 GPU, and 128g memory. The experiments were written in the PyTorch framework of the Python language, with versions 3.8 (ubuntu20.04) and 1.11.0 respectively. The experiments used Cuda to implement GPU parallel computing, and the Cuda version was 11.3. The parameters of the present invention are summarized in Table 1.

[0099] Table 1: Detailed parameter settings in all experiments.

[0100]

[0101] In order to verify the performance of the MIRUMM model proposed in this invention under the condition of uncertain modal missing, this invention needs to conduct experiments with the baseline model under different uncertain modal missing rates. Therefore, this invention selects 5 most advanced multimodal models for comparison. The introduction of the five baseline models is as follows:

[0102] (1) Mult: Mult processes non-aligned multimodal data in an end-to-end manner. It extends the ordinary Transformer to a cross-modal Transformer through a bidirectional cross-modal attention mechanism, which helps capture the interaction between different modalities in the latent space. Mult can effectively solve the multimodal misalignment problem caused by the variable sampling rates of different modalities and the long-term dependencies between cross-modal elements.

[0103] (2) MAG_B: The MAG_B model designs a multimodal adaptive gate MAG, which allows BERT to accept multimodal non-language data during fine-tuning. MAG can generate position shifts in the semantic space to adapt to audio and visual information. MAG_B can be flexibly configured between BERT layers to receive input from non-language modalities.

[0104] (3) MISA: MISA is able to learn complementary information to minimize redundancy and incorporate a diverse set of information. MISA adopts a multi-task learning framework, which projects different modalities into two different subspaces through the same encoder and different encoders respectively and learns them, thereby obtaining a multimodal representation with modality-invariant and modality-specific properties. On the one hand, MISA uses the same encoder to learn the common features of all modalities; on the other hand, MISA designs different encoders to capture the unique properties of each modality. To this end, the training objectives include four aspects: similarity loss, difference loss, reconstruction loss, and specific task loss.

[0105] (4) IF-MMIN: The IF-MMIN method addresses the challenges brought by the differences between different modalities by learning invariant features and using them to predict and fill in the missing modal data, especially when the modal data is incomplete or uncertain. During the IF-MMIN training process, IF-MMIN reads specific features and modality-invariant features, and learns a highly robust joint representation through the missing modality imagination module. A large number of experiments have demonstrated that this model can significantly improve the performance of multimodal emotion recognition.

[0106] (5) EMRFM: EMRFM is an effective multimodal representation and fusion method for intent recognition in real multimodal scenarios. EMRFM constructs modality-sharing and modality-specific encoders to learn the shared and specific feature representations of the modalities, and designs an adaptive multimodal feature fusion method based on attention-gated neural networks to eliminate noise features. Experiments show that the multimodal representation of the EMRFM model can well learn the shared and specific features of the modalities. The multimodal feature fusion of this model achieves adaptive fusion and effectively reduces possible noise interference.

[0107] As the evaluation index, experiments are conducted on the multi-task learning parameters λ1, λ2, λ3 of model training and the temperature coefficient τ of the pre-training module. Although the best hyperparameters can be obtained by grid searching all hyperparameters at the same time, the excessive hyperparameters of the present invention will increase the training time of the model. Therefore, when conducting the hyperparameter sensitivity analysis experiment, the present invention first analyzes the hyperparameters λ1, λ2, λ3 of multi-task learning, and then analyzes the pre-training module temperature coefficient τ when the multi-task learning parameters λ1, λ2, λ3 are fixed.

[0108] For the multi-task learning parameters λ1, λ2, and λ3, the present invention does not adjust the three hyperparameters of multi-task learning. In contrast, the present invention first randomly fixes two of the three hyperparameters, then adjusts the remaining hyperparameter, and finally finds the hyperparameter with the best performance. Although the approach of the present invention cannot obtain the best parameter combination, it can significantly reduce the hyperparameter optimization time. The parameter sensitivity experimental results of the multi-task learning parameters λ1, λ2, and λ3 are shown in Figure 2. Figure 6 As shown. The experimental results show that when the temperature coefficient is randomly fixed to 7, when λ1 is 1 and λ2 is 0.7, the F1 score of the model is as high as 70.37 when λ3 is set to 0.5. Therefore, the present invention selects the λ3 hyperparameter of pre-trained multi-task learning to be 0.5. Similarly, when the temperature coefficient is randomly fixed to 7, when λ1 is 1 and λ3 is 0.5, the F1 score of the model is as high as 70.41 when λ2 is set to 0.5; when λ2 is 0.5 and λ3 is 0.5, the F1 score of the model is as high as 71.09 when λ1 is set to 0.8.

[0109] For the pre-training temperature coefficient τ, when the parameters of λ1, λ2, and λ3 in multi-task learning are fixed to 0.8, 0.5, and 0.5, the effect of the value of the pre-training temperature coefficient τ on the F1 score of the model is as follows: Figure 7 As shown in the figure. The experimental results show that the F1 score of the MIRUMM model reaches its peak when the temperature coefficient τ is 3, 7, and 10. In particular, when the pre-training parameter temperature coefficient τ is 7, the F1 score of the MIRUMM model is as high as 71.09. Therefore, the final hyperparameter selections are λ1 of 0.8, λ2 of 0.5, λ3 of 0.5, and τ of 7.

[0110] In order to evaluate the performance of MIRUMM under the condition of uncertain modal missingness, the present invention has conducted a large number of experiments on two public datasets, MInRec and EMOTyDA. The experimental results show that the F1 scores of all models all show a downward trend with the increase of missing rate, as shown in Table 2. However, sometimes the accuracy rate will show an abnormal increase with the increase of missing rate. This is because the missing samples are mainly concentrated in a few categories. In order to improve the accuracy during the training process, the model tends to predict the majority category. However, since the F1 score considers both precision and recall, it may perform poorly on the minority category, resulting in a decrease in F1. The experimental results show that compared with other basic models, the MIRUMM proposed in the present invention has better performance in multimodal intent recognition under uncertain modal missingness.

[0111] On the public dataset MInRec, the present invention sets the missing rate of uncertain modalities to 0%, 10%, 20%, 30%, 40% and 50%, and tests the MIRUMM model and the baseline model proposed by the present invention. The experimental results show that the MIRUMM model outperforms other baseline models in terms of both accuracy and F1 score. The detailed results are shown in Table 2. Compared with other baseline models, the MIRUMM model has a minimum F1 score improvement of 0.25 when the missing rate is 20%; a maximum F1 score improvement of 1.94 when the missing rate is 50%; a minimum accuracy improvement of 0.91 when the missing rate is 20%; and a maximum accuracy improvement of 2.05 when the missing rate is 30%. It can be seen that the MIRUMM model performs better than the baseline model on the public dataset MInRec.

[0112] Table 2: Comparative experimental results of missing rates of different uncertain modes.

[0113]

[0114]

[0115] Similarly, on the EMOTyDA public data set, the present invention sets the missing rate of uncertain modalities to 0%, 10%, 20%, 30%, 40% and 50%, and tests the MIRUMM model and other baseline models proposed by the present invention. The experimental results show that when the missing rate of uncertain modalities is set to 0%, 10%, and 40%, the MIRUMM model outperforms other baseline models in terms of accuracy and F1 score. When the missing rate of uncertain modalities is set to 20%, the MIRUMM model is only 0.49 lower than the accuracy of the MAG_B model in terms of accuracy; when the missing rate of uncertain modalities is set to 30%, the MIRUMM model is only 0.61 lower than the MAG_B model in terms of accuracy; when the missing rate of uncertain modalities is set to 50%, the MIRUMM model is only 4.23 lower than the MAG_B model in terms of accuracy. Although the accuracy performance of the MIRUMM model is not as good as that of MAG_B under the above missing rates, the F1 score index values ​​of the MIRUMM model all exceed MAG_B. The F1 score takes into account the performance of the classification model in dealing with imbalanced categories and focusing on misclassification more comprehensively than the accuracy rate, and is especially useful in multi-classification problems. Therefore, the MIRUMM model has better overall performance.

[0116] In addition, when the missing rate is 0%, compared with the best baseline model, the MIRUMM model improves the F1 score by 1.21 and 0.63 on the MInRec and EMOTyDA datasets, and the accuracy by 0.4 and 0.57, respectively. This shows that the MIRUMM model also performs well when no modal missing occurs. It can be seen that the overall performance of the MIRUMM model proposed in the present invention is higher than other baseline models not only when uncertain modal missing occurs in the multimodal dataset, but also when no modal missing occurs.

[0117] In order to better verify the impact of different modules and modalities on the MIRUMM model, the present invention conducted an ablation experiment on the public dataset MIntRec. The experiment is divided into modal ablation experiment and module ablation experiment. The detailed results are shown in Table 3. When conducting modal ablation experiments, single-modal ablation experiments and bimodal ablation experiments are conducted respectively. When conducting single-modal ablation experiments, since only one modality, text, is used, similarity learning is not performed after cross-modal fusion. In addition, there is no uncertain modal loss in the single modality, and the single modality only conducts experiments with a loss rate of 0%. The bimodal ablation experiment is a pairwise combination of the three modalities of text T, vision V and audio A. The module ablation experiment removes the multimodal feature fusion module (Fusion), uses splicing instead after extracting modality-specific features, removes the pre-trained modality (Pre) and removes the modality loss prompt (Prompt) module. Similar to the modal ablation experiment, the present invention conducts uncertain modal loss experiments with loss rates ranging from 0% to 50% for ablation experiments of different modules. In summary, all ablation experiments are shown in Table 3, from top to bottom: (1) text unimodal experiment; (2) visual unimodal experiment; (3) audio unimodal experiment; (4) text and audio bimodal experiment; (5) text and visual bimodal experiment; (6) visual and audio bimodal experiment; (7) complete trimodal experiment; (8) remove multimodal feature fusion module; (9) remove pre-training module; (10) remove modality missing prompt mode. Experiments (1)-(7) are modality ablation experiments; experiments (7)-(10) are module ablation experiments. The experimental results are shown in Table 3.

[0118] Table 3: MIRUMM ablation experiments.

[0119]

[0120]

[0121] From the results of the modality ablation experiments (1)-(7) in Table 3, it can be seen that in multimodal intent recognition, the text modality is the most important, and the visual modality and audio modality can only play a supplementary role to the text modality, as shown in the modality ablation experiments (1)-(7) in Table 3. Specifically, the present invention finds, based on the content of the modality ablation experiment, that the performance of text plus visual and audio modalities in the single-modal case is improved, wherein the F1 score and accuracy of the text modality plus audio modality are 3.18 and 2.03 higher than those of the text modality plus audio modality, respectively. This is because audio features may be more resistant to some environmental noise than visual features. In addition, audio modality and text modality are similar in some aspects, and the speech in the audio can be converted into text form, and they are both a form of language expression. However, the performance of the audio modality plus the visual modality is greatly reduced, which indicates that the text modality is the key modality for multimodal intent recognition. One explanation for these results is that the text modality is easier to interpret than the visual and audio modalities because the user's intent can be directly conveyed through text, while the visual and audio modalities may require more complex feature extraction and processing steps. It can be seen that the text modality has a huge advantage in the multimodal intent recognition model, and the present invention can use the importance of the text modality to improve the model performance. In addition, the experiment once again proved the advantages of high correlation and complementarity between different modal data in multimodal intent recognition.

[0122] When the uncertain modality missing rate increases from 0% to 50%, the performance of the MIRUMM model decreases to varying degrees when any of the three modules, namely, the multimodal feature fusion module, pre-training module, and modality missing prompt module, is removed, as shown in the ablation experiments of modules (7)-(10) in Table 3. Therefore, the multimodal feature fusion module, pre-training module, and modality missing prompt module play a very important role in MIRUMM and can effectively solve the performance degradation problem caused by uncertain modality missing. Specifically, when the missing rate of uncertain modalities ranges from 0% to 50%, the F1 scores decrease by 1.99, 0.86, 1.47, 3.4, 2.72, and 1.84 respectively when the multimodal feature fusion module is removed; the F1 scores decrease by 0.86, 1.75, 1.61, 2.05, 2.15, and 0.24 respectively when the pre-trained model is removed; and the F1 scores decrease by 1.77, 2.26, 2.32, 2.17, 1.31, and 0.72 respectively when the modal prompt module is removed. When the missing rate is 0%, 30%, 40%, and 50%, the F1 score decreases the most when the multimodal feature fusion module is removed, which shows that the multimodal feature fusion module plays a dominant role in MIRUMM.

[0123] In order to study the changes in feature distribution after multimodal feature fusion, the present invention uses the T-SNE (t-distributed Stochastic Neighbor Embedding, T-SNE) toolkit to visualize features under different uncertain modal missing rates. T-SNE is a nonlinear dimensionality reduction technique used to map high-dimensional data to two-dimensional or three-dimensional space for visualization. It is mainly used to explore structures in high-dimensional data, discover clustering structures in data, and retain similarity relationships in visualization. Specifically, the present invention visualizes the 445 vectors finally learned by Transformer on the test set of the MInRec dataset, where red and green represent the coarse-grained classification categories of "expressing emotions or attitudes" and "achieving goals", respectively. The visualization results are shown as follows: Figure 8 As shown in (a)-(e). Most of the same category vectors are distributed together, but some outliers are incorrectly distributed in another category. In addition, as the missing ratio increases, the number of confusions at the junction of different category distributions increases. This shows that the performance of the MIRUMM model will be affected as the number of uncertain modal missing increases. The uncertain modal missing causes the MIRUMM model to lack sufficient features to distinguish different categories. However, the distribution of the features obtained after the overall multimodal feature fusion under the missing condition is similar to the feature distribution under the full modality condition. It can be seen that the model of the present invention still has good robustness under the condition of uncertain modal missing.

[0124] In order to analyze the performance of the MIRUMM model in classification at different granularities, the present invention conducted experiments on classification at different granularities on the MInRec dataset, and the experimental results are shown in Table 4. The coarse-grained classification of the MInRec dataset includes 2 categories, "expressing emotions or attitudes" and "achieving goals", and the fine-grained classification includes 20 categories. The experimental results in Table 4 show that the F1 scores and accuracy rates of coarse-grained classification and fine-grained classification both decrease with the increase of the missing rate of uncertain modalities. Through the changes in the F1 scores and accuracy rates of classification at different granularities, it can be found that when coarse-grained classification with fewer categories is performed, the overall performance of the MIRUMM model is greatly improved. In addition, with the increase of the missing rate of uncertain modalities, the decrease in the F1 score and accuracy rate of coarse-grained classification is smaller than that of fine-grained classification. Specifically, when the missing rate of uncertain modalities in the MInRec dataset increases from 0% to 50%, the F1 score and accuracy of coarse-grained classification decrease by 3.74 and 3.63, with an average decrease of 0.75 and 0.73 each time; the F1 score and accuracy of fine-grained classification decrease by 10.94 and 12.04, with an average decrease of 2.19 and 2.41 each time. This is because the categories are broader in coarse-grained classification, and for some samples with missing uncertain modalities, the model may still be able to make relatively accurate classifications through other available information. Compared with coarse-grained classification, multiple more specific and detailed subcategories in fine-grained classification cause confusion, making it difficult for the model to converge. It can be seen that MIRUMM performs well in classifications of different granularities and has strong generalization ability.

[0125] Table 4: Comparison of different particle size classifications.

[0126]

[0127] Embodiment 2:

[0128] This implementation provides a multimodal intent recognition system for uncertain modality loss, including:

[0129] The feature extraction unit is configured to: obtain multimodal data with uncertain modality missing, perform feature extraction of the multimodal data through an LSTM encoder, and obtain features of different modalities;

[0130] The feature fusion unit is configured to: minimize the similarity loss between text modality features and visual modality features and audio modality features, and fuse different modality features through an attention-based bidirectional gated multimodal feature fusion method;

[0131] The modality missing prompt unit is configured to: input the fused multimodal features into the Transformer model for learning, and add different attention level prompts to the keys and values ​​of the multi-head self-attention layer of each Transformer block according to the uncertain modality missing situation, so as to guide the Transformer model to pay attention to the missing modality;

[0132] The intent recognition unit is configured to use the features output by the Transformer model as the input of the classifier to obtain a multimodal intent recognition result.

[0133] The working methods of the various units of the system are the same as the specific details provided in Example 1 and will not be repeated here.

[0134] It is understandable that the above-mentioned units can be separately or all combined into one or several other units to constitute, or some of the units can be further divided into multiple smaller units in function to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the function of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the image processing device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0135] According to another embodiment of the present application, the system described in this embodiment can be constructed and the image processing method of the embodiment of the present application can be implemented by running a computer program (including program code) capable of executing the steps involved in the corresponding method described in Example 1 on a general computing device such as a computer including processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). The computer program can be recorded on, for example, a computer-readable recording medium, and loaded into the above-mentioned computing device through the computer-readable recording medium and run therein.

[0136] Embodiment 3:

[0137] This implementation provides an electronic device, which includes a processor, a communication interface, and a computer-readable storage medium, wherein the processor, the communication interface, and the computer-readable storage medium may be connected via a bus or other means.

[0138] Among them, the communication interface is used to receive and send data, the computer-readable storage medium can be stored in the memory of the electronic device, the computer-readable storage medium is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer-readable storage medium.

[0139] A processor (or CPU (Central Processing Unit)) is the computing core and control core of an electronic device, which is suitable for implementing one or more instructions, and specifically suitable for loading and executing one or more instructions to implement corresponding method flows or corresponding functions.

[0140] The processor is configured to perform the following process:

[0141] Obtain multimodal data with uncertain modal missing, extract features of multimodal data through LSTM encoder, and obtain features of different modalities;

[0142] Minimize the similarity loss between text modality features and visual modality features and audio modality features, and fuse different modality features through attention-based bidirectional gated multimodal feature fusion;

[0143] The fused multimodal features are input into the Transformer model for learning. According to the lack of uncertain modalities, different attention-level cues are added to the keys and values ​​of the multi-head self-attention layer of each Transformer block to guide the Transformer model to pay attention to the missing modalities.

[0144] The features output by the Transformer model are used as the input of the classifier to obtain multimodal intent recognition results.

[0145] The specific working methods of each process are the same as the specific details provided in Example 1 and will not be repeated here.

[0146] Embodiment 4:

[0147] This embodiment provides a computer-readable storage medium (Memory), which is a memory device in an electronic device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the electronic device and, of course, extended storage media supported by the electronic device. The computer-readable storage medium provides a storage space that stores the processing system of the electronic device.

[0148] In addition, the storage space also stores one or more instructions suitable for being loaded and executed by the processor, and these instructions may be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here may be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage; optionally, it may also be at least one computer-readable storage medium located away from the aforementioned processor.

[0149] In one embodiment, the computer-readable storage medium stores one or more instructions; the processor loads and executes the one or more instructions stored in the computer-readable storage medium to implement the following process:

[0150] Obtain multimodal data with uncertain modal missing, extract features of multimodal data through LSTM encoder, and obtain features of different modalities;

[0151] Minimize the similarity loss between text modality features and visual modality features and audio modality features, and fuse different modality features through attention-based bidirectional gated multimodal feature fusion;

[0152] The fused multimodal features are input into the Transformer model for learning. According to the lack of uncertain modalities, different attention-level cues are added to the keys and values ​​of the multi-head self-attention layer of each Transformer block to guide the Transformer model to pay attention to the missing modalities.

[0153] The features output by the Transformer model are used as the input of the classifier to obtain multimodal intent recognition results.

[0154] The specific working methods of each process are the same as the specific details provided in Example 1 and will not be repeated here.

[0155] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multimodal intent recognition method for uncertain modality loss, characterized in that: The process includes: Obtain multimodal data with uncertain modal missing, extract features of multimodal data through LSTM encoder, and obtain features of different modalities; Minimize the similarity loss between text modality features and visual modality features and audio modality features, and fuse different modality features through attention-based bidirectional gated multimodal feature fusion, including: Use cross-modal attention to learn text-to-vision related features, vision-to-text related features, text-to-audio related features, and audio-to-text related features, as shown in the following formula: Among them, (h1, h2)∈(t, a), (t, v), (a, t), (v, t), denote the parameters of respective cross-modal attention learning; Use bidirectional gating to fuse text-to-vision related features and vision-to-text related features as visual fusion features, and use bidirectional gating to fuse text-to-audio related features and audio-to-text related features as audio fusion features; Using bidirectional gating to fuse text-to-vision related features and vision-to-text related features as vision fusion features, including: performing a tensor product operation on the text-to-vision related features and the gate function, and adding the operation result to the vision-to-text related features as the vision fusion feature; Using bidirectional gating to fuse text-to-audio related features and audio-to-text related features as audio fusion features, including: performing a tensor product operation on the text-to-audio related features and the gate function, and adding the operation result to the audio-to-text related features as the audio fusion feature; The gate function is: cat represents the concatenation operation, Conv represents the convolution layer, and Sig represents the Sigmoid function of the tensor; The audio fusion features, visual fusion features and text modality features are spliced ​​to complete the fusion of different modality features; The fused multimodal features are input into the Transformer model for learning. According to the lack of uncertain modalities, different attention-level cues are added to the keys and values ​​of the multi-head self-attention layer of each Transformer block to guide the Transformer model to pay attention to the missing modalities. The features output by the Transformer model are used as the input of the classifier to obtain multimodal intent recognition results.

2. The multimodal intention recognition method for uncertain modality loss according to claim 1, characterized in that: The CMD distance is used to measure the similarity loss between text modality features and visual modality features and audio modality features.

3. The multimodal intention recognition method for uncertain modality loss according to claim 1, characterized in that: Depending on the lack of uncertain modalities, different attention-level cues are added to the key K and value V of the multi-head self-attention layer of each Transformer block to guide the Transformer model to pay attention to the missing modality. The output of the encoder includes: H m =MultiHead(Q,K′,V′)=MultiHead(H all ,[H all ||P m ],[H all ||P m ]); H m =Layernorm(H all +H m ); The encoder output H m As the input of the decoder, the output of the decoder is: H d =MultiHead(H m ,H m ,H m ); H d =Layernorm(H d +H m ); in, and are weight matrices, and are all deviations, ReLU represents the activation function, H all is the joint representation of multi-modal features, P m is a trainable d-dimensional vector used as a prompt for keys and values, P m ∈R h×d , h is the sequence length, d is the model dimension, m is the case of modality missing, K ′ To add the attention level prompt key, V ′ A value to add an attention level hint to.

4. The multimodal intention recognition method for uncertain modality loss as claimed in claim 3, characterized in that: Add the attention level prompt key K ′ ,include: K′=concat(P m ,K)=concat(P m ,H all ); Add the value V of the attention level hint ′ ,include: V′=concat(P m ,V)=concat(P m ,H all ); Among them, concat is a concatenation operation, K and V are the key and value without adding attention-level prompts.

5. The multimodal intention recognition method for uncertain modality loss according to claim 1, characterized in that: The pre-training loss is used to shorten the distance between the pre-training model and the main recognition model. The input of the pre-training model is the same as the input of the main recognition model. The pre-training model and the main recognition model are two independent models with the same model structure and do not share parameters. The weighted sum of the classification loss, similarity loss and pre-training loss is used as the overall training loss.

6. A multimodal intention recognition system for uncertain modality loss, characterized in that: include: The feature extraction unit is configured to: obtain multimodal data with uncertain modality missing, perform feature extraction of the multimodal data through an LSTM encoder, and obtain features of different modalities; The feature fusion unit is configured to minimize the similarity loss between text modality features and visual modality features and audio modality features, and fuse different modality features through an attention-based bidirectional gated multimodal feature fusion method, including: Use cross-modal attention to learn text-to-vision related features, vision-to-text related features, text-to-audio related features, and audio-to-text related features, as shown in the following formula: Among them, (h1, h2)∈(t, a), (t, v), (a, t), (v, t), denote the parameters of respective cross-modal attention learning; Using bidirectional gating to fuse text-to-vision related features and vision-to-text related features as visual fusion features, using bidirectional gating to fuse text-to-audio related features and audio-to-text related features as audio fusion features; using bidirectional gating to fuse text-to-vision related features and vision-to-text related features as visual fusion features, including: performing a tensor product operation on the text-to-vision related features and the gate function, and adding the operation result to the vision-to-text related features as the visual fusion feature; Using bidirectional gating to fuse text-to-audio related features and audio-to-text related features as audio fusion features, including: performing a tensor product operation on the text-to-audio related features and the gate function, and adding the operation result to the audio-to-text related features as the audio fusion feature; The gate function is: cat represents the concatenation operation, Conv represents the convolution layer, and Sig represents the Sigmoid function of the tensor; The audio fusion features, visual fusion features and text modality features are spliced ​​to complete the fusion of different modality features; The modality missing prompt unit is configured to: input the fused multimodal features into the Transformer model for learning, and add different attention level prompts to the keys and values ​​of the multi-head self-attention layer of each Transformer block according to the uncertain modality missing situation, so as to guide the Transformer model to pay attention to the missing modality; The intent recognition unit is configured to use the features output by the Transformer model as the input of the classifier to obtain a multimodal intent recognition result.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which is suitable for being loaded by a processor and executing the multimodal intention recognition method for uncertain modality loss as described in any one of claims 1-5.

8. A computer device, characterized in that: include: A processor and a computer readable storage medium; a processor adapted to execute a computer program; A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the multimodal intent recognition method for uncertain modality loss is implemented as described in any one of claims 1-5.

Images