Multimodal intention recognition method, system, device, and medium under uncertain modality

Abstract

The present invention belongs to the field of data processing technology, and specifically relates to a multimodal intent recognition method, system, device, and medium under uncertain modalities. In order to cope with the uncertainty problem of text modality quality, text features and common features are subjected to multi-head attention processing to generate dynamic text weights and act on the text modality, thereby dynamically optimizing the text modality; in order to solve the problem of insufficient information caused by uncertain modalities, the cosine similarity function is used to calculate the similarity between the modality with the highest priority and the corresponding modality in a full-modal sample data set, and similar full-modal samples are obtained and spliced ​​to form target enhancement features, and then the target enhancement features and fusion features are deeply associated and learned through cross-attention processing, thereby enhancing the quality of the fusion features.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and specifically relates to a multimodal intention recognition method, system, device, and medium under uncertain modalities. Background Art

[0002] Intention is the driving force behind human behavior, reflecting people's goals and needs in specific situations. Accurately identifying people's intentions has broad application prospects in public security management, human-computer interaction, medical assistance, and e-commerce. Multimodal intent recognition is a study of identifying user intentions through multimodal data related to users (text, images, and audio, etc.). In practical applications, due to factors such as the dynamic nature of the environment, multimodal data collection is incomplete or lost, resulting in modal uncertainty problems that often occur in real applications, that is, some modalities are sometimes absent and sometimes not.

[0003] In the field of multimodal intent recognition, some researchers have also studied the missing modality problem and have achieved certain results. For example, the method based on graph structure text propagation embedding performs similarity completion on the missing text modality; the pre-trained text-to-audio model is used to transcribe text data into audio data, thereby completing the audio modality. However, the following problems still exist:

[0004] (1) Existing multimodal intent recognition models usually perform fixed completion on the missing modality when it is known that a certain modality is missing. However, in a modality uncertainty environment, existing models cannot flexibly cope with the uncertainty of the modality, resulting in model failure.

[0005] (2) Existing multimodal intent recognition models usually unconditionally emphasize the importance of the text modality and often ignore the uncertainty of the quality of the text modality, which leads to a decline in model performance.

[0006] (3) Existing multimodal intent recognition models only focus on learning the multimodal features of the current sample and are unable to cope with the problem of insufficient information caused by uncertain modalities, resulting in poor results after multimodal fusion.

[0007] In short, existing research works all perform fixed completion operations on the missing modalities when a certain model is known to be missing, which cannot deal with the modality uncertainty problem well. Therefore, solving the problem of multimodal intent recognition under uncertain modalities has become a daunting challenge. Summary of the invention

[0008] The present invention provides a method, system, device and medium for multimodal intention recognition under uncertain modalities.

[0009] The technical solution of the present invention is as follows:

[0010] The present invention provides a multimodal intention recognition method under uncertain modality, comprising the following steps:

[0011] S1: Acquire uncertain modal data, where the uncertain modal data is text modality and / or audio modality and / or image modality, perform feature extraction, and obtain text features and / or audio features and / or image features respectively;

[0012] S2: The text features, audio features, and image features are processed by full connection, ReLU activation function, and discard in sequence to obtain high-level features of the text modality, high-level features of the audio modality, and high-level features of the image modality. The high-level features of the text modality, the high-level features of the audio modality, and the high-level features of the image modality are concatenated to generate common features.

[0013] After the text features and common features are concatenated, they are processed by multi-head attention to obtain the first relationship matrix. The Tanh activation function is used to generate the weights of the text features according to the first relationship matrix.

[0014] The weight of the text feature is matrix-multiplied with the text feature to obtain the optimized text feature;

[0015] S3: Audio features, image features, and optimized text features are fused through multimodal features to obtain fused features;

[0016] S4: obtaining a full-modal sample data set, determining the modal priority based on the uncertain modal data, and using the cosine similarity function to calculate the similarity between the modality with the highest priority and the corresponding modality in the full-modal sample data set to obtain a similar full-modal sample set;

[0017] The omnimodal features in similar omnimodal sample sets are concatenated with semantic identifiers to obtain target enhanced features;

[0018] The target enhanced features and fused features are processed by cross attention to obtain enhanced fused features;

[0019] S5: The enhanced fusion features are classified and processed to obtain the predicted intent results.

[0020] In step S2, the Tanh activation function is used to generate the weights of the text features according to the first relationship matrix, specifically:

[0021] After the first relationship matrix is ​​processed by residual connection, layer normalization and ReLU activation function are performed in sequence to obtain the second relationship matrix;

[0022] After the second relationship matrix is ​​fully connected, it is processed by the Tanh activation function and numerically scaled in turn to generate the weights of the text features.

[0023] In step S2, the weight of the text feature is matrix-multiplied with the text feature to obtain the optimized text feature, which is according to the formula: ,accomplish;

[0024] In the formula, Represents optimized text features; Represents the weight of text features; Represents text features; Indicates the existence status of the text mode, with a value of 0 or 1.

[0025] In step S4, the omnimodal features in the similar omnimodal sample set are concatenated with the semantic identifier to obtain the target enhanced features, specifically:

[0026] Based on similar omnimodal sample sets, the text features, audio features, image features and semantic identifiers in each similar omnimodal sample are concatenated to obtain similar enhanced features of a single sample.

[0027] Similar enhanced features of multiple single samples are concatenated to obtain the target enhanced features.

[0028] In step S3, the audio features, image features, and optimized text features are fused through multimodal features to obtain fused features, specifically:

[0029] The audio features, image features, and optimized text features are processed by self-attention to obtain audio attention features, image attention features, and text attention features;

[0030] The text attention features and the image attention features are processed cross-modally to obtain text-to-image related features and image-to-text related features;

[0031] The text attention features and the audio attention features are processed cross-modally to obtain text-to-audio related features and audio-to-text related features;

[0032] The text-to-image related features and the image-to-text related features are combined to obtain the target image features; the text-to-audio related features and the audio-to-text related features are combined to obtain the target audio features;

[0033] The text attention features, target image features and target audio features are concatenated to obtain fused features.

[0034] The text-to-image related features and the image-to-text related features are combined to obtain target image features; the text-to-audio related features and the audio-to-text related features are combined to obtain target audio features, specifically:

[0035] The text-to-image related features and the image-to-text related features are sequentially concatenated, convolved, and processed with the Sigmoid activation function, and then added to the image-to-text related features to obtain the target image features;

[0036] The text-to-audio related features and the audio-to-text related features are concatenated, convolved, and processed with the Sigmoid activation function in sequence, and then added to the audio-to-text related features to obtain the target audio features.

[0037] In step S4, a full-modal sample data set is obtained, and based on the uncertain modal data, the modal priority is determined, and the similarity between the modality with the highest priority and the corresponding modality in the full-modal sample data set is calculated using the cosine similarity function to obtain a similar full-modal sample set. The operation is as follows:

[0038] If the acquired uncertain modal data has a text modality, the text modality is the modality with the highest priority. The cosine similarity function is used to calculate the similarity between the text modality and the text modality in the full modality sample data to obtain a similar full modality sample set.

[0039] If the acquired uncertain modality data does not have a text modality but has an image modality, the image modality is the modality with the highest priority, and the cosine similarity function is used to calculate the similarity between the image modality and the image modality in the full modality sample data to obtain a similar full modality sample set;

[0040] If the acquired uncertain modal data only contains the audio modality, the audio modality is the modality with the highest priority. The cosine similarity function is used to calculate the similarity between the audio modality and the audio modality in the full modality sample data to obtain a similar full modality sample set.

[0041] The present invention also provides a multimodal intention recognition system under uncertain modality, comprising:

[0042] Feature extraction module: used to obtain uncertain modal data, the uncertain modal data is text modality and / or audio modality and / or image modality, perform feature extraction, and obtain text features and / or audio features and / or image features respectively;

[0043] Text adaptive optimization module: used to process text features, audio features, and image features through full connection processing, ReLU activation function processing, and discard processing in sequence to obtain high-level features of text modality, high-level features of audio modality, and high-level features of image modality, and then splice the high-level features of text modality, audio modality, and image modality to generate common features;

[0044] After the text features and common features are concatenated, they are processed by multi-head attention to obtain the first relationship matrix. The Tanh activation function is used to generate the weights of the text features according to the first relationship matrix.

[0045] The weight of the text feature is matrix-multiplied with the text feature to obtain the optimized text feature;

[0046] Multimodal fusion module: used to fuse audio features, image features, and optimized text features through multimodal features to obtain fused features;

[0047] Similar modality enhancement module: used to obtain a full-modality sample data set, determine the modality priority based on the uncertain modal data, and use the cosine similarity function to calculate the similarity between the modality with the highest priority and the corresponding modality in the full-modality sample data set to obtain a similar full-modality sample set;

[0048] The omnimodal features in similar omnimodal sample sets are concatenated with semantic identifiers to obtain target enhanced features;

[0049] The target enhanced features and fused features are processed by cross attention to obtain enhanced fused features;

[0050] Intent recognition module: used to enhance the fusion features through classification processing to obtain the predicted intent results.

[0051] The present invention also provides a multimodal intention recognition device under uncertain modality, comprising a processor and a memory, wherein the processor implements the multimodal intention recognition method under uncertain modality when executing a computer program stored in the memory.

[0052] The present invention also provides a medium for storing a computer program, wherein the computer program, when executed by a processor, implements the multimodal intention recognition method under uncertain modalities.

[0053] Beneficial Effects

[0054] The present invention provides an intent recognition method that can accurately predict multimodal intents under uncertain modalities. On the one hand, in order to cope with the uncertainty of text modality quality, multi-head attention processing is performed on text features and common features to generate dynamic text weights and act on text modalities, thereby dynamically optimizing text modalities; on the other hand, in order to solve the problem of insufficient information caused by uncertain modalities, the cosine similarity function is used to calculate the similarity between the modality with the highest priority and the corresponding modality in the full modality sample data set, and similar full modality samples are obtained, and spliced ​​to form target enhancement features, and then the target enhancement features and fusion features are deeply associated with each other through cross-attention processing, thereby enhancing the quality of the fusion features. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1This is a comparison of the accuracy of the proposed method TAOSME with the MISA, EMRFM and SDIF-DA baseline models in binary classification.

[0056] Figure 2 This is the F1 score comparison of the proposed method TAOSME with the MISA, EMRFM and SDIF-DA baseline models in binary classification.

[0057] Figure 3 This is a comparison of the accuracy of the proposed method TAOSME with the MISA, EMRFM and SDIF-DA baseline models in twenty categories.

[0058] Figure 4 The F1 score comparison of the proposed method TAOSME with the baseline models MISA, EMRFM and SDIF-DA in twenty categories.

[0059] Figure 5 The changing trends of accuracy and F1 scores for different similar numbers of full-modal samples under a missing rate of 30%. DETAILED DESCRIPTION

[0060] The following examples are intended to illustrate the present invention rather than to further limit the present invention.

[0061] The present invention provides a multimodal intention recognition method under uncertain modality, comprising the following steps:

[0062] S1: Acquire uncertain modal data, where the uncertain modal data is text modality and / or audio modality and / or image modality, perform feature extraction, and obtain text features and / or audio features and / or image features respectively.

[0063] Among them, uncertain modal data includes seven situations, namely: text modality, audio modality and image modality all exist; only text modality and audio modality exist; only text modality and image modality exist; only audio modality and image modality exist; only text modality exists; only audio modality exists; only image modality exists.

[0064] The specific definitions are as follows:

[0065] The present invention provides multimodal intent recognition data including three modalities: , where the symbol * has two values, 0 and 1, 0 represents that the modal data does not exist, and 1 represents that the modal data exists. Represents possible multimodal data, Indicates text mode. Indicates the audio mode, Indicates image mode. To represent the mode Does not exist, use To represent the mode Existence, among which For example, assuming that the text modality exists, the audio modality does not exist, and the image modality does not exist, the multimodal intent recognition data can be represented as: .

[0066] Regarding feature extraction, specifically:

[0067] The text modality is processed by multi-head attention to obtain text features, which are expressed as follows:

[0068] ;

[0069] In the formula, represents text features, and , , Respectively represent the length and dimension of text features; A query vector representing the text modality; A key vector representing the text modality; A vector of values ​​representing the text modality.

[0070] The audio modality is processed by multi-head attention to obtain audio features, which are expressed as follows:

[0071] ;

[0072] In the formula, represents the audio features, and , , Respectively represent the length and dimension of the audio feature; A query vector representing the audio modality; A key vector representing the audio modality; A vector of values ​​representing the audio modes.

[0073] The image modality is processed by multi-head attention to obtain image features, which are expressed as follows:

[0074] ;

[0075] In the formula, represents the image features, and , , Respectively represent the length and dimension of image features; A query vector representing the image modality; A key vector representing the image modality; A vector of values ​​representing the modality of the image.

[0076] The present invention adopts multi-head attention to extract features and simultaneously focuses on information from different semantic subspaces, thereby providing richer and more diverse feature representation capabilities and better capturing the correlation and detail information in text modalities, audio modalities, and image modalities.

[0077] S2: The text features, audio features, and image features are processed by full connection, ReLU activation function, and discard in sequence to obtain high-level features of the text modality, high-level features of the audio modality, and high-level features of the image modality. The high-level features of the text modality, the high-level features of the audio modality, and the high-level features of the image modality are concatenated to generate common features.

[0078] After the text features and common features are concatenated, they are processed by multi-head attention to obtain the first relationship matrix. The Tanh activation function is used to generate the weights of the text features according to the first relationship matrix.

[0079] The weight of the text feature is matrix-multiplied with the text feature to obtain the optimized text feature.

[0080] Existing multimodal intent recognition methods usually place the text modality in a dominant position in the multimodal fusion process. However, when the text modality does not exist, a higher weight is still used for multimodal fusion on the text modality. Such a unified fusion method ignores the uncertainty of the quality of the text modality between different samples, resulting in inaccurate intent recognition results.

[0081] The present invention avoids this problem by learning and capturing the common features between different modalities.

[0082] First, the text features, audio features, and image features are processed by full connection, ReLU activation function, and discard in turn to obtain high-level features of text modality, high-level features of audio modality, and high-level features of image modality. They are expressed as follows:

[0083] ;

[0084] In the formula, High-level features representing text modality; Represents the fully connected processing of text features; is the weight matrix of text features, is the text feature, is the bias vector of text features; ReLU activation function processing; Indicates discard processing.

[0085] ;

[0086] In the formula, Represent high-level features of audio modalities; Represents fully connected processing of audio features; is the weight matrix of audio features, is the audio feature, is the bias vector of the audio feature.

[0087] ;

[0088] In the formula, Represent high-level features of image modalities; Represents the fully connected processing of image features; is the weight matrix of image features, is the image feature, is the bias vector of the image feature.

[0089] Secondly, the high-level features of the text modality, the high-level features of the audio modality, and the high-level features of the image modality are concatenated to generate common features. They are expressed as follows:

[0090] ;

[0091] In the formula, Indicates common characteristics; Represents a splicing operation; , , They represent the high-level features of text modality, high-level features of audio modality, and high-level features of image modality respectively.

[0092] The common features not only contain the specific information of different modalities themselves, but also the interactive information between different modalities.

[0093] Then, the text features are concatenated with the common features and processed by multi-head attention to obtain the first relationship matrix. It is expressed as follows:

[0094] ;

[0095] In the formula, It represents the features after the text features and common features are combined; Indicates common characteristics; is a text feature.

[0096] ;

[0097] In the formula, represents the first relationship matrix; , , Respectively The query vector, key vector, and value vector of .

[0098] Afterwards, the Tanh activation function is used to generate the weights of the text features according to the first relationship matrix, specifically:

[0099] After the first relationship matrix is ​​processed by residual connection, layer normalization and ReLU activation function are performed in sequence to obtain the second relationship matrix;

[0100] After the second relationship matrix is ​​fully connected, it is processed by Tanh activation function and numerical scaling in turn to generate the weight of text features. It is expressed as follows:

[0101] ;

[0102] In the formula, represents the second relationship matrix; represents the first relationship matrix; It represents the features after the text features and common features are combined; Representation layer normalization.

[0103] ;

[0104] In the formula, represents the third relationship matrix; Represents the second relationship matrix Perform full connection operation; Indicates Tanh activation function processing.

[0105] ;

[0106] In the formula, Represents the weight of text features; Indicates that Scaled to a value between 0 and 1.

[0107] Finally, the weight of the text feature is matrix multiplied with the text feature to obtain the optimized text feature, which is according to the formula: ,accomplish;

[0108] In the formula, Represents optimized text features; Represents the weight of text features; Represents text features; Indicates the existence status of the text mode, with a value of 0 or 1.

[0109] The present invention applies the weight of text features to the text mode and dynamically adjusts the dominant text mode. Specifically, when the text mode does not exist, the present invention defines the text mode as a state that needs to be adjusted, and the formula is expressed as: ; When text modal exists, define the text modal as not requiring adjustment.

[0110] S3: Audio features, image features, and optimized text features are fused through multimodal features to obtain fused features.

[0111] Specifically:

[0112] The audio features, image features, and optimized text features are processed by self-attention to obtain audio attention features, image attention features, and text attention features;

[0113] The text attention features and the image attention features are processed cross-modally to obtain text-to-image related features and image-to-text related features;

[0114] The text attention features and the audio attention features are processed cross-modally to obtain text-to-audio related features and audio-to-text related features;

[0115] The text-to-image related features and the image-to-text related features are combined to obtain the target image features; the text-to-audio related features and the audio-to-text related features are combined to obtain the target audio features;

[0116] The text attention features, target image features and target audio features are concatenated to obtain fused features.

[0117] The present invention uses cross-modal attention processing to capture the relevant knowledge between the modalities and achieve the fusion of the two modal information. Considering that the audio modality and the image modality are not as important as the text modality, the present invention does not calculate the cross-modal features of audio to image and image to audio. Therefore, cross-modal attention is used to learn the relevant features of text to image. ; Image to text related features ; Text to audio related features and audio-to-text related features .

[0118] Furthermore, the text-to-image related features and the image-to-text related features are combined to obtain target image features; the text-to-audio related features and the audio-to-text related features are combined to obtain target audio features, specifically:

[0119] The text-to-image related features and the image-to-text related features are sequentially concatenated, convolved, and processed with the Sigmoid activation function, and then added to the image-to-text related features to obtain the target image features; they are expressed as follows:

[0120] ;

[0121] In the formula, Represents the target image features; Represents text-to-image related features; Represents image-to-text related features; Indicates splicing; represents convolution; Indicates Sigmoid activation function processing.

[0122] The text-to-audio related features and the audio-to-text related features are sequentially concatenated, convolved, and processed with the Sigmoid activation function, and then added to the audio-to-text related features to obtain the target audio features. It is expressed as follows:

[0123] ;

[0124] In the formula, represents the target audio feature; Represents text-to-audio related features; Represents audio-to-text related features.

[0125] S4: Obtain a full-modal sample data set, determine the modal priority based on the uncertain modal data, and use the cosine similarity function to calculate the similarity between the modality with the highest priority and the corresponding modality in the full-modal sample data set to obtain a similar full-modal sample set. The operation is as follows:

[0126] If the acquired uncertain modal data has a text modality, the text modality is the modality with the highest priority. The cosine similarity function is used to calculate the similarity between the text modality and the text modality in the full modality sample data to obtain a similar full modality sample set.

[0127] If the acquired uncertain modality data does not have a text modality but has an image modality, the image modality is the modality with the highest priority, and the cosine similarity function is used to calculate the similarity between the image modality and the image modality in the full modality sample data to obtain a similar full modality sample set;

[0128] If the acquired uncertain modal data only has the audio modality, the audio modality is the modality with the highest priority, and the cosine similarity function is used to calculate the similarity between the audio modality and the audio modality in the full modality sample data to obtain a similar full modality sample set;

[0129] The omnimodal features in similar omnimodal sample sets are concatenated with semantic identifiers to obtain target enhanced features;

[0130] The target enhanced features and fused features are processed by cross attention to obtain enhanced fused features.

[0131] In a modal uncertainty environment, it is often difficult to fully explore the associations and specific features between the modalities, which often leads to the loss of important information, affects the quality of the final multimodal fusion, and thus reduces the accuracy of intent recognition.

[0132] Although some modalities are unavailable, there are still cases where all three modalities are complete. In order to effectively complete similar modality enhancement learning and thus enhance the quality of fusion features, it is necessary to obtain a full-modality sample dataset and retrieve highly relevant samples from it based on similarity.

[0133] For the acquired uncertain modal data, when the text modality exists, regardless of whether the image modality and the audio modality exist or not, the text modality is taken as the modality with the highest priority, and then similar full-modal sample data is sought; when the text modality does not exist, but the image modality exists, the image modality is taken as the modality with the highest priority, and then similar full-modal sample data is sought; when both the text modality and the image modality do not exist, the audio modality is taken as the modality with the highest priority, and then similar full-modal sample data is sought.

[0134] Regarding the cosine similarity function, it is expressed as follows:

[0135] ;

[0136] In the formula, Indicates similarity; Indicates that the current sample has the highest priority modal feature; Indicates the full modality sample data set and The corresponding modal features; Indicates the number of feature elements.

[0137] The obtained similar full-modal sample set can be expressed as: ;

[0138] In the formula, Represents a similar set of full-modal samples; Represents the text features in a single sample; Represents image features in a single sample; Represents audio features in a single sample; is the label corresponding to the sample, indicating the semantic identification information; Indicates the number of similar full-modal samples.

[0139] Furthermore, the omnimodal features in similar omnimodal sample sets are concatenated with semantic identifiers to obtain target enhanced features, specifically:

[0140] Based on similar omnimodal sample sets, the text features, audio features, image features and semantic identifiers in each similar omnimodal sample are concatenated to obtain similar enhanced features of a single sample.

[0141] Similar enhanced features of multiple single samples are concatenated to obtain the target enhanced features.

[0142] In addition, the present invention uses cross-attention to perform deep association learning on the target enhanced features and the fused features, aiming to utilize the deep interaction between the target enhanced features and the fused features to enhance the fused features of the current sample. First, the target enhanced features are used as the key vector and the value vector, and the fused features of the current sample are used as the query vector. The cross-attention mechanism is used to calculate the correlation weight between the query vector and the key vector to dynamically capture the semantic association between the two. Then, the correlation weight is used to perform weighted aggregation on the value vector to generate enhanced fused features containing contextual information. This interaction not only retains the key information of the current sample, but also introduces supplementary context from similar samples, thereby effectively improving the quality and expressiveness of the fused features of the current sample.

[0143] S5: The enhanced fusion features are classified and processed to obtain the predicted intent results.

[0144] It is expressed as follows:

[0145] ;

[0146] In the formula, Indicates the intended outcome of the prediction; represents the fully connected processing of enhanced fusion features; and Represents the weight matrix of enhanced fusion features; and Represents the bias of enhancing fusion features; ReLU activation function processing.

[0147] The intention recognition method provided by the present invention can accurately predict multimodal intentions under uncertain modalities. On the one hand, in order to cope with the uncertainty of text modality quality, multi-head attention processing is performed on text features and common features to generate dynamic text weights and act on text modalities, thereby dynamically optimizing text modalities; on the other hand, in order to solve the problem of insufficient information caused by uncertain modalities, the cosine similarity function is used to calculate the similarity between the modality with the highest priority and the corresponding modality in the full modality sample data set, and similar full modality samples are obtained and spliced ​​to form target enhancement features, and then the target enhancement features and fusion features are deeply associated and learned through cross-attention processing, thereby enhancing the quality of fusion features.

[0148] The present invention also provides a multimodal intention recognition system under uncertain modality, comprising:

[0149] Feature extraction module: used to obtain uncertain modal data, the uncertain modal data is text modality and / or audio modality and / or image modality, perform feature extraction, and obtain text features and / or audio features and / or image features respectively;

[0150] Text adaptive optimization module: used to process text features, audio features, and image features through full connection processing, ReLU activation function processing, and discard processing in sequence to obtain high-level features of text modality, high-level features of audio modality, and high-level features of image modality, and then splice the high-level features of text modality, audio modality, and image modality to generate common features;

[0151] After the text features and common features are concatenated, they are processed by multi-head attention to obtain the first relationship matrix. The Tanh activation function is used to generate the weights of the text features according to the first relationship matrix.

[0152] The weight of the text feature is matrix-multiplied with the text feature to obtain the optimized text feature;

[0153] Multimodal fusion module: used to fuse audio features, image features, and optimized text features through multimodal features to obtain fused features;

[0154] Similar modality enhancement module: used to obtain a full-modality sample data set, determine the modality priority based on the uncertain modal data, and use the cosine similarity function to calculate the similarity between the modality with the highest priority and the corresponding modality in the full-modality sample data set to obtain a similar full-modality sample set;

[0155] The omnimodal features in similar omnimodal sample sets are concatenated with semantic identifiers to obtain target enhanced features;

[0156] The target enhanced features and fused features are processed by cross attention to obtain enhanced fused features;

[0157] Intent recognition module: used to enhance the fusion features through classification processing to obtain the predicted intent results.

[0158] The present invention also provides a multimodal intention recognition device under uncertain modality, comprising a processor and a memory, wherein the processor implements the multimodal intention recognition method under uncertain modality when executing a computer program stored in the memory.

[0159] The present invention also provides a medium for storing a computer program, wherein the computer program, when executed by a processor, implements the multimodal intention recognition method under uncertain modalities.

[0160] Experimental results and analysis

[0161] In order to verify the effectiveness of the proposed method (referred to as TAOSME), sufficient experimental verification was carried out on two public datasets. First, the public datasets used in the experiment were introduced; second, the experimental environment and parameter settings were introduced; then, the five baseline models used in the comparative experiment were introduced; finally, the comparative experiment, ablation experiment and other related experimental details were described in detail.

[0162] 1. Dataset

[0163] This application uses two well-known multimodal intent recognition benchmark datasets, MInRec and EMOTyDA, to verify the effectiveness of the proposed method. The following section will describe these two datasets and their feature extraction process in detail.

[0164] MInRec: Dataset MInRec is the first benchmark dataset for real-world multimodal intent recognition, containing 2,224 instances of text, video, and audio modalities. MInRec contains 2 coarse-grained classifications and 20 fine-grained intent classifications, and the data is divided into training set, validation set, and test set in a ratio of 2:1:1.

[0165] EMOTyDA: Dataset EMOTyDA is a large-scale dataset for conversation behavior classification, in which all data are labeled with 12 common conversation behavior labels. Due to the limitation of computing resources, this application selects 2,500 representative samples as the experimental dataset based on the EMOTyDA dataset, and divides it into training set, validation set and test set in a ratio of 2:1:1.

[0166] 2. Data preprocessing

[0167] BERT is built on top of Transformer and has powerful language representation and feature extraction capabilities. In the experiment of this application, a pre-trained BERT (bert-base-uncased, with 12 Transformer layers) was used to extract text features. For image features, the video is first extracted into single images frame by frame, and key frames are selected. Then, the pre-trained Faster R-CNN and ResNet-50 are used to extract features of all key frames. Finally, the key frame features are mapped to annotated bounding box areas using existing object detection methods to obtain image features. Next, wav2vec is used to extract audio features. Wav2vec is a pre-trained model that maps raw audio samples to a feature space to better represent data through unsupervised training.

[0168] In this application, the features of the three modes are combined in series to facilitate subsequent multimodal intent recognition. Through the above method, this application can comprehensively analyze the data characteristics under different modes to support the accuracy of intent recognition.

[0169] 3. Experimental setup

[0170] The experiments in this application were completed on a personal computer equipped with the following hardware: the operating system is Windows 10, equipped with Intel(R) Core(TM) i9- 10900K and Nvidia 3090 graphics processors, and the memory capacity is 96GB. The method of this application was implemented using Pytorch 1.4.0, and Python 3.6 was used as the programming language. In order to reduce the overall loss, the Adam optimizer was used. The experimental parameters of this article are summarized in Table 1.

[0171] Table 1 Experimental parameter settings

[0172]

[0173] This application uses the Accuracy (ACC) and Macro-F1 (F1) indicators to evaluate the performance gap between the proposed method (denoted as TAOSME) and the baseline model. The formulas for ACC and F1 are defined as follows:

[0174] ;

[0175] ;

[0176] in, Indicates the number of correctly predicted samples, N is the total number of samples, C is the number of categories, is the positive prediction value of the i-th category, is the recall value of the i-th category.

[0177] 4. Baseline Model

[0178] In order to verify the performance of TAOSME of this application method under uncertain modes, it is necessary to conduct experiments using baseline models under different mode missing rates. Therefore, six advanced multimodal models are selected for comparison. The descriptions of the six baseline models are as follows:

[0179] Mult (#): Mult uses an end-to-end approach to process unaligned multimodal data. It extends the ordinary Transformer to a cross-modal Transformer through a bidirectional cross-modal attention mechanism, which helps capture the interactions between different modalities in the latent space.

[0180] MAG_Bert (#): MAG_Bert designs a multimodal adaptation gate MAG that allows BERT to accept multimodal non-language data during fine-tuning. MAG can produce position shifts in the semantic space to adapt to audio and visual information. MAG_Bert can be flexibly configured between BERT layers to receive input from non-language modalities.

[0181] IF-MMIN (#): The IF-MMIN method addresses the challenges posed by differences between modalities by learning invariant features and using them to predict and fill in missing modal data, especially when the modal data is incomplete or uncertain. In extensive experiments, the model performs well, outperforming baseline methods and significantly improving multimodal emotion recognition performance.

[0182] MISA (#): MISA adopts a multi-task learning framework to learn different modalities by projecting them into two spaces: a modality-invariant subspace and a modality-specific subspace, thereby obtaining a multimodal representation with both modality-invariant and modality-specific properties. On the other hand, different subspaces are designed to capture the unique properties of each modality. To this end, the training objectives include four parts: similarity loss, difference loss, reconstruction loss, and task-specific loss.

[0183] EMRFM (#): EMRFM is an effective multimodal representation and fusion method for intent recognition in real multimodal scenarios. This method constructs modality-shared and modality-specific encoders to learn modality-shared and specific feature representations, and designs an adaptive multimodal feature fusion method based on attention-gated neural networks to eliminate noise features. Experiments show that EMRFM's multimodal representation learns the shared and specific features of the modalities well, and the multimodal feature fusion achieves adaptive fusion and effectively reduces possible noise interference.

[0184] SDIF-DA (#): SDIF-DA is a shallow-to-deep interaction framework based on data augmentation. SDIF-DA utilizes a shallow-to-deep interaction module to gradually and effectively align and fuse features from text, video, and audio modalities.

[0185] 5. Performance Verification

[0186] In order to evaluate the performance of TAOSME in multimodal intent recognition under uncertain modalities, a large number of experiments were conducted on two public datasets, MInRec and EMOTyDA, as shown in Table 2. Specifically, this application focuses on a more general case, that is, the lost modality may occur in the training, validation, and testing phases, where each modality of each data sample may be lost. This application defines the modality missing rate as the proportion of lost modality data to the entire dataset. Then, by setting different modality missing rates (0.1-0.5), the accuracy (ACC) and F1 score of TAOSME are compared with those of each benchmark model.

[0187] Table 2 Comparison results of various methods under different modal missing rates

[0188]

[0189] For the public dataset MInRec, this experiment sets the modal missing rate to 0.1, 0.2, 0.3, 0.4 and 0.5. The experimental results are shown in Table 2. As can be seen from Table 2, the method TAOSME of this application outperforms other baseline models in both evaluation indicators (ACC and F1). In addition, compared with other baseline models, when the modal missing rate is 0.3, the method TAOSME of this application achieves higher scores in ACC and F1, increasing the ACC value by 2.05% to 4.99% and the F1 indicator value by 0.76% to 4.42%. Therefore, according to the results of Table 2, it can be concluded that the overall performance of the method TAOSME of this application is better than other baseline models on the MInRec dataset.

[0190] For the public dataset EMOTyDA, this experiment also sets the modal missing rate to 0.1, 0.2, 0.3, 0.4 and 0.5. The experimental results are shown in Table 2. It can be seen from Table 2 that the method TAOSME of this application is better than other baseline models overall. Compared with other baseline models, when the modal missing rate is 0.2, the method TAOSME of this application achieves higher scores in ACC and F1, increasing the ACC value by 1.5% to 4.63% and the F1 index value by 1.08% to 9.68%. However, when only the modal missing rate is 0.5, the ACC value of TAOSME is 0.75% lower than that of the MISA model, because the ACC value of MISA is abnormally improved at a modal missing rate of 40%. Therefore, according to the results of Table 2, it can be concluded that the overall performance of the method TAOSME of this application is better than other baseline models on the EMOTyDA dataset.

[0191] 6. Multi-classification

[0192] In order to test the multi-classification performance of the proposed method TAOSME, experiments were conducted on the MInRec dataset with different granularities. The coarse-grained classification of the MInRec dataset includes two categories, "expressing emotions or attitudes" and "achieving goals", and the fine-grained classification includes 20 categories. This experiment selects three relatively advanced models, MISA, EMRFM, and SDIF-DA, as baseline models. The experimental results are shown in Figure 2. Figure 1 , Figure 2 , Figure 3 , Figure 4 As shown in the figure, the vertical axis represents the evaluation index (F1 or ACC), and the horizontal axis represents the modality missing rate.

[0193] from Figure 1 and Figure 2 It can be seen that for binary classification, when the missing rate is set to 0.1, the F1 and ACC values ​​of TAOSME of this application are 1.17% and 1.57% higher than those of the baseline model MISA, respectively. When the missing rate is 0.4, the F1 value of TAOSME is 1.01% higher than that of the baseline model EMRFM, and the ACC value of TAOSME is 1.11% higher than that of the baseline model EMRFM. When the missing rate is set to 0.5, compared with the baseline model SDIF-DA, TAOSME improves by 2.15% in F1 and 2.26% in ACC.

[0194] from Figure 3 and Figure 4 It can be seen that for twenty categories, when the missing rate is set to 0.1, the F1 and ACC values ​​of TAOSME of this application are 2.49% and 1.82% higher than those of the baseline model MISA, respectively. When the missing rate is 0.4, the F1 value of TAOSME is 3.19% higher than that of the baseline model EMRFM, and the ACC value of TAOSME is 3.23% higher than that of the baseline model EMRFM. When the missing rate is set to 0.5, compared with the baseline model SDIF-DA, TAOSME improves by 0.85% in F1 and 1.36% in ACC.

[0195] Based on the above experimental results, it can be concluded that the proposed method TAOSME has better performance in multi-class intent recognition.

[0196] 7. Ablation experiment

[0197] In order to verify the performance of TAOSME in different modalities and the effectiveness of different modules of TAOSME, modality and module ablation experiments were conducted based on the MInRec dataset. "T", "A" and "V" are used to represent text, audio and image modalities respectively.

[0198] 7.1 Modal Ablation Experiment

[0199] This experiment considers the following two situations:

[0200] A. Use any two modalities for intent recognition (e.g., T+V, T+A, and V+A). In this scenario, the modality loss rate is set to 0.1, 0.2, 0.3, 0.4, and 0.5. For the combination of image and audio (V+A), since no text modality is involved, the image and audio modalities are extracted by the Transformer encoder, then input into the common space for splicing, and directly classified for intent classification;

[0201] B. Using three modalities (T+A+V) at the same time, the method of this application is used for intent recognition. In this scenario, the modality loss rate is set to 0.1, 0.2, 0.3, 0.4 and 0.5.

[0202] The experimental results of modality ablation are shown in Table 3, with the best results in bold. As can be seen from Table 3, in scenario A, the bimodal combination containing the text modality achieves better results than the bimodal combination without the text modality. Among the two modality combinations, the ACC value of the bimodal combination without the text modality is reduced by about 50% to 53% compared with the bimodal combination without the image or audio modality. In scenario B, the best results are obtained when all three modalities are used simultaneously.

[0203] Table 3 Modal ablation experimental results under different modal missing rates

[0204]

[0205] 7.2 Module Ablation Experiment

[0206] In this experiment, some model variants are generated by removing different modules from the method TAOSME of this application, and the effectiveness of different modules in TAOSME is verified by testing the performance of the model variants.

[0207] The generated model variants are: (1) Remove the text adaptive optimization module from TAOSME to generate the model variant Model-TAO. (2) Remove the similar modality enhancement module from TAOSME to generate the model variant Model-SME.

[0208] The results of the module ablation experiment are shown in Table 4. Specifically, when the modality missing rate ranges from 0.1 to 0.5, compared with the model variant Model-SME, removing the similar modality enhancement module reduces the ACC value by 1.14%, 2.14%, 1.63%, 1.64% and 1.46%, and reduces the F1 score by 0.73%, 1.47%, 1.67%, 2.64% and 1.46%; compared with the model variant Model-TAO, removing the text adaptive optimization module reduces the ACC value by 0.68%, 0.28%, 2.12%, 1.18% and 1.15%, and also reduces the F1 score by 0.21%, 0.91%, 1.12%, 2.07% and 0.86%. At the missing rate of 0.3 and 0.4, the F1 score of removing the similar modality enhancement module is the largest. Therefore, the similar modality enhancement module is dominant in TAOSME.

[0209] In summary, the text adaptive optimization module and the similar modality enhancement module play a very important role in TAOSME, and can effectively solve the performance degradation problem caused by uncertain modalities.

[0210] Table 4 Module ablation experimental results under different modality missing rates

[0211]

[0212] 8. Determination of the number of similar full-modal samples

[0213] This application needs to retrieve similar full-modal samples through the current sample features using the measurement method of the cosine similarity function, so as to learn similar information and improve the quality of the fused features. However, it is very important to carefully balance the number of similar full-modal samples. If there are too few similar full-modal samples, the current fusion features cannot fully absorb the modal commonalities, limiting information interaction and fusion; if there are too many similar full-modal samples, it may cause information overload and introduce noise that interferes with model learning and affects performance. Therefore, finding a balance point that can effectively propagate modal commonalities while avoiding interference from redundant information is the key to improving model efficiency and accuracy. In order to thoroughly check the rationality of the number of similar full-modal samples retrieved, targeted experiments were conducted on the MInRec dataset, and the results are shown in the figure. Figure 2 As shown in the figure, the missing rate of 0.3 is used to compare the ACC and F1 of different similar numbers of full-modal samples, because the missing rate of 0.3 is relatively high, which neither causes the model to completely lack modal information nor causes the modal information to be too rich, but is at a moderate level. This intermediate missing rate can more clearly observe the impact of the number of retrieved samples on the propagation of modal commonality and model performance.

[0214] from Figure 5 From the experimental data presented in Figure 2, we can see that as the number of similar full-modal samples increases from 1 to 3, TAOSME's ACC fluctuates between 62.73% and 64.55%, and F1 fluctuates between 60.58% and 62.61%. When the number of similar full-modal samples reaches 4, all evaluation indicators are optimal. When there are more than 4 nodes, the performance indicators decrease. Therefore, retrieving 4 similar full-modal samples can maintain high operating efficiency while ensuring model accuracy, achieving an ideal balance between performance and speed.

Claims

1. A multimodal intent recognition method under uncertain modalities, characterized in that: The following steps are involved: S1: Acquire uncertain modal data, where the uncertain modal data is text modality and / or audio modality and / or image modality, perform feature extraction, and obtain text features and / or audio features and / or image features respectively; S2: The text features, audio features, and image features are processed by full connection, ReLU activation function, and discard in sequence to obtain high-level features of the text modality, high-level features of the audio modality, and high-level features of the image modality. The high-level features of the text modality, the high-level features of the audio modality, and the high-level features of the image modality are concatenated to generate common features. After the text features and common features are concatenated, they are processed by multi-head attention to obtain the first relationship matrix. The Tanh activation function is used to generate the weights of the text features according to the first relationship matrix. The weight of the text feature is matrix-multiplied with the text feature to obtain the optimized text feature; S3: Audio features, image features, and optimized text features are fused through multimodal features to obtain fused features; S4: obtaining a full-modal sample data set, determining the modal priority based on the uncertain modal data, and using the cosine similarity function to calculate the similarity between the modality with the highest priority and the corresponding modality in the full-modal sample data set to obtain a similar full-modal sample set; The omnimodal features in similar omnimodal sample sets are concatenated with semantic identifiers to obtain target enhanced features; The target enhanced features and fused features are processed by cross attention to obtain enhanced fused features; S5: The enhanced fusion features are classified and processed to obtain the predicted intent results.

2. The multimodal intention recognition method under uncertain modality according to claim 1, characterized in that: In step S2, the Tanh activation function is used to generate the weights of the text features according to the first relationship matrix, specifically: After the first relationship matrix is ​​processed by residual connection, layer normalization and ReLU activation function are performed in sequence to obtain the second relationship matrix; After the second relationship matrix is ​​fully connected, it is processed by the Tanh activation function and numerically scaled in turn to generate the weights of the text features.

3. The multimodal intention recognition method under uncertain modality according to claim 1, characterized in that: In step S2, the weight of the text feature is matrix-multiplied with the text feature to obtain the optimized text feature, which is according to the formula: ,accomplish; In the formula, Represents optimized text features; Represents the weight of text features; Represents text features; Indicates the existence status of the text mode, with a value of 0 or 1.

4. The multimodal intention recognition method under uncertain modality according to claim 1, characterized in that: In step S4, the omnimodal features in the similar omnimodal sample set are concatenated with the semantic identifier to obtain the target enhanced features, specifically: Based on similar omnimodal sample sets, the text features, audio features, image features and semantic identifiers in each similar omnimodal sample are concatenated to obtain similar enhanced features of a single sample. Similar enhanced features of multiple single samples are concatenated to obtain the target enhanced features.

5. The method for multimodal intention recognition under uncertain modality according to claim 1, characterized in that: In step S3, the audio features, image features, and optimized text features are fused through multimodal features to obtain fused features, specifically: The audio features, image features, and optimized text features are processed by self-attention to obtain audio attention features, image attention features, and text attention features; The text attention features and the image attention features are processed cross-modally to obtain text-to-image related features and image-to-text related features; The text attention features and the audio attention features are processed cross-modally to obtain text-to-audio related features and audio-to-text related features; The text-to-image related features and the image-to-text related features are combined to obtain the target image features; the text-to-audio related features and the audio-to-text related features are combined to obtain the target audio features; The text attention features, target image features and target audio features are concatenated to obtain fused features.

6. The method for multimodal intention recognition under uncertain modality according to claim 5, characterized in that: The text-to-image related features and the image-to-text related features are combined to obtain target image features; the text-to-audio related features and the audio-to-text related features are combined to obtain target audio features, specifically: The text-to-image related features and the image-to-text related features are sequentially concatenated, convolved, and processed with the Sigmoid activation function, and then added to the image-to-text related features to obtain the target image features; The text-to-audio related features and the audio-to-text related features are concatenated, convolved, and processed with the Sigmoid activation function in sequence, and then added to the audio-to-text related features to obtain the target audio features.

7. The method for multimodal intention recognition under uncertain modality according to claim 1, characterized in that: In step S4, a full-modal sample data set is obtained, and based on the uncertain modal data, the modal priority is determined, and the similarity between the modality with the highest priority and the corresponding modality in the full-modal sample data set is calculated using the cosine similarity function to obtain a similar full-modal sample set. The operation is as follows: If the acquired uncertain modal data has a text modality, the text modality is the modality with the highest priority. The cosine similarity function is used to calculate the similarity between the text modality and the text modality in the full modality sample data to obtain a similar full modality sample set. If the acquired uncertain modality data does not have a text modality but has an image modality, the image modality is the modality with the highest priority, and the cosine similarity function is used to calculate the similarity between the image modality and the image modality in the full modality sample data to obtain a similar full modality sample set; If the acquired uncertain modal data only contains the audio modality, the audio modality is the modality with the highest priority. The cosine similarity function is used to calculate the similarity between the audio modality and the audio modality in the full modality sample data to obtain a similar full modality sample set.

8. A multimodal intention recognition system under uncertain modality, characterized in that: include: Feature extraction module: used to obtain uncertain modal data, the uncertain modal data is text modality and / or audio modality and / or image modality, perform feature extraction, and obtain text features and / or audio features and / or image features respectively; Text adaptive optimization module: used to process text features, audio features, and image features through full connection processing, ReLU activation function processing, and discard processing in sequence to obtain high-level features of text modality, high-level features of audio modality, and high-level features of image modality, and then splice the high-level features of text modality, audio modality, and image modality to generate common features; After the text features and common features are concatenated, they are processed by multi-head attention to obtain the first relationship matrix. The Tanh activation function is used to generate the weights of the text features according to the first relationship matrix. The weight of the text feature is matrix-multiplied with the text feature to obtain the optimized text feature; Multimodal fusion module: used to fuse audio features, image features, and optimized text features through multimodal features to obtain fused features; Similar modality enhancement module: used to obtain a full-modality sample data set, determine the modality priority based on the uncertain modal data, and use the cosine similarity function to calculate the similarity between the modality with the highest priority and the corresponding modality in the full-modality sample data set to obtain a similar full-modality sample set; The omnimodal features in similar omnimodal sample sets are concatenated with semantic identifiers to obtain target enhanced features; The target enhanced features and fused features are processed by cross attention to obtain enhanced fused features; Intent recognition module: used to enhance the fusion features through classification processing to obtain the predicted intent results.

9. A multimodal intention recognition device under uncertain modality, characterized in that: It comprises a processor and a memory, wherein when the processor executes the computer program stored in the memory, it implements the multimodal intention recognition method under uncertain modality as described in any one of claims 1-7.

10. A medium, characterized in that Used to store a computer program, wherein when the computer program is executed by a processor, the method for multimodal intention recognition under uncertain modalities as described in any one of claims 1 to 7 is implemented.

Images