Mongolian multimodal sentiment analysis method based on adaptive supermodality and multi-layer attention

Abstract

A Mongolian multimodal sentiment analysis method based on adaptive supermodality and multi-layer attention extracts text, audio and video features from multimodal data respectively, and then uses a gated Transformer encoder to extract low-level text features, low-level audio features and low-level video features respectively. In an adaptive supermodal learning layer, low-level text features are used to guide low-level audio features and low-level video features to generate intermediate supermodal features. Subsequently, the intermediate supermodal features are used to act on the low-level text features and a gated Transformer encoder is used to extract next-scale text features, thereby realizing bidirectional cross-modal attention interaction. Multiple interactions obtain final text features and final supermodal features, and a joint multimodal representation is obtained through a cross-modal Transformer. The soft attention mechanism is then used for weighted fusion, and finally multimodal sentiment classification is realized through a fully connected layer. The present invention can greatly improve the effect of Mongolian multimodal sentiment analysis.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence and natural language processing, and in particular relates to a Mongolian multimodal sentiment analysis method based on adaptive supermodality and multi-layer attention. Background Art

[0002] Sentiment analysis is the process of analyzing, processing, summarizing, and reasoning about data containing emotional information. The theory and algorithmic construction of sentiment analysis involve multiple aspects, including artificial intelligence (AI), computer vision (CV), and natural language processing (NLP), making it a multidisciplinary research field.

[0003] Traditional sentiment analysis is often limited to a single information source, such as text or speech, and struggles to fully capture the complexities of human emotions. Multimodal sentiment analysis (MSA) aims to integrate diverse information sources, such as vision, speech, and text, to more accurately identify and understand human emotions. However, compared to sentiment analysis research in mainstream languages ​​like Chinese and English, sentiment analysis in Mongolian, as a minority language, still suffers from limitations in accuracy and speed due to a lack of samples and the unique nature of its language structure. Summary of the Invention

[0004] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a Mongolian multimodal sentiment analysis method based on adaptive supermodality and multi-layer attention, so as to solve the problems existing in Mongolian sentiment analysis, such as easy loss of effective information, weak ability to extract sentiment information, and insufficient fusion caused by lack of deep fusion of multimodal features.

[0005] In order to achieve the above objectives, the present invention first preprocesses the raw data to obtain the initial features of each modality. Secondly, information filtering is implemented through a gated Transformer encoder to obtain low-level text features and initial supermodal features. Then, it enters the adaptive supermodal learning module, where the low-level text features and the initial supermodal interact through bidirectional cross-modal attention to obtain the final text features and final supermodal features. Then, a joint multimodal representation is obtained through a cross-modal Transformer, and the soft attention mechanism is used to assign different weights to the joint multimodal representation, and finally multimodal sentiment classification is achieved through a fully connected layer.

[0006] The technical solution adopted by the present invention can be further described as follows:

[0007] A Mongolian multimodal sentiment analysis method based on adaptive supermodality and multi-layer attention, characterized by comprising the following steps:

[0008] Step 1: Extract text features, audio features, and video features from multimodal data consisting of text, audio, and video.

[0009] Step 2, using a gated Transformer encoder to extract low-level text features, low-level audio features, and low-level video features from the text features, audio features, and video features, respectively;

[0010] Step 3: Using an adaptive supermodal learning module consisting of n gated Transformer encoders and n+1 adaptive supermodal learning layers, an initial supermodal feature is defined. In an adaptive supermodal learning layer, the low-level text feature is used to guide the low-level audio feature and the low-level video feature to generate an intermediate supermodal feature. Subsequently, the intermediate supermodal feature is applied to the low-level text feature and a gated Transformer encoder is used to extract the next-scale text feature to achieve bidirectional cross-modal attention interaction. The intermediate supermodal feature and the next-scale text feature are then subjected to n bidirectional cross-modal attention interactions to obtain the final text feature and the final supermodal feature.

[0011] In step 4, the final text features and the final supermodal features are passed through a cross-modal Transformer to obtain a joint multimodal representation, and then the soft attention mechanism is used to assign different weights to the joint multimodal representation for weighted fusion, and finally multimodal sentiment classification is achieved through a fully connected layer.

[0012] Compared with the prior art, the present invention has the following beneficial effects:

[0013] In the Mongolian multimodal sentiment analysis task, avoiding the loss of effective information and improving the ability to extract sentiment information are key issues in improving sentiment classification performance. In the traditional Transformer model, the feedforward layer is only a linear transformation of the input features, which makes it difficult to distinguish valuable information and easily causes information loss. The Transformer encoder based on the gated information channel constructed by the present invention introduces a gated information channel to replace the feedforward layer, adaptively selects to retain the initial features or the updated features, thereby avoiding the loss of important information, and is more efficient and accurate. In addition, the present invention dynamically learns emotion-related information by introducing an adaptive supermodal learning (AHL) module, generates an intermediate supermodal tag containing less irrelevant information, increases the weight of emotion-related features, thereby improving the Mongolian emotion information extraction capability, and facilitates the final multimodal fusion.

[0014] In Mongolian multimodal sentiment analysis, modalities such as text, vision, and audio contain key emotional information. However, existing multimodal fusion methods fail to effectively capture the complex interactive relationships between modalities, resulting in insufficient feature fusion. To address this problem, the present invention constructs a multi-layer attention mechanism. First, the internal information of each modality is modeled through a multi-head self-attention mechanism, and then a bidirectional cross-modal attention mechanism is used to promote the interaction between text features and visual and audio modalities. Finally, a unidirectional cross-modal attention mechanism is used to further fuse supermodal and text features. This layered dynamic interaction strategy can effectively promote information fusion between modalities, thereby solving the problem of insufficient multimodal feature fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flow chart of the present invention.

[0016] Figure 2 It is a schematic diagram of the overall architecture of the model of the present invention.

[0017] Figure 3 Schematic diagram of the gated Transformer encoder architecture of the present invention.

[0018] Figure 4 It is a schematic diagram of the gated information channel structure of the present invention.

[0019] Figure 5 Schematic diagram of the dual-scale text feature construction module of the present invention.

[0020] Figure 6 Schematic diagram of the adaptive supermodal learning layer structure of the present invention.

[0021] Figure 7 Schematic diagram of the multi-head self-attention mechanism of the present invention.

[0022] Figure 8 Schematic diagram of the cross-modal attention mechanism of the present invention.

[0023] Figure 9 Schematic diagram of the soft attention mechanism of the present invention.

[0024] Figure 10 This is a graph showing the accuracy change trend of the model training cycle of the present invention. DETAILED DESCRIPTION

[0025] The embodiments of the present invention are described in detail below with reference to the accompanying drawings and examples.

[0026] Research on multimodal sentiment analysis of Mongolian can help further improve basic Mongolian technologies such as speech recognition, text understanding, and sentiment analysis, providing technical support for its informatization applications. Multimodal sentiment analysis uses data composed of text, images, audio, and even video to obtain information from different modalities, effectively combining multimodal information to more accurately predict the emotional state of the observed target.

[0027] To this end, the present invention adopts a Mongolian multimodal sentiment analysis method based on adaptive supermodality and multi-layer attention, which mainly improves the effect of Mongolian multimodal sentiment analysis from the following three aspects.

[0028] First, a Transformer encoder model based on gated information channels is constructed to improve the efficiency of filtering redundant information features.

[0029] Specifically, in order to alleviate the problems of information loss and invalid feature transfer in traditional Transformers, the present invention constructs a Transformer encoder model based on the gated information channel (GIC), namely the gated Transformer encoder (GTE). The Transformer model has become the mainstream model structure in cross-modal tasks. Its core self-attention mechanism can effectively capture the dependencies within the sequence. However, the traditional Transformer encoder has specific shortcomings, especially when processing complex multimodal features, it is often unable to fully utilize the key information of each modality. Its feedforward network is relatively simple, and it fails to achieve feature screening and updating, which easily leads to information loss and invalid feature transfer.

[0030] To address the above issues, the present invention introduces a gated information channel on the basis of the traditional Transformer to replace the feedforward layer, realize information filtering, and avoid the loss of effective information. First, the data is output from the position code as the input of the encoder and enters the multi-head self-attention mechanism layer. The multi-head self-attention mechanism is used to calculate the similarity of different positions, and then the attention distribution is obtained through the Softmax function processing, thereby achieving focus and understanding of the input sequence. Then, feature screening and updating are performed through the gated information channel. The gated information channel includes a memory gate and an update gate. The memory gate is used to store valuable information, while the additional channel is used to memorize new information; the update gate is used to update the feature fully connected layer.

[0031] Secondly, we construct an audio-visual feature fusion model based on adaptive supermodal representation, use text features of different scales to guide the audio-visual modality, form a supermodality that complements the text modality, and improve the performance of multimodal sentiment analysis.

[0032] Specifically, although Mongolian multimodality provides richer emotional information for sentiment analysis than unimodality, the potential for irrelevant and conflicting cross-modal information weakens the ability to extract emotional information, hindering further performance improvements. Due to emotional irrelevance, information from different modalities can be ambiguous and conflicting, especially from non-dominant modalities such as audio and visual. This interference significantly limits the performance of multimodal sentiment analysis methods.

[0033] In response to the above problems, the present invention constructs an audio and video feature fusion model based on adaptive supermodal representation, uses text features to guide the audio-visual modality, forms a supermodality that complements the text modality, and then uses the supermodality to influence the text features, so that the text features of the next scale can better guide the audio-visual modality, thereby improving the performance of multimodal sentiment analysis and alleviating the problem of weak Mongolian emotional information extraction ability. First, each modality is converted into a unified form by using a gated Transformer encoder with an initialization mark. This operation not only suppresses redundant information between modalities, but also compresses the length of long sequences, facilitating efficient model calculation. Then, an adaptive supermodal learning (AHL) module is introduced, in which text features of different scales are used to guide the visual and audio modalities to generate intermediate supermodal tags. The tags contain less information that is not related to emotion, thereby improving the Mongolian emotional information extraction ability, so as to facilitate the final multimodal fusion.

[0034] Thirdly, a Mongolian multimodal sentiment analysis model based on a multi-layer attention mechanism is constructed to achieve deep fusion of multimodal information and improve the performance and generalization of Mongolian sentiment analysis.

[0035] Specifically, when using multimodal technology for sentiment analysis tasks, mainstream multimodal sentiment analysis methods based on deep learning often encounter some challenges, among which the problems of insufficient feature fusion between modalities and unreasonable weight distribution of each modality are particularly obvious.

[0036] In response to the above problems, the present invention constructs a Mongolian multimodal sentiment analysis model based on a multi-layer attention mechanism, and constructs a multi-layer attention mechanism for multimodal sentiment analysis. Through the multi-level interaction of multi-head self-attention mechanism, cross-modal attention mechanism and soft attention mechanism, the deep fusion of internal information and multimodal information is realized, and the problem of insufficient feature fusion between modalities and unreasonable weight distribution of each modality is alleviated. First, the information of each modality is modeled using a multi-head self-attention mechanism to obtain the initial features of each modality. In the adaptive supermodal learning layer, a bidirectional cross-modal attention mechanism is used to update the supermodal features by calculating the relationship between the obtained text features and the remaining two modalities. The supermodal features are further affected by the reverse path to enable the high-level features of the text to be more fully learned. Then, the unidirectional cross-modal attention mechanism is used in the cross-modal transformer to promote interaction between modalities, realize the fusion of text features and supermodal features, and obtain a joint multimodal representation. Finally, the soft attention mechanism is used to assign different weights to the joint multimodal representation, so that the model pays more attention to the task-related modal information in the fused representation, and finally realizes multimodal sentiment classification through the fully connected layer.

[0037] Hereinafter, the method of the present invention will be described in further detail in combination with the above principles.

[0038] like Figure 1 and Figure 2 As shown in FIG, the Mongolian multimodal sentiment analysis method based on adaptive supermodality and multi-layer attention of the present invention mainly includes the following steps:

[0039] Step 1: For multimodal data, extract the initial features of each modality.

[0040] In the present invention, multimodal data consists of text, audio and video, and the extracted features include text features, audio features and video features. In a standard multimodal dataset, text, audio and video are aligned by their respective IDs.

[0041] In terms of functional modules, this step can be defined as a preprocessing module. Specifically, mBERT, Librosa, and OpenFace can be used in this module to process text, audio, and video data respectively, and extract the initial features of each modality. The specific processing method is as follows:

[0042] (a) Text processing

[0043] mBERT is a multilingual pre-trained model based on the BERT architecture, specifically designed for processing text in multiple languages, including Mongolian. When using mBERT to process Mongolian text, the multilingual BERT model is loaded to perform word segmentation and encoding on the text. The processed input is then passed into the model to extract embedded features. These feature vectors effectively represent the semantic information of the text.

[0044] (b) Audio processing

[0045] First, use the Librosa library to read audio data and obtain the sampling rate and audio signal. Next, normalize the audio signal to ensure it is within the range [-1, 1]. Extract audio features such as MFCCs (Mel-Frequency Cepstral Coefficients) and spectral features. Finally, use the Short-Time Fourier Transform (STFT) to analyze the audio signal in the frequency domain and examine its spectral characteristics.

[0046] (c) Video processing

[0047] First, we use OpenCV to read video frames and select key frames as images for feature extraction. We then use OpenFace for face detection and feature extraction. Once a face is detected, OpenFace extracts a series of facial features, including facial key points and facial expression features.

[0048] The original data is preprocessed to obtain the initial features of each mode Where m∈{t,a,v}, t,a,v refer to text, audio and video respectively.

[0049] Step 2: Use the gated Transformer encoder to obtain the low-level text features, low-level audio features, and low-level video features from step 1.

[0050] In this step, the initial features obtained after data preprocessing are further extracted through the gated Transformer encoder. The gated Transformer encoder constructed by the present invention replaces the feedforward layer in the traditional Transformer encoder with a gated information channel, which enables the model to selectively update features, thereby enhancing the accuracy of feature expression and avoiding excessive information interference. The structure of the gated Transformer encoder is as follows: Figure 3 As shown in the figure, N× represents that this gated Transformer encoder is stacked N times. The structure shown is a sublayer. The gated Transformer encoder is composed of multiple stacked sublayers, each of which retains the Transformer's multi-head self-attention mechanism.

[0051] The improved model still retains the Transformer's multi-head self-attention mechanism, which is used to calculate similarities between different positions in the input data. First, the data is positionally encoded as input and enters the multi-head self-attention layer, which calculates the dependencies between positions in the sequence. The attention weights are normalized using the Softmax function, allowing the model to focus on more important sequence positions.

[0052] The present invention sets up a gated information channel to filter out unnecessary data features. The gated information channel consists of two parts: a memory gate and an update gate. The memory gate is used to store valuable initial feature information, while the additional channel is used to memorize new information. The update gate is used to update the feature fully connected layer. The present invention designs a network structure similar to GRU, such as Figure 4 shown.

[0053] In the gated Transformer encoder, the features H∈R of each input modality are first extracted through a hierarchical multi-head self-attention mechanism. T×d , where T is the number of time steps and d is the feature dimension. Then, the memory gate and update gate in the gated information channel are used to effectively filter and fuse the extracted features.

[0054] Memory gates are used to store information and add some new content to features:

[0055]

[0056] Among them, GELU is the activation function, G M Represents the new content added by the memory gate to the feature, M∈R d×d is the parameter matrix, σ represents the Sigmoid activation function. The memory gate retains valuable initial feature information and combines it with new information to generate new features H M :

[0057]

[0058] The update gate is used to update the original features and avoid adding too much noise to the features. Its calculation formula is:

[0059]

[0060] Among them, G U represents the update gate, H new Represents the updated features, U∈R d×d is the parameter matrix. This formula ensures the flexibility of the update process, that is, retaining the key information in the original features while making necessary updates to some features.

[0061] Through this design, the model can dynamically choose to retain the initial features or the updated features, thus avoiding the loss of important information. At the same time, the gated information channel effectively avoids the interference of redundant noise information on the features.

[0062] Step 3: Use an adaptive supermodal learning module consisting of n gated Transformer encoders and n+1 adaptive supermodal learning layers to define initial supermodal features. In an adaptive supermodal learning layer, use the low-level text features to guide the low-level audio features and low-level video features to generate intermediate supermodal features. Then, use the intermediate supermodal features to act on the low-level text features and use a gated Transformer encoder to extract the next-scale text features to achieve bidirectional cross-modal attention interaction. The intermediate supermodal features and the next-scale text features continue to perform n bidirectional cross-modal attention interactions to obtain the final text features and the final supermodal features.

[0063] After modal information filtering, this step further uses the Adaptive Supermodal Learning (AHL) module to learn refined supermodal representations that contain relevant and highly complementary text features. The AHL module, consisting of two Gated Transformer Encoders (GTEs) and three AHL layers, aims to learn text features at different scales. Guided by the text features, it adaptively learns a supermodal feature from the visual and audio modalities. The supermodal feature is then leveraged to influence the text features, enabling the next-scale text features to better guide the audiovisual modality, thereby improving the performance of MSA.

[0064] This step is mainly divided into two parts: the construction of dual-scale text features and the related execution process of the adaptive supermodal learning layer.

[0065] (1) Construction of dual-scale text features.

[0066] Initial text features extracted by mBERT It will be passed to the subsequent GTE layer for further semantic modeling and feature enhancement, and the self-attention mechanism will be used to capture the long-distance dependencies in the text.

[0067] Will Defined as low-level text features. With this feature, the present invention introduces two GTE layers to learn medium-scale and high-scale (i.e. ), that is, in this embodiment, n=2. Figure 5 As shown in the figure, unlike the GTE layer in the modal information filtering stage (which passes basic information to the initialized token) (this stage requires three GTEs, one for each modality), the GTE layer in this stage directly constructs dual-scale text features based on text features. The method is as follows:

[0068]

[0069] Among them, i∈{2,3}, are text features at different scales, with a size of T×d, and The i-th layer gated Transformer encoder and corresponding parameters for text feature learning, is the corresponding supermodal feature H hyper Text features after reverse influence. In practice, 8-head attention can be used to model the information of each modality.

[0070] The two-layer GTE extracts the dual-scale features of the text layer by layer. Each layer of GTE uses the self-attention mechanism to model the text sequence, gradually enhancing the understanding of the context and semantic modeling capabilities. The output of the last layer of GTE It includes text features from local to global, and goes through multiple layers of abstraction and filtering.

[0071] (2) Related execution process of the adaptive supermodal learning layer.

[0072] In the adaptive supermodal learning layer, for text features of different scales First initialize a supermodal feature Then, the bidirectional cross-modal attention mechanism is used to update the obtained text features by calculating the relationship between the audio features and the video features. And use the updated supermodal features Reverse impact Get the text features after reverse influence

[0073] Specifically, if Figure 6 As shown, the extracted As the query vector query, the low-level audio features obtained through GTE As the key vector key, we can get the similarity matrix α between the text modality and the audio modality, which is expressed as:

[0074]

[0075] Among them, softmax represents the weight normalization operation, is a learnable parameter, d k The dimension of each attention head. In practice, 8 attention heads are used and d k Set to 16.

[0076] In cross-modal attention, Q t is the query vector matrix of the target modality text, Ka is the key vector matrix from the audio modality, representing the feature matrix used to match the query vector matrix. The T in the upper right corner of the variable in the formula represents the transpose of the variable. and is a learnable linear transformation weight matrix, which is used to transform Mapping to the key vector space and H t Mapped to the query vector space.

[0077] Similarly, low-level video features As the key vector, similar to α, the similarity matrix β between the text modality and the visual modality can be expressed as:

[0078]

[0079] Among them, K v is the key vector matrix from the visual modality, representing the feature matrix used to match the query vector matrix, is a learnable linear transformation weight matrix used to transform Mapped to the key vector space.

[0080] Therefore, the supermodal feature It can be updated by weighted audio features and weighted visual features, expressed as:

[0081]

[0082] in, represents the supermodal features output by the jth adaptive supermodal learning layer, j∈{1,2,3}, and is a learnable linear transformation weight matrix, which is used to transform and Mapped to the same space as the supermode, V a The eigenvector representing the audio mode, V v A feature vector representing the visual modality.

[0083] Updated supermodal features It is necessary to reverse the text mode and further update the text features get

[0084]

[0085] in, is a learnable weight matrix, which is used to Mapping to the query vector space and Mapping to a key vector space and a value vector space.

[0086] In step 4, the final text features and the final supermodal features are passed through the cross-modal Transformer to obtain a joint multimodal representation. The soft attention mechanism is then used to assign different weights to the joint multimodal representation for weighted fusion, and finally multimodal sentiment classification is achieved through the fully connected layer.

[0087] The overall architecture of this invention utilizes a multi-layered attention mechanism, including multi-head self-attention, cross-modal attention, and soft attention, ultimately achieving a deep fusion of internal and multimodal information. These mechanisms are described below.

[0088] (1) Multi-head self-attention mechanism.

[0089] The multi-head self-attention mechanism models single modality information. For different modalities such as text, audio, and vision, the multi-head self-attention mechanism is used to model their features. The multi-head self-attention mechanism of the present invention is implemented in the gated Transformer encoder. The multi-head self-attention mechanism can capture the internal correlation features of a single modality and help the model understand the important information in each modality. The structure of the multi-head self-attention mechanism is as follows Figure 7 shown.

[0090] (a) Linear transformation.

[0091] For each head, first perform a linear transformation through the learned weight matrix to generate Q m , K m and V m (Subscript m represents the mode, m belongs to {t, a, v}, Q (query vector), K (key vector), V (value vector) for each mode):

[0092]

[0093] where n is the index of the head, m∈{t,a,v} represents the text, audio and visual modalities respectively, and is the corresponding weight matrix, Enter a sequence for a modality.

[0094] (b) Calculate attention score

[0095] For each head, calculate the attention matrix:

[0096]

[0097] Among them, d k is the dimension of each attention head.

[0098] (c) Output of multi-head self-attention

[0099] The output of the multi-head attention mechanism is the concatenation of the outputs of all heads and then a linear transformation. There are h heads, and the final multi-head attention output is:

[0100] MultiHead(Q m ,K m ,V m )=Copncat(head1,head2,…,head h )W O

[0101] The output of each header is:

[0102]

[0103] W O is the weight matrix used to transform the concatenated result into output.

[0104] (2) Cross-modal attention mechanism

[0105] (a) Unidirectional cross-modal attention mechanism

[0106] In the cross-modal Transformer, the present invention adopts a unidirectional cross-modal attention mechanism to achieve the fusion between text features and super-modal features. Through this mechanism, the information interaction between the features of each modality is further promoted, so that the text modality can absorb the effective features of other modalities, further enhancing the effect of modal fusion. The structure of the cross-modal attention mechanism is as follows: Figure 8 shown.

[0107]

[0108] in, and Represent the supermodal feature matrix K and the weight matrix V respectively. hyper is a value vector composed of the final supermodal features H hyper Mapping a matrix by value The final supermodal feature H hyper Mapped to the value space.

[0109] In the multimodal fusion of the present invention, the initialized token is represented by H0∈R 1×d , firstly, H0 and and Concatenate and concatenate to get a new text feature H t and a new supermodal feature H hyper , and then use the cross-modal transformer to transfer the necessary joint and complementary information to these tokens. In practice, the cross-modal transformer transforms the text feature H t(as K) and supermodal features H hyper (as key and value) are fused together to obtain a joint multimodal representation H p ∈R 1×d Used for subsequent sentiment analysis.

[0110] The cross-modal fusion transformer is represented as CrossTrans, so the fusion process can be expressed as:

[0111]

[0112] (b) Bidirectional cross-modal attention mechanism

[0113] In the adaptive supermodal learning layer, a bidirectional cross-modal attention mechanism is used to calculate the relationship between text features and the features of the remaining two modalities (audio and visual) to update the supermodal features. At the same time, the updated supermodal features are fed back to the text features through a reverse path, allowing the high-level features of the text to more fully learn the complementary information of the other modalities, thereby achieving deep fusion.

[0114] (3) Soft Attention Mechanism

[0115] The present invention uses a soft attention mechanism to assign different weights to the joint multimodal representation for weighted fusion, and obtains the final weighted feature representation H final This module assigns different weights to different features according to their importance in sentiment classification, so that it can more accurately rely on useful modal information when making decisions. Figure 9 As shown, the process is mainly divided into the following three steps:

[0116] (a) Calculate the attention score:

[0117]

[0118] in, is the joint multimodal representation H p The kth feature, W s ∈R d×1 is a learnable weight matrix, e k yes Attention score;

[0119] (b) Calculate soft attention weights:

[0120] α k =softmax(e k )

[0121] Among them, α k yes The attention weight of

[0122] (c) Weighted output representation:

[0123]

[0124] To validate the effectiveness of our method, we conducted a test experiment using a Mongolian multimodal emotion dataset consisting of 2,100 multimodal samples covering seven discrete emotions: happiness, anger, sadness, surprise, fear, disgust, and neutral (i.e., a relatively smooth, emotionless voice). For each emotion, we used 300 sentences. For each emotion category, we randomly selected 60% of the data as a training set, 10% as a validation set, and 30% as a test set.

[0125] The present invention intends to use accuracy, precision, recall and F1 value as indicators to evaluate model performance. Specifically, the calculation formula for accuracy is The calculation formula for accuracy is The calculation formula for recall is The calculation formula of F1 value is: Among them, Accuracy represents the accuracy rate, that is, the proportion of the number of samples correctly predicted by the model to the total number of samples; Precision represents the precision rate, that is, the proportion of samples predicted by the model to be positive that are actually positive; Recall represents the recall rate, that is, the proportion of samples that are actually positive that are correctly predicted to be positive; F1 represents the F1 value, that is, the harmonic mean of the precision rate and the recall rate, which comprehensively reflects the performance of the two. TP represents the number of samples that are actually positive and predicted to be positive by the model. FN represents the number of samples that are predicted by the model to be negative but are actually positive. FP represents the number of samples that are predicted by the model to be positive but are actually negative. TN represents the number of samples that are actually negative and predicted to be negative by the model. The higher the scores of the accuracy rate, precision rate, recall rate and F1 value, the better the performance of the sentiment analysis model. As shown in Table 1, the confusion matrix required for the calculation is given:

[0126] Table 1 Confusion matrix

[0127] Predicted as positive Predicted as negative Actual positive example TP FN Actually a negative example FP TN

[0128] The accuracy change trend in 100 training cycles is as follows Figure 10As shown. Both the training accuracy and the test accuracy increase rapidly within the first 20 epochs, which is the initial stage of model learning. This shows that the model is effectively fitting the training data and can also perform well on the test set. At the 40th epoch, the test accuracy reached the highest, which was 92.38%. The test accuracy subsequently gradually stabilized between 0.85-0.9. This shows that the Mongolian multimodal sentiment analysis model based on adaptive supermodal representation and multi-layer attention mechanism proposed in the present invention has good results. The above experiments can fully demonstrate the effectiveness of the method proposed in the present invention, and this method can provide theoretical and practical references for Mongolian multimodal sentiment analysis tasks.

Claims

1. A Mongolian multimodal sentiment analysis method based on adaptive supermodality and multi-layer attention, characterized by: The steps include: Step 1: Extract text features, audio features, and video features from multimodal data consisting of text, audio, and video. Step 2, using a gated Transformer encoder to extract low-level text features, low-level audio features, and low-level video features from the text features, audio features, and video features, respectively; Step 3: Using an adaptive supermodal learning module consisting of n gated Transformer encoders and n+1 adaptive supermodal learning layers, an initial supermodal feature is defined. In an adaptive supermodal learning layer, the low-level text feature is used to guide the low-level audio feature and the low-level video feature to generate an intermediate supermodal feature. Subsequently, the intermediate supermodal feature is used to act on the low-level text feature and a gated Transformer encoder is used to extract the next-scale text feature to achieve bidirectional cross-modal attention interaction. The intermediate supermodal feature and the next-scale text feature are then subjected to n bidirectional cross-modal attention interactions to obtain the final text feature and the final supermodal feature. Wherein, n=2, and the adaptive supermodal learning module constructs a dual-scale text feature as follows: Among them, i∈{2,3}, are text features at different scales, with a size of T×d, is a low-level text feature, and The i-th layer gated Transformer encoder and corresponding parameters for text feature learning, is the corresponding supermodal feature H hyper Text features after reverse impact; The adaptive supermodal learning layer is used for text features of different scales. First initialize a supermodal feature Then, the bidirectional cross-modal attention mechanism is used to update the obtained text features by calculating the relationship between the audio features and the video features. And use the updated supermodal features Reverse impact Get the text features after reverse influence In step 4, the final text features and the final supermodal features are passed through a cross-modal Transformer to obtain a joint multimodal representation, and then the soft attention mechanism is used to assign different weights to the joint multimodal representation for weighted fusion, and finally multimodal sentiment classification is achieved through a fully connected layer.

2. The Mongolian multimodal sentiment analysis method based on adaptive supermodality and multi-layer attention according to claim 1 is characterized in that: The gated Transformer encoder replaces the existing feedforward layer of the Transformer encoder with a gated information channel. The gated information channel consists of a memory gate and an update gate. The gated Transformer encoder is composed of multiple sublayers, each of which retains the Transformer's multi-head self-attention mechanism. For the input modality, the gated Transformer encoder first extracts features H∈R through a layered multi-head self-attention mechanism. T×d , and then the extracted features are filtered and fused using the memory gate and update gate of the gated information channel, where T is the number of time steps and d is the feature dimension.

3. The Mongolian multimodal sentiment analysis method based on adaptive supermodality and multi-layer attention according to claim 2 is characterized in that: The memory gate is used to store information, retain valuable initial feature information, and combine it with new information to generate a new feature H M , expressed as: Among them, GELU is the activation function, G M Indicates the new content added by the memory gate to the feature, M∈R d×d is the parameter matrix, σ represents the Sigmoid activation function; The update gate is used to update the original features and avoid adding too much noise to the features. Its calculation formula is: Among them, G U represents the update gate, H new Represents the updated features, U∈R d×d is the parameter matrix.

4. The Mongolian multimodal sentiment analysis method based on adaptive supermodality and multi-layer attention according to claim 1, characterized in that: In the adaptive supermodal learning layer, As query vectors, low-level audio features and low-level video features As key vectors, we obtain the similarity matrix α between the text modality and the audio modality, and the similarity matrix β between the text modality and the visual modality, which are expressed as: Among them, softmax represents the weight normalization operation, is a learnable parameter, d k is the dimension of each attention head. In cross-modal attention, Q t is the query vector matrix of the target modality text, K a and K v are the key vector matrices from the audio modality and the visual modality, respectively, representing the feature matrices used to match the query vector matrix; and is a learnable linear transformation weight matrix, which is used to transform Map to the key vector space, H t Mapping to the query vector space and Mapped to the key vector space.

5. The Mongolian multimodal sentiment analysis method based on adaptive supermodality and multi-layer attention according to claim 1, characterized in that: The supermodal feature Updated by weighted audio features and weighted visual features, expressed as: in, represents the supermodal features output by the jth adaptive supermodal learning layer, j∈{1,2,3}, and is a learnable linear transformation weight matrix, which is used to transform and Mapped to the same space as the supermode, V a The eigenvector representing the audio mode, V v A feature vector representing the visual modality.

6. The Mongolian multimodal sentiment analysis method based on adaptive supermodality and multi-layer attention according to claim 1, characterized in that: The supermodal feature Reverse the text mode to update the text features get Expressed as: in, is a learnable weight matrix, which is used to Mapping to the query vector space and Mapping to a key vector space and a value vector space.

7. The Mongolian multimodal sentiment analysis method based on adaptive supermodality and multi-layer attention according to claim 1, characterized in that: In the cross-modal Transformer, a unidirectional cross-modal attention mechanism is used to achieve the fusion of text features and super-modal features. The fusion process is expressed as: H p =CrossTrans(H t ,H hyper )∈R 1×d H0 is the initialization token, H t It is a new text feature obtained by concatenating H0 and the final text feature. hyper It is a new supermodal feature obtained by concatenating H0 with the final supermodal feature. is the final text feature, is the final supermodal feature, H p is the obtained multimodal representation.

8. The Mongolian multimodal sentiment analysis method based on adaptive supermodality and multi-layer attention according to claim 4, characterized in that: The utilizing of the soft attention mechanism to assign different weights to the joint multimodal representation for weighted fusion includes: (a) Calculate the attention score: in, is the joint multimodal representation H p The kth feature, W s ∈R d×1 is a learnable weight matrix, e k yes Attention score; (b) Calculate soft attention weights: a k =softmax(e k ) Among them, α k yes The attention weight of (c) Weighted output representation: Among them, H final is the final weighted feature representation.

Images