Mongolian multi-mode sentiment analysis method based on self-adaptive super-mode and multi-layer attention

Abstract

The invention discloses a Mongolian multi-modal sentiment analysis method based on self-adaptive super-modal and multi-layer attention, and the method comprises the steps: respectively extracting text, audio and video features from multi-modal data, and respectively extracting low-level text features, low-level audio features and low-level video features through a gated Transform encoder; in a self-adaptive super-modal learning layer, low-level text features are used for guiding low-level audio features and low-level video features to generate middle super-modal features, then the middle super-modal features act on the low-level text features, next-scale text features are extracted by using a gated Transform encoder, bidirectional cross-modal attention interaction is realized, and the self-adaptive super-modal learning layer is used for learning the low-level audio features and the low-level video features. According to the Mongolian sentiment analysis method, the final text features and the final super-modal features are obtained through multi-time interaction, a combined multi-modal representation is obtained through cross-modal Transformer, then weighted fusion is conducted through a soft attention mechanism, finally multi-modal sentiment classification is achieved through a full-connection layer, and the Mongolian sentiment analysis effect can be greatly improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence and natural language processing, and in particular relates to a Mongolian multimodal sentiment analysis method based on adaptive supermodality and multi-layer attention. Background Art

[0002] Sentiment analysis is the process of analyzing, processing, summarizing and reasoning data with emotional information. The theory and algorithm construction of sentiment analysis involves artificial intelligence (AI), computer vision (CV) and natural language processing (NLP), and is a multidisciplinary research field.

[0003] Traditional sentiment analysis is usually limited to a single information source, such as text or speech, and it is difficult to fully capture the complex emotional state of human beings. Multimodal sentiment analysis (MSA) aims to integrate different information sources such as vision, speech and text to more accurately identify and understand human emotions. However, compared with sentiment analysis research in mainstream languages ​​such as Chinese and English, Mongolian, as a minority language, still has deficiencies in accuracy and speed due to lack of samples and special language structure. Summary of the invention

[0004] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a Mongolian multimodal sentiment analysis method based on adaptive supermodality and multi-layer attention, so as to solve the problems existing in Mongolian sentiment analysis, such as easy loss of effective information, weak ability of sentiment information extraction, and insufficient fusion caused by lack of deep fusion of multimodal features.

[0005] In order to achieve the above objectives, the present invention first preprocesses the raw data to obtain the initial features of each modality. Secondly, information filtering is implemented through a gated Transformer encoder to obtain low-level text features and initial supermodal features. Then, entering the adaptive supermodal learning module, the low-level text features and the initial supermodal interact through bidirectional cross-modal attention to obtain the final text features and the final supermodal features. Then, a joint multimodal representation is obtained through a cross-modal Transformer, and then a soft attention mechanism is used to assign different weights to the joint multimodal representation, and finally multimodal sentiment classification is achieved through a fully connected layer.

[0006] The technical solution adopted by the present invention can be further described as follows:

[0007] A Mongolian multimodal sentiment analysis method based on adaptive supermodality and multi-layer attention, characterized by comprising the following steps:

[0008] Step 1, extracting text features, audio features and video features from multimodal data consisting of text, audio and video respectively;

[0009] Step 2, using a gated Transformer encoder, respectively extracting low-level text features, low-level audio features, and low-level video features from the text features, audio features, and video features;

[0010] Step 3, using an adaptive supermodal learning module composed of n gated Transformer encoders and n+1 adaptive supermodal learning layers, defining initial supermodal features, in an adaptive supermodal learning layer, using the low-level text features to guide the low-level audio features and low-level video features to generate intermediate supermodal features, then using the intermediate supermodal features to act on the low-level text features and using a gated Transformer encoder to extract next-scale text features, to achieve bidirectional cross-modal attention interaction, the intermediate supermodal features and the next-scale text features continue to perform n bidirectional cross-modal attention interactions, and obtain final text features and final supermodal features;

[0011] Step 4: The final text features and the final supermodal features are used through a cross-modal Transformer to obtain a joint multimodal representation, and then the soft attention mechanism is used to assign different weights to the joint multimodal representation for weighted fusion, and finally multimodal sentiment classification is achieved through a fully connected layer.

[0012] Compared with the prior art, the present invention has the following beneficial effects:

[0013] In the Mongolian multimodal sentiment analysis task, avoiding the loss of effective information and improving the ability to extract sentiment information are key issues in improving sentiment classification performance. In the traditional Transformer model, the feedforward layer is only a linear transformation of the input features, which makes it difficult to distinguish valuable information and easily causes information loss. The Transformer encoder based on the gated information channel constructed by the present invention introduces the gated information channel to replace the feedforward layer, adaptively selects to retain the initial features or the updated features, thereby avoiding the loss of important information, and is more efficient and accurate. In addition, the present invention dynamically learns emotion-related information by introducing an adaptive supermodal learning (AHL) module, generates an intermediate supermodal tag containing less irrelevant information, and increases the weight of emotion-related features, thereby improving the Mongolian emotion information extraction ability, so as to facilitate the final multimodal fusion.

[0014] In Mongolian multimodal sentiment analysis, modalities such as text, vision and audio contain key sentiment information. However, existing multimodal fusion methods fail to effectively capture the complex interactive relationship between modalities, resulting in insufficient feature fusion. In order to solve this problem, the present invention constructs a multi-layer attention mechanism. First, the internal information of each modality is modeled through a multi-head self-attention mechanism, and then a bidirectional cross-modal attention mechanism is used to promote the interaction between text features and visual and audio modalities. Finally, a unidirectional cross-modal attention mechanism is used to further fuse super-modal and text features. This hierarchical dynamic interaction strategy can effectively promote information fusion between modalities, thereby solving the problem of insufficient multimodal feature fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flow chart of the present invention.

[0016] Figure 2 It is a schematic diagram of the overall architecture of the model of the present invention.

[0017] Figure 3 Schematic diagram of the gated Transformer encoder architecture of the present invention.

[0018] Figure 4 It is a schematic diagram of the structure of the gated information channel of the present invention.

[0019] Figure 5 It is a schematic diagram of a dual-scale text feature construction module of the present invention.

[0020] Figure 6 It is a schematic diagram of the structure of the adaptive supermodal learning layer of the present invention.

[0021] Figure 7 Schematic diagram of the multi-head self-attention mechanism of the present invention.

[0022] Figure 8 It is a schematic diagram of the cross-modal attention mechanism of the present invention.

[0023] Fig. 9 Schematic diagram of the soft attention mechanism of the present invention.

[0024] Fig.10 It is a graph showing the changing trend of accuracy during the training cycle of the model of the present invention. DETAILED DESCRIPTION

[0025] The embodiments of the present invention are described in detail below with reference to the accompanying drawings and examples.

[0026] Research on Mongolian multimodal sentiment analysis can help further improve Mongolian speech recognition, text understanding, sentiment analysis and other basic technologies, and provide technical support for its information application. Multimodal sentiment analysis uses data composed of text, images, audio and even video data to obtain information from different modalities, thereby effectively combining multimodal information and more accurately predicting the emotional state of the observed target.

[0027] To this end, the present invention adopts a Mongolian multimodal sentiment analysis method based on adaptive supermodality and multi-layer attention, which mainly improves the effect of Mongolian multimodal sentiment analysis from the following three aspects.

[0028] First, a Transformer encoder model based on gated information channels is constructed to improve the efficiency of filtering redundant information features.

[0029] Specifically, in order to alleviate the problems of information loss and invalid feature transfer in traditional Transformer, the present invention constructs a Transformer encoder model based on gated information channel (GIC), namely gated Transformer encoder (GTE). The Transformer model has become the mainstream model structure in cross-modal tasks. Its core self-attention mechanism can effectively capture the dependencies within the sequence. However, the traditional Transformer encoder has specific shortcomings, especially when processing complex multimodal features, it is often unable to fully utilize the key information of each modality. Its feedforward network is relatively simple, and it fails to realize feature screening and updating, which easily causes information loss and invalid feature transfer.

[0030] In view of the above problems, the present invention introduces a gated information channel to replace the feedforward layer on the basis of the traditional Transformer to achieve information filtering and avoid the loss of effective information. First, the data is output from the position code as the input part of the encoder and enters the multi-head self-attention mechanism layer. The multi-head self-attention mechanism is used to calculate the similarity of different positions, and then the attention distribution is obtained through the Softmax function processing, thereby achieving the focus and understanding of the input sequence. Then, feature screening and updating are performed through the gated information channel. The gated information channel includes a memory gate and an update gate. The memory gate is used to store valuable information, and the additional channel is used to memorize new information; the update gate is used to update the feature fully connected layer.

[0031] Secondly, an audio-visual feature fusion model based on adaptive supermodal representation is constructed, and text features of different scales are used to guide the audio-visual modality, forming a supermodality that complements the text modality and improving the performance of multimodal sentiment analysis.

[0032] Specifically, although Mongolian multimodality can provide richer emotional information for sentiment analysis than unimodality, the potential emotional irrelevance and conflicting information across modalities weakens the ability to extract emotional information, thus hindering further performance improvement. Due to emotional irrelevance, information from different modalities may be ambiguous and conflicting, especially from non-dominant modalities such as audio and visual modalities. Such interference information greatly limits the performance of multimodal sentiment analysis methods.

[0033] In view of the above problems, the present invention constructs an audio-visual feature fusion model based on adaptive supermodal representation, uses text features to guide the audio-visual modality, forms a supermodality that complements the text modality, and then uses the supermodality to influence the text features, so that the text features of the next scale can better guide the audio-visual modality, thereby improving the performance of multimodal sentiment analysis and alleviating the problem of weak Mongolian sentiment information extraction ability. First, each modality is converted into a unified form by using a gated Transformer encoder with an initialization mark. This operation not only suppresses the redundant information between modalities, but also compresses the length of long sequences, which facilitates efficient model calculation. Then, an adaptive supermodal learning (AHL) module is introduced, in which text features of different scales are used to guide the visual and audio modalities to generate intermediate supermodal tags, which contain less information unrelated to emotion, improve the Mongolian sentiment information extraction ability, and facilitate the final multimodal fusion.

[0034] Thirdly, a Mongolian multimodal sentiment analysis model based on a multi-layer attention mechanism is constructed to achieve deep fusion of multimodal information and improve the performance and generalization of Mongolian sentiment analysis.

[0035] Specifically, when using multimodal technology for sentiment analysis tasks, mainstream multimodal sentiment analysis methods based on deep learning often encounter some challenges, among which the problems of insufficient feature fusion between modalities and unreasonable weight distribution of each modality are particularly obvious.

[0036] In view of the above problems, the present invention constructs a Mongolian multimodal sentiment analysis model based on a multi-layer attention mechanism, and constructs a multi-layer attention mechanism for multimodal sentiment analysis. Through the multi-level interaction of multi-head self-attention mechanism, cross-modal attention mechanism and soft attention mechanism, the deep fusion of internal information and multimodal information is realized, and the problem of insufficient feature fusion between modalities and unreasonable weight distribution of each modality is alleviated. First, the information of each modality is modeled using a multi-head self-attention mechanism to obtain the initial features of each modality. In the adaptive supermodal learning layer, a bidirectional cross-modal attention mechanism is used to update the supermodal features by calculating the relationship between the obtained text features and the remaining two modalities. The supermodal features further affect the text features through the reverse path, so that the high-level features of the text are more fully learned. Then, the unidirectional cross-modal attention mechanism is used in the cross-modal transformer to promote interaction between modalities, realize the fusion of text features and supermodal features, and obtain a joint multimodal representation. Finally, the soft attention mechanism is used to assign different weights to the joint multimodal representation, so that the model pays more attention to the task-related modal information in the fused representation, and finally realizes multimodal sentiment classification through the fully connected layer.

[0037] Hereinafter, the method of the present invention is further described in detail in combination with the above principles.

[0038] like Figure 1 and Figure 2 As shown, the Mongolian multimodal sentiment analysis method based on adaptive supermodality and multi-layer attention of the present invention mainly includes the following steps:

[0039] Step 1: For multimodal data, extract the initial features of each modality respectively.

[0040] In the present invention, the multimodal data consists of text, audio and video, and the extracted features include text features, audio features and video features. In a standard multimodal data set, data alignment is performed between text, audio and video using their respective IDs.

[0041] In terms of functional modules, this step can be defined as a preprocessing module. Specifically, mBERT, Librosa, and OpenFace can be used in this module to process text, audio, and video data respectively, and extract the initial features of each modality. The specific processing method is as follows:

[0042] (a) Text processing

[0043] mBERT is a multilingual pre-trained model based on the BERT architecture, which is specially designed for processing texts in multiple languages, including Mongolian. When using mBERT to process Mongolian text, the text is segmented and encoded by loading the multilingual BERT model, and the processed input is passed into the model to extract embedded features. These feature vectors effectively express the semantic information of the text.

[0044] (b) Audio Processing

[0045] First, use the Librosa library to read audio data and obtain the sampling rate and audio signal. Then, normalize the audio signal to ensure that the signal is in the range of [-1,1]; extract audio features such as MFCC (Mel Frequency Cepstral Coefficients) and spectral features. Finally, use the Short Time Fourier Transform (STFT) to perform frequency domain analysis on the audio signal to view the spectral characteristics of the signal.

[0046] (c) Video Processing

[0047] First, use OpenCv to read the video frames and select certain key frames as images for feature extraction. Then use OpenFace for face detection and feature extraction. Once the face is detected, OpenFace will extract a series of facial features, including facial key points and facial expression features.

[0048] The original data is preprocessed to obtain the initial features of each mode. Where m∈{t,a,v}, t,a,v refer to text, audio and video respectively.

[0049] Step 2: Use the gated Transformer encoder to obtain the low-level text features, low-level audio features, and low-level video features from the text features, audio features, and video features obtained in step 1.

[0050] In this step, each initial feature obtained after data preprocessing is further extracted through a gated Transformer encoder. The gated Transformer encoder constructed by the present invention replaces the feedforward layer in the traditional Transformer encoder with a gated information channel, so that the model can selectively update the features, thereby enhancing the accuracy of feature expression and avoiding excessive information interference. The structure of the gated Transformer encoder is as follows Figure 3 As shown in the figure, N× represents that the gated Transformer encoder is stacked N times. The structure shown in the figure is a sub-layer. The gated Transformer encoder is composed of multiple stacked sub-layers, and each sub-layer retains the multi-head self-attention mechanism of the Transformer.

[0051] The improved model still retains the Transformer's multi-head self-attention mechanism to calculate the similarity between different positions of the input data. First, the data is position-encoded as input and enters the multi-head self-attention layer to calculate the dependencies between each position in the sequence. The attention weights are normalized by the Softmax function, so that the model can focus on more important sequence positions.

[0052] The present invention sets a gated information channel to filter out unnecessary data features. The gated information channel consists of two parts: a memory gate and an update gate. The memory gate is used to store valuable initial feature information, while the additional channel is used to memorize new information. The update gate is used to update the feature fully connected layer. The present invention designs a network structure similar to GRU, such as Figure 4 shown.

[0053] In the gated Transformer encoder, the features H∈R of each input modality are first extracted through a hierarchical multi-head self-attention mechanism. T×d , where T is the number of time steps and d is the feature dimension. Then, the memory gate and update gate in the gated information channel are used to effectively filter and fuse the extracted features.

[0054] Memory gates are used to store information and add some new content to the feature:

[0055]

[0056] Among them, GELU is the activation function, G M Represents the new content added by the memory gate to the feature, M∈R d×d is the parameter matrix, and σ represents the Sigmoid activation function. The memory gate retains valuable initial feature information and combines it with new information to generate new features H M :

[0057]

[0058] The update gate is used to update the original features and avoid adding too much noise to the features. Its calculation formula is:

[0059]

[0060] Among them, G U represents the update gate, H new Represents the updated features, U∈R d×d is the parameter matrix. This formula ensures the flexibility of the update process, that is, retaining the key information in the original features while making necessary updates to some features.

[0061] Through such a design, the model can dynamically choose to retain the initial features or the updated features, thereby avoiding the loss of important information. At the same time, the gated information channel effectively avoids the interference of redundant noise information on the features.

[0062] Step 3, using an adaptive supermodal learning module consisting of n gated Transformer encoders and n+1 adaptive supermodal learning layers, define initial supermodal features, in an adaptive supermodal learning layer, use the low-level text features to guide the low-level audio features and low-level video features to generate intermediate supermodal features, then use the intermediate supermodal features to act on the low-level text features and use a gated Transformer encoder to extract next-scale text features to achieve bidirectional cross-modal attention interaction, the intermediate supermodal features and the next-scale text features continue to perform n bidirectional cross-modal attention interactions to obtain final text features and final supermodal features.

[0063] After filtering the modal information, in this step, the Adaptive Supermodal Learning (AHL) module is further used to learn a refined supermodal representation containing relevant information and highly complementary text features. The AHL module consists of two gated Transformer encoders (GTE) and three AHL layers, which aims to learn text features at different scales and adaptively learn a supermodal feature from the visual and audio modalities under the guidance of the text features. The supermodality is then used to influence the text features so that the text features at the next scale can better guide the audiovisual modality, thereby improving the performance of MSA.

[0064] This step is mainly divided into two parts, namely the construction of dual-scale text features and the related execution process of the adaptive supermodal learning layer.

[0065] (1) Construction of dual-scale text features.

[0066] Initial text features extracted by mBERT It will be passed to the subsequent GTE layer for further semantic modeling and feature enhancement, and the self-attention mechanism will be used to capture the long-distance dependencies in the text.

[0067] Will is defined as low-level text features. With this feature, the present invention introduces two GTE layers to learn medium-scale and high-scale (i.e. ), that is, in this embodiment, n=2. Figure 5 As shown in the figure, unlike the GTE layer in the modal information filtering stage (which passes basic information to the initialized token) (this stage requires 3 GTEs, one for each modality), the GTE layer in this stage directly constructs dual-scale text features for text features as follows:

[0068]

[0069] Among them, i∈{2,3}, are text features at different scales, with a size of T×d, and The i-th layer gated Transformer encoder and corresponding parameters for text feature learning, is the corresponding supermodal feature H hyper Text features after reverse influence. In practice, 8-head attention can be used to model the information of each modality.

[0070] The two-layer GTE extracts the dual-scale features of the text layer by layer. Each layer of GTE uses the self-attention mechanism to model the text sequence, gradually enhancing the understanding of the context and the ability of semantic modeling. The output of the last layer of GTE It contains text features from local to global, and goes through multiple layers of abstraction and filtering.

[0071] (2) Related execution process of the adaptive supermodal learning layer.

[0072] In the adaptive supermodal learning layer, for text features of different scales First initialize a supermodal feature Then, the bidirectional cross-modal attention mechanism is used to update the relationship between the obtained text features and the audio features and video features. And use the updated supermodal features Reverse impact Get the text features after reverse influence

[0073] Specifically, Figure 6 As shown, the extracted As the query vector query, the low-level audio features obtained through GTE As the key vector key, we can get the similarity matrix α between the text modality and the audio modality, expressed as:

[0074]

[0075] Among them, softmax represents the weight normalization operation. is a learnable parameter, d k is the dimension of each attention head. In practice, we use 8 attention heads and divide d k Set to 16.

[0076] In cross-modal attention, Q t is the query vector matrix of the target modality text, Ka is the key vector matrix from the audio modality, representing the feature matrix used to match the query vector matrix. The T in the upper right corner of the variable in the formula represents the transpose of this variable. and is a learnable linear transformation weight matrix, which is used to transform Mapping to the key vector space and transforming H t Mapped to the query vector space.

[0077] Similarly, low-level video features As the key vector, similar to α, the similarity matrix β between the textual modality and the visual modality can be expressed as:

[0078]

[0079] Among them, K v is the key vector matrix from the visual modality, representing the feature matrix used to match the query vector matrix, is a learnable linear transformation weight matrix used to transform Mapped to the key vector space.

[0080] Therefore, the supermodal feature It can be updated by weighted audio features and weighted visual features, expressed as:

[0081]

[0082] in, represents the supermodal features output by the jth adaptive supermodal learning layer, j∈{1,2,3}, and is a learnable linear transformation weight matrix, which is used to transform and Mapped to the same space as the supermode, V a The eigenvector representing the audio mode, V v A feature vector representing the visual modality.

[0083] Updated supermodal features Need to act inversely on the text mode to further update the text features get

[0084]

[0085] in, is a learnable weight matrix, which is used to transform Mapping to the query vector space and Mapping to a key vector space and a value vector space.

[0086] In step 4, the final text features and the final supermodal features are passed through the cross-modal Transformer to obtain a joint multimodal representation, and then the soft attention mechanism is used to assign different weights to the joint multimodal representation for weighted fusion, and finally multimodal sentiment classification is achieved through the fully connected layer.

[0087] In the overall architecture of the present invention, a multi-layer attention mechanism is adopted, including a multi-head self-attention mechanism, a cross-modal attention mechanism, and a soft attention mechanism, which ultimately achieves a deep fusion of internal information and multimodal information. They are introduced as follows.

[0088] (1) Multi-head self-attention mechanism.

[0089] The multi-head self-attention mechanism models single modality information. For different modalities such as text, audio, and vision, the multi-head self-attention mechanism is used to model their features. The multi-head self-attention mechanism of the present invention is implemented in the gated Transformer encoder. The multi-head self-attention mechanism can capture the internal correlation features of a single modality and help the model understand the important information in each modality. The structure of the multi-head self-attention mechanism is as follows Figure 7 shown.

[0090] (a) Linear transformation.

[0091] For each head, first a linear transformation is performed through the learned weight matrix to generate Q m , K m and V m (subscript m represents the mode, m belongs to {t,a,v}, Q (query vector), K (key vector), V (value vector) of each mode):

[0092]

[0093] where n is the index of the head, m∈{t,a,v} represents the text, audio and visual modalities respectively, and is the corresponding weight matrix, Enter a sequence for a modality.

[0094] (b) Calculate the attention score

[0095] For each head, calculate the attention matrix:

[0096]

[0097] Among them, d k is the dimension of each attention head.

[0098] (c) Output of multi-head self-attention

[0099] The output of the multi-head attention mechanism is the concatenation of the outputs of all heads and then a linear transformation. There are h heads, and the final multi-head attention output is:

[0100] MultiHead(Q m ,K m ,V m )=Copncat(head1,head2,…,head h )W O

[0101] The output of each header is:

[0102]

[0103] W O is the weight matrix used to transform the concatenated result into output.

[0104] (2) Cross-modal attention mechanism

[0105] (a) Unidirectional cross-modal attention mechanism

[0106] In the cross-modal Transformer, the present invention adopts a unidirectional cross-modal attention mechanism to achieve the fusion of text features and super-modal features. Through this mechanism, the information interaction between the features of each modality is further promoted, so that the text modality can absorb the effective features of other modalities, further enhancing the effect of modal fusion. The structure of the cross-modal attention mechanism is as follows Figure 8 shown.

[0107]

[0108] in, and The matrix K and the weight matrix V represent the supermodal features respectively. hyper is a value vector consisting of the final supermodal features H hyper Mapping a matrix by value The final supermodal feature H hyper Mapped to the value space.

[0109] In the multimodal fusion of the present invention, the initialized token is represented by H0∈R 1×d First, H0 and and Concatenate and concatenate to get a new text feature H t and a new supermodal feature H hyper , and then use the cross-modal transformer to pass the necessary joint and complementary information to these tokens. In practice, the cross-modal transformer transforms the text feature H t(as K) and the supermodal feature H hyper (as key and value) are fused together to obtain a joint multimodal representation H p ∈R 1×d Used for subsequent sentiment analysis.

[0110] The cross-modal fusion transformer is represented as CrossTrans, so the fusion process can be expressed as:

[0111]

[0112] (b) Bidirectional cross-modal attention mechanism

[0113] In the adaptive supermodal learning layer, a bidirectional cross-modal attention mechanism is used to calculate the relationship between text features and the remaining two modal (audio and visual) features to update the supermodal features. At the same time, the updated supermodal features are fed back to the text features through the reverse path, so that the high-level features of the text can more fully learn the supplementary information of other modalities, thereby achieving deep fusion.

[0114] (3) Soft Attention Mechanism

[0115] The present invention uses a soft attention mechanism to assign different weights to the combined multimodal representation for weighted fusion, and obtains the final weighted feature representation H final This module assigns different weights to different features according to their importance in sentiment classification, so that it can more accurately rely on useful modal information when making decisions. Fig. 9 As shown, the process is mainly divided into the following three steps:

[0116] (a) Calculate the attention score:

[0117]

[0118] in, is the joint multimodal representation H p The kth feature of s ∈R d×1 is a learnable weight matrix, e k yes Attention score;

[0119] (b) Calculate soft attention weights:

[0120] α k =softmax(e k )

[0121] Among them, α k yes The attention weight of

[0122] (c) Weighted output representation:

[0123]

[0124] In order to verify the effect of the method of the present invention, the present invention conducted a test experiment, which used a Mongolian multimodal emotion dataset including 2100 multimodal samples and 7 discrete emotions, including happiness, anger, sadness, surprise, fear, disgust and neutral (i.e., a relatively stable voice without emotion), and 300 sentences for each emotion. 60% of the data for each emotion category was randomly selected as a training set, 10% as a validation set, and 30% as a test set.

[0125] The present invention intends to use accuracy, precision, recall and F1 value as indicators for evaluating model performance. Specifically, the calculation formula for accuracy is: The calculation formula for accuracy is: The calculation formula for recall is: The calculation formula of F1 value is: Among them, Accuracy represents accuracy, that is, the ratio of the number of samples correctly predicted by the model to the total number of samples; Precision represents precision, that is, the ratio of samples predicted by the model to be positive that are actually positive; Recall represents recall, that is, the ratio of samples actually positive that are correctly predicted to be positive; F1 represents F1 value, that is, the harmonic mean of precision and recall, which comprehensively reflects the performance of both. TP represents the number of samples that are actually positive and predicted to be positive by the model. FN represents the number of samples predicted by the model to be negative but are actually positive. FP represents the number of samples predicted by the model to be positive but are actually negative. TN represents the number of samples that are actually negative and predicted to be negative by the model. The higher the scores of accuracy, precision, recall and F1 value, the better the performance of the sentiment analysis model. As shown in Table 1, the confusion matrix required for the calculation is given:

[0126] Table 1 Confusion matrix table

[0127] Predicted as positive Predicted as negative Actual positive example TP FN Negative example FP TN

[0128] The accuracy trend in 100 training cycles is as follows Fig.10As shown. Both the training accuracy and the test accuracy increase rapidly within the first 20 epochs, which is the initial stage of model learning, indicating that the model is effectively fitting the training data and can also perform well on the test set. In the 40th epoch, the test accuracy reached the highest, which was 92.38%. Later, the test accuracy gradually stabilized between 0.85-0.9. It can be seen that the Mongolian multimodal sentiment analysis model based on adaptive supermodal representation and multi-layer attention mechanism proposed in the present invention has a good effect. The above experiments can fully demonstrate the effectiveness of the method proposed in the present invention, and the method can provide theoretical and practical references for Mongolian multimodal sentiment analysis tasks.

Claims

1. A Mongolian multimodal sentiment analysis method based on adaptive supermodality and multi-layer attention, characterized in that: The steps include: Step 1, extracting text features, audio features and video features from multimodal data consisting of text, audio and video respectively; Step 2, using a gated Transformer encoder, respectively extracting low-level text features, low-level audio features, and low-level video features from the text features, audio features, and video features; Step 3, using an adaptive supermodal learning module composed of n gated Transformer encoders and n+1 adaptive supermodal learning layers, defining initial supermodal features, in an adaptive supermodal learning layer, using the low-level text features to guide the low-level audio features and low-level video features to generate intermediate supermodal features, then using the intermediate supermodal features to act on the low-level text features and using a gated Transformer encoder to extract next-scale text features, to achieve bidirectional cross-modal attention interaction, the intermediate supermodal features and the next-scale text features continue to perform n bidirectional cross-modal attention interactions, and obtain final text features and final supermodal features; Step 4: The final text features and the final supermodal features are used through a cross-modal Transformer to obtain a joint multimodal representation, and then the soft attention mechanism is used to assign different weights to the joint multimodal representation for weighted fusion, and finally multimodal sentiment classification is achieved through a fully connected layer.

2. The Mongolian multimodal sentiment analysis method based on adaptive supermodality and multi-layer attention according to claim 1 is characterized in that: The gated Transformer encoder replaces the existing feedforward layer of the Transformer encoder with a gated information channel, which is composed of a memory gate and an update gate. The gated Transformer encoder is composed of multiple stacked sublayers, each of which retains the multi-head self-attention mechanism of the Transformer. For the input modality, the gated Transformer encoder first extracts the feature H∈R through the hierarchical multi-head self-attention mechanism. T×d , and then use the memory gate and update gate of the gated information channel to filter and fuse the extracted features, where T is the number of time steps and d is the feature dimension.

3. The Mongolian multimodal sentiment analysis method based on adaptive supermodality and multi-layer attention according to claim 2 is characterized in that: The memory gate is used to store information, retain valuable initial feature information, and combine it with new information to generate a new feature H. M , expressed as: Among them, GELU is the activation function, G M represents the new content added by the memory gate to the feature, M∈R d×d is the parameter matrix, σ represents the Sigmoid activation function; The update gate is used to update the original features and avoid adding too much noise to the features. Its calculation formula is: Among them, G U represents the update gate, H new Represents the updated features, U∈R d×d is the parameter matrix.

4. The Mongolian multimodal sentiment analysis method based on adaptive supermodality and multi-layer attention according to any one of claims 1 to 3, characterized in that: n=2, the adaptive supermodal learning module constructs dual-scale text features in the following way: Among them, i∈{2,3}, are text features at different scales, with a size of T×d, is a low-level text feature, and The i-th layer gated Transformer encoder and corresponding parameters for text feature learning, is the corresponding supermodal feature H hyper Text features after reverse influence.

5. The Mongolian multimodal sentiment analysis method based on adaptive supermodality and multi-layer attention according to claim 4 is characterized in that: The adaptive supermodal learning layer is used for text features of different scales. First initialize a supermodal feature Then, the bidirectional cross-modal attention mechanism is used to update the relationship between the obtained text features and the audio features and video features. And use the updated supermodal features Reverse impact Get the text features after reverse influence 6. The Mongolian multimodal sentiment analysis method based on adaptive supermodality and multi-layer attention according to claim 5 is characterized in that: In the adaptive supermodal learning layer, As query vector, low-level audio features and low-level video features As key vectors, we obtain the similarity matrix α between the text modality and the audio modality, and the similarity matrix β between the text modality and the visual modality, expressed as: Among them, softmax represents the weight normalization operation. is a learnable parameter, d k is the dimension of each attention head. In cross-modal attention, Q t is the query vector matrix of the target modality text, K a and K v are the key vector matrices from the audio modality and the visual modality, respectively, representing the feature matrices used to match the query vector matrix; and is a learnable linear transformation weight matrix, which is used to transform Mapping to the key vector space, H t Mapping to the query vector space and Mapped to the key vector space.

7. The Mongolian multimodal sentiment analysis method based on adaptive supermodality and multi-layer attention according to claim 5, characterized in that: The supermodal feature Updated by weighted audio features and weighted visual features, expressed as: in, represents the supermodal features output by the jth adaptive supermodal learning layer, j∈{1,2,3}, and is a learnable linear transformation weight matrix, which is used to transform and Mapped to the same space as the supermode, V a The eigenvector representing the audio mode, V v A feature vector representing the visual modality.

8. The Mongolian multimodal sentiment analysis method based on adaptive supermodality and multi-layer attention according to claim 5, characterized in that: The supermodal feature Reverse the text mode to update the text features get It is expressed as: in, is a learnable weight matrix, which is used to transform Mapping to the query vector space and Mapping to a key vector space and a value vector space.

9. The Mongolian multimodal sentiment analysis method based on adaptive supermodality and multi-layer attention according to claim 5, characterized in that: In the cross-modal Transformer, a unidirectional cross-modal attention mechanism is used to achieve the fusion of text features and super-modal features. The fusion process is expressed as: H p =CrossTrans(H t ,H hyper )∈R 1×d H0 is the initialization token, H t It is a new text feature obtained by concatenating H0 with the final text feature. hyper It is a new supermodal feature obtained by concatenating H0 with the final supermodal feature. is the final text feature, is the final supermodal feature, H p is the resulting multimodal representation.

10. The Mongolian multimodal sentiment analysis method based on adaptive supermodality and multi-layer attention according to claim 6, characterized in that: The method of using a soft attention mechanism to assign different weights to the joint multimodal representation for weighted fusion includes: (a) Calculate the attention score: Among them, H pk is the joint multimodal representation H p The kth feature of s ∈R d×1 is a learnable weight matrix, e k Yes H pk Attention score; (b) Calculate soft attention weights: a k =softmax(e k ) Among them, α k Yes H pk The attention weight of (c) Weighted output representation: Among them, H final is the final weighted feature representation.

Images