Multi-modal sentiment analysis method and system

Abstract

The invention discloses a multi-modal sentiment analysis method and system, and relates to the technical field of sentiment analysis, and the method comprises the steps: carrying out the modal feedback interaction of a text single-modal feature, an audio single-modal feature and a visual single-modal feature of feature extraction of multi-modal sentiment data, generating text feedback features, audio feedback features and visual feedback features; encoding the text feedback feature, the audio feedback feature and the visual feedback feature through a language encoder and a non-language encoder, and determining the text feedback feature and a mixed encoding feature formed by splicing the audio encoding feature and the visual encoding feature; performing feature screening interaction by adopting the text coding features and the mixed coding features, and determining text screening features, mixed screening features and cross screening features; and outputting an emotion analysis result based on the text screening features, the mixed screening features and the cross screening features. Based on the above scheme, information mixing and intermodal interference can be prevented, and emotion analysis reliability can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of sentiment analysis, and in particular to a multimodal sentiment analysis method and system. Background Art

[0002] The method based on multimodal sentiment analysis combines data sources from multiple modalities, including verbal and nonverbal. In classroom communication, students' texts and language expressions (such as questions and answers) belong to verbal modalities, while facial expressions, body language and tones in audio belong to nonverbal modalities. By combining verbal and nonverbal sentiment information analysis, teachers can understand students' emotional changes more comprehensively and accurately. These emotional changes reflect students' participation, understanding, concentration and other aspects of learning status, which can provide strong support for teachers to adjust teaching strategies in a timely manner, improve classroom interaction and improve teaching effectiveness.

[0003] In traditional multimodal sentiment analysis methods, features are usually extracted from each modality first, then fused and the sentiment classification results are output through a classifier. However, in the special context of the classroom, students' language modal information often comes from after-class homework and other forms, and non-language modalities are mainly obtained directly from current classroom interactions. When the above traditional methods are applied to special contexts such as classrooms, they are prone to problems such as insufficient extraction of common information between modalities and noise influence, resulting in low reliability of sentiment analysis. Summary of the invention

[0004] The present invention provides a multimodal sentiment analysis method and system, which improves the technical problem of low sentiment analysis reliability of traditional multimodal sentiment analysis methods.

[0005] A first aspect of the present invention provides a multimodal sentiment analysis method, comprising:

[0006] Acquire multimodal emotion data, perform feature extraction on the multimodal emotion data, and output text unimodal features, audio unimodal features, and visual unimodal features;

[0007] Performing modal feedback interaction based on the text unimodal feature, the audio unimodal feature, and the visual unimodal feature to generate text feedback features, audio feedback features, and visual feedback features;

[0008] The text feedback feature is encoded by a language encoder to determine a text encoding feature, and the audio feedback feature and the visual feedback feature are respectively input into a non-language encoder to construct corresponding audio encoding features and visual encoding features and then splice them into a mixed encoding feature;

[0009] Using the text encoding feature and the mixed encoding feature to perform feature screening interaction, and determine text screening features, mixed screening features and cross screening features;

[0010] The text screening feature, the mixed screening feature and the cross screening feature are concatenated and input into a linear layer for sentiment classification, and a sentiment analysis result is output.

[0011] Optionally, the acquiring of multimodal emotion data, performing feature extraction on the multimodal emotion data, and outputting text unimodal features, audio unimodal features, and visual unimodal features, includes:

[0012] Obtain text sentiment data, audio sentiment data, and visual sentiment data;

[0013] Extracting original text features of the text sentiment data based on a text feature extraction model;

[0014] Extracting original audio features of the audio emotion data through an audio feature extraction model;

[0015] Using a visual feature extraction model to extract the original visual features of the visual emotion data;

[0016] The original text features, the original audio features and the original visual features are sequentially input into a unimodal feature extractor for feature extraction, and text unimodal features, audio unimodal features and visual unimodal features are output.

[0017] Optionally, performing modal feedback interaction based on each of the text unimodal feature, the audio unimodal feature and the visual unimodal feature to generate text feedback features, audio feedback features and visual feedback features includes:

[0018] Using a Sigmoid activation function to perform nonlinear mapping on the text unimodal feature, the audio unimodal feature, and the visual unimodal feature, respectively, to generate a text feedback mask, an audio feedback mask, and a visual feedback mask;

[0019] The feedback masks of any two modalities are averaged and then multiplied element-wise with the unimodal features of the third modality to determine the text feedback features, audio feedback features, and visual feedback features.

[0020] Optionally, encoding the text feedback feature by a language encoder to determine the text encoding feature includes:

[0021] After encoding the absolute position of the text feedback feature, the feature is added element by element to the text feedback feature to determine the text position feature;

[0022] After layer normalization processing is performed on the text position features, they are mapped into a text query matrix, a text key matrix, and a text value matrix, and multi-head self-attention operations are performed to output text attention features;

[0023] Performing a residual connection between the text position feature and the text attention feature and then performing layer normalization to obtain a text normalization feature;

[0024] The text normalization features are subjected to feature conversion through a feedforward neural network layer to output text encoding features.

[0025] Optionally, the processing of the non-language encoder includes:

[0026] Linearly projecting the non-language input features input into the non-language encoder into a first non-language query matrix, a first non-language key matrix, and a first non-language value matrix, and performing a multi-head self-attention operation to determine a non-language multi-head attention feature;

[0027] Performing residual connection between the non-language input feature and the non-language multi-head attention feature and then performing layer normalization to generate a non-language multi-head normalized feature;

[0028] Mapping the text feedback feature into a second non-language query matrix and mapping the non-language multi-head normalized feature into a second non-language key matrix and a second non-language value matrix, and then performing a cross-attention operation to construct a non-language cross-attention feature;

[0029] Performing a residual connection on the text feedback feature and the non-language cross-attention feature and then performing layer normalization to generate a non-language cross-normalized feature;

[0030] After performing element-by-element multiplication of the non-language input feature and the non-language cross-normalized feature, a non-language gating value is calculated using a Sigmoid activation function;

[0031] After concatenating the non-language input feature and the non-language cross-normalized feature, multiplying the non-language gating value element by element to output a non-language gating feature;

[0032] The non-language gating feature is subjected to feature transformation through a feed-forward neural network layer to determine a non-language output feature.

[0033] Optionally, the adopting the text encoding feature and the mixed encoding feature to perform feature screening interaction to determine the text screening feature, the mixed screening feature and the cross screening feature includes:

[0034] The text encoding feature and the mixed encoding feature are segmented respectively to obtain corresponding first features and second features;

[0035] Input each of the first features into the LSTM layer for feature extraction and then multiply the first features element by element with the associated second features to generate a third feature;

[0036] A Sigmoid activation function is used to map each of the third features to construct a screening gating value;

[0037] Respectively multiplying the text encoding feature and the mixed encoding feature by the corresponding screening gating values ​​element by element, and outputting the text screening feature and the mixed screening feature;

[0038] A cross-attention operation is performed with the text screening feature projection as a query matrix and the mixed screening feature projection as a key matrix and a value matrix to determine the cross-screening feature.

[0039] A second aspect of the present invention provides a multimodal sentiment analysis system, comprising:

[0040] A feature acquisition module, used to acquire multimodal emotion data, perform feature extraction on the multimodal emotion data, and output text unimodal features, audio unimodal features, and visual unimodal features;

[0041] A feedback module, configured to perform modal feedback interaction based on the text unimodal feature, the audio unimodal feature and the visual unimodal feature, and generate text feedback features, audio feedback features and visual feedback features;

[0042] A language and non-language encoding module, used to encode the text feedback features through a language encoder to determine text encoding features, and input the audio feedback features and the visual feedback features into a non-language encoder respectively, construct corresponding audio encoding features and visual encoding features, and then splice them into a mixed encoding feature;

[0043] A language and non-language mixed module, used to use the text coding feature and the mixed coding feature to perform feature screening interaction, and determine text screening features, mixed screening features and cross screening features;

[0044] The classification module is used to splice the text screening features, the mixed screening features and the cross screening features, input them into the linear layer for sentiment classification, and output the sentiment analysis results.

[0045] A third aspect of the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the multimodal sentiment analysis method as described in any one of the above items.

[0046] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed, the multimodal sentiment analysis method as described in any one of the above items is implemented.

[0047] A fifth aspect of the present invention provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the multimodal sentiment analysis method as described in any one of the above items.

[0048] It can be seen from the above technical solutions that the present invention has the following advantages:

[0049] The above scheme of the present invention provides a multimodal sentiment analysis method, including: acquiring multimodal sentiment data, performing feature extraction on the multimodal sentiment data, and outputting text unimodal features, audio unimodal features, and visual unimodal features; performing modal feedback interaction based on the text unimodal features, audio unimodal features, and visual unimodal features to generate text feedback features, audio feedback features, and visual feedback features; encoding the text feedback features through a language encoder to determine the text coding features, and inputting the audio feedback features and the visual feedback features into a non-language encoder respectively, constructing corresponding audio coding features and visual coding features, and then splicing them into mixed coding features; performing feature screening interaction using text coding features and mixed coding features to determine text screening features, mixed screening features, and cross screening features; splicing the text screening features, mixed screening features, and cross screening features and then inputting them into a linear layer for sentiment classification, and outputting a sentiment analysis result. Based on the above scheme, after feature extraction of multimodal sentiment data, information interaction and fusion between different modalities are performed, and the language modality, i.e., text modality, and non-language modalities, i.e., audio modality and visual modality, are encoded and feature screened and fused respectively. This allows the feature extraction methods of the language modality and non-language modality to adapt to their respective characteristics, prevents information mixing and interference between modalities, and helps improve the reliability of sentiment analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0051] Figure 1 A flowchart of the steps of a multimodal sentiment analysis method provided by an embodiment of the present invention;

[0052] Figure 2 A network architecture diagram of a multimodal sentiment analysis network provided by an embodiment of the present invention;

[0053] Figure 3 A schematic diagram of a multimodal sentiment analysis network provided by an embodiment of the present invention;

[0054] Figure 4 A schematic diagram of the structure of a speech encoder provided by an embodiment of the present invention;

[0055] Figure 5 A schematic diagram of the structure of a non-language encoder provided by an embodiment of the present invention;

[0056] Figure 6 A schematic diagram of the structure of the LSTM gating block provided in an embodiment of the present invention;

[0057] Figure 7 A structural block diagram of a multimodal sentiment analysis system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The embodiments of the present invention provide a multimodal sentiment analysis method and system for improving the technical problem of low sentiment analysis reliability of traditional multimodal sentiment analysis methods.

[0059] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] See also Figure 1 , Figure 1 A flowchart of the steps of a multimodal sentiment analysis method provided by an embodiment of the present invention.

[0061] The present invention provides a multimodal sentiment analysis method, comprising:

[0062] Step 101: Acquire multimodal emotion data, perform feature extraction on the multimodal emotion data, and output text unimodal features, audio unimodal features, and visual unimodal features.

[0063] Step 101 includes the following sub-steps:

[0064] S11. Obtain text emotion data, audio emotion data, and visual emotion data.

[0065] It should be noted that multimodal sentiment data can be obtained from the CMU-MOSI and CMU-MOSEI datasets; the CMU-MOSI dataset is a widely used benchmark dataset in multimodal sentiment analysis (MSA) research. The dataset contains a series of YouTube monologues in which the narrator expresses personal opinions on topics such as movies. There are a total of 93 videos covering 89 different speakers. MOSI contains 2198 subjective sentence-video clips that have been manually annotated with a rating range of [ ,3], where + indicates strong negative emotions, and +3 indicates strong positive emotions. The CMU-MOSEI dataset is an improvement on MOSI, with more sentences, richer samples, and a diversity of speakers and topics. The dataset contains 23,453 annotated video clips (sentences) from 5,000 videos, covering 1,000 different speakers and 250 different topics.

[0066] In this embodiment, the multimodal emotion data mainly involves text emotion data of the text mode (Text, t) as a language mode, audio emotion data of the audio mode (Audio, a) as a non-language mode, and visual emotion data of the visual mode (Vision, v). The multimodal emotion data is input as follows: Figure 2 The trained multimodal sentiment analysis network is processed as shown in the figure. First, the original features of the sentiment data are extracted using the feature extraction model. , where the data modality , is the sequence length, is the sequence dimension.

[0067] Optionally, the training process of the multimodal sentiment analysis network includes: obtaining training multimodal sentiment data and real sentiment analysis results; inputting the training multimodal sentiment data into an untrained multimodal sentiment analysis network, and outputting predicted sentiment analysis results; using the real sentiment analysis results and the predicted sentiment analysis results to calculate the loss function value, iteratively optimizing the untrained multimodal sentiment analysis network according to the loss function value until the loss function value converges, and determining the trained multimodal sentiment analysis network; this embodiment performs classification tasks based on the real sentiment analysis results and the predicted sentiment analysis results to calculate the loss function value, and in the specific implementation, a binary cross entropy loss function based on mean square error is used for calculation, and for the batch The loss function value is calculated as follows:

[0068] ;

[0069] In the formula, is the loss function value, is the batch size when using batch training method, For the training samples, is the real sentiment analysis result. To predict sentiment analysis results.

[0070] S12. Extract original text features of text sentiment data based on a text feature extraction model.

[0071] It should be noted that all videos in the dataset have both Chinese and English versions. This embodiment only uses the Chinese version. Figure 2 As shown, the text feature extraction model can use a pretrained BERT model (Pretrained BERT), such as the Chinese BERT-base-uncased model; the native sequence as text sentiment data is recorded as , add two unique tags "[CLS]" and "[SEP]" to indicate the beginning and end of each sentence, and then use the pre-trained Chinese BERT-base-uncased model to encode the input text sentiment data to obtain the corresponding text original features ,in, is a text feature extraction model, The model parameters are extracted for text features. In this way, word vectors can be quickly extracted from the sequence, and each word can be represented as a 768-dimensional word vector. In the CMU-MOSI dataset and CMU-MOSEI dataset, the original text features of the text sentiment data are ,in , .

[0072] S13. Extracting original audio features of the audio emotion data through an audio feature extraction model.

[0073] It should be noted that, in this embodiment, for audio emotion data, the COVERAP acoustic analysis framework can be used as an audio feature extraction model to extract manual features of the audio, including 12 Mel-frequency cepstral coefficients (MFCCs), pitch, volume, glottal source parameters, and other features related to emotion and intonation, thereby obtaining the original audio features of the audio emotion data; in the CMU-MOSI dataset, the original audio features are ,in , , while in the CMU-MOSEI dataset .

[0074] S14. Use a visual feature extraction model to extract the original visual features of the visual emotion data.

[0075] It should be noted that the visual emotion data in the dataset mainly comes from facial expressions, head shaking and other information. In the specific implementation, the OpenFace 2.0 (Baltrusaitis et al., 2018) model can be used to extract facial features at the frame level. These facial features are encoded based on the Facial Action Coding System (Ekman and Rosenberg, 1997), thereby obtaining the visual original features of the visual emotion data; in the CMU-MOSI dataset, the visual original features are ,in , , while in the CMU-MOSEI dataset .

[0076] S15. Input the original text features, the original audio features, and the original visual features into the unimodal feature extractor in sequence for feature extraction, and output text unimodal features, audio unimodal features, and visual unimodal features.

[0077] It should be noted that the feature extraction model is used to preprocess and precode the sentiment data of each modality to generate a unified and quantifiable original feature, which provides a basis for subsequent sentiment classification or regression tasks, such as Figure 2 and Figure 3 As shown in the figure, after the data feature extraction is completed, the scalar long short-term memory network (sLSTM) is used as a unimodal feature extractor to perform deeper feature extraction on each original feature, thereby obtaining the corresponding unimodal feature , where the data modality , is a unimodal feature extractor, It is a single-modal feature extractor parameter. On the one hand, it can perform feature mining on time series data. On the other hand, compared with traditional LSTM, sLSTM can support multiple heads, allowing independent memory storage and mixing operations in each head, and can process data in parallel, thereby improving processing efficiency.

[0078] Step 102: Perform modal feedback interaction based on text unimodal features, audio unimodal features, and visual unimodal features to generate text feedback features, audio feedback features, and visual feedback features.

[0079] Step 102 includes the following sub-steps:

[0080] S21. Use the Sigmoid activation function to perform nonlinear mapping on the text unimodal features, audio unimodal features and visual unimodal features respectively to generate text feedback masks, audio feedback masks and visual feedback masks.

[0081] It should be noted that for each modality’s unimodal features , the feedback mask can be obtained through the Sigmoid activation function ,in , is the Sigmoid activation function.

[0082] S22. Perform a mean operation on the feedback masks of any two modalities and then multiply the masks element-by-element with the unimodal features of the third modality to determine text feedback features, audio feedback features, and visual feedback features.

[0083] It should be noted that after obtaining the feedback mask of each modality, the feedback mask is merged with the single modality feature by element-by-element multiplication:

[0084] ;

[0085] Among them, the data mode , ;

[0086] Under this operation, the feedback masks of any two modalities can be applied to the unimodal features of the third modality. The mutual feedback of information between different modalities can enhance the expression of the relationship between the modalities, and the importance of different modalities can be weighed more dynamically, thereby improving the performance of multimodal fusion.

[0087] Step 103: Encode the text feedback features through a language encoder to determine text coding features, and input the audio feedback features and visual feedback features into a non-language encoder respectively, construct corresponding audio coding features and visual coding features, and then splice them into mixed coding features.

[0088] It should be noted that if Figure 3 As shown in Figure 2, after completing the modal feedback interaction, language and non-verbal modality coding (NVV coding) is performed, as shown in Figure 2. Figure 2 As shown, encoding the linguistic modality and non-linguistic modality separately can not only utilize the self-attention and cross-attention mechanisms to obtain richer information between and within modalities, but also can be well applied to the problem of differences between linguistic modalities and non-linguistic modalities in classroom situations.

[0089] In a specific implementation, encoding the text feedback feature by a language encoder to determine the text encoding feature includes:

[0090] After encoding the absolute position of the text feedback feature, it is added element by element to the text feedback feature to determine the text position feature;

[0091] After layer normalization of the text position features, they are mapped into text query matrix, text key matrix and text value matrix, and multi-head self-attention operation is performed to output text attention features;

[0092] The text position feature is residually connected with the text attention feature and then normalized to obtain the text normalized feature.

[0093] The text normalization features are transformed through the feedforward neural network layer to output text encoding features.

[0094] It should be noted that for the verbal modality, that is, the text modality, this embodiment adopts the following Figure 4 The language encoder shown in the figure performs feature processing. The language encoder includes a normalization layer (Norm), a multi-head self-attention layer (MultiHead Self Attention), and a feedforward neural network layer (FeedForward):

[0095] Text feedback feature First, it is encoded by absolute position (Position Encoding), and then combined with the text feedback feature Add to determine text position features , where It is the absolute position code;

[0096] Then, the text position features are normalized by the normalization layer and mapped into the text query matrix , Text Key Matrix and a matrix of text values , where is the linear projection matrix for text queries, is the text key linear projection matrix, is the linear projection matrix of the text value, and is input into the multi-head self-attention layer for multi-head self-attention operation, outputting the text attention feature ,in, is the softmax activation function, is the transpose of the text key matrix, is the scaling factor;

[0097] Furthermore, the text position feature is residually connected with the text attention feature and then input into the normalization layer for layer normalization to obtain the text normalization feature , where is the normalization layer; finally, the text normalization features are input into the feedforward neural network layer for feature conversion, and the text encoding features are output .

[0098] In a specific embodiment, the processing of the non-language encoder includes:

[0099] Linearly projecting the non-language input features of the input non-language encoder into a first non-language query matrix, a first non-language key matrix, and a first non-language value matrix, and performing a multi-head self-attention operation to determine a non-language multi-head attention feature;

[0100] The non-language input features are residually connected with the non-language multi-head attention features, and then layer normalization is performed to generate non-language multi-head normalized features;

[0101] Mapping the text feedback features into a second non-language query matrix, mapping the non-language multi-head normalized features into a second non-language key matrix and a second non-language value matrix, and then performing a cross-attention operation to construct a non-language cross-attention feature;

[0102] Perform residual connection on the text feedback features and the non-language cross-attention features and then perform layer normalization to generate non-language cross-normalized features;

[0103] After element-wise multiplication of the non-language input features and the non-language cross-normalized features, the non-language gating value is calculated using the Sigmoid activation function;

[0104] After concatenating the non-language input features with the non-language cross-normalized features, they are multiplied element-by-element with the non-language gating value to output the non-language gating features.

[0105] The non-language gating features are transformed through a feed-forward neural network layer to determine the non-language output features.

[0106] It should be noted that for non-verbal modalities, namely audio and visual modalities, this embodiment uses a non-verbal encoder for feature extraction. The two modalities are input into the non-verbal encoder separately, and the parameters of the two are not shared. Figure 5 As shown in the figure, the non-language encoder includes a multi-head self-attention layer (MultiHead SelfAttention), a residual connection and normalization layer (Add&Norm), a cross-attention layer (CrossModal Attention), a gating layer (Gate) and a feedforward neural network layer (FeedForward). Take the audio feedback feature as the non-language input feature of the non-language encoder as an example:

[0107] First, the audio feedback feature Linear projection to the first non-linguistic query matrix , the first non-linguistic key matrix and the first non-linguistic value matrix , where is the first non-linguistic query linear projection matrix, is the first non-linguistic key linear projection matrix, is the first non-language value linear projection matrix, and then inputs the multi-head self-attention layer to perform multi-head self-attention operations to obtain non-language multi-head attention features , where is the transpose of the first non-language key matrix, thereby enhancing the intra-modal features of the audio modality; then, through the residual connection and normalization layer, the non-language input features and the non-language multi-head attention features are residually connected and then layer normalized to generate non-language multi-head normalized features ;

[0108] Secondly, the text feedback features of the text modality are linearly mapped to the second non-linguistic query matrix , the non-language multi-head normalized feature map of the audio modality is used as the second non-language key matrix and the second non-linguistic value matrix , where is the linear projection matrix for the second non-linguistic query, is the second non-linguistic key linear projection matrix, is the linear projection matrix of the second non-language value; then it is input into the cross attention layer for cross attention operation to obtain the corresponding non-language multi-head attention feature , where is the transpose of the second non-language key matrix, so that the encoding of the non-language modality can screen out more useful features for multimodal fusion under the supervision of the language modality; the text feedback features after residual connection and the non-language cross-attention features are input into the normalization layer for layer normalization, and the non-language cross-normalized features are output ;

[0109] Furthermore, the gating layer is used to control the fusion of non-language cross-normalized features and audio feedback features based on the gate mechanism to prevent the loss of information within the modality: the non-language cross-normalized features and audio feedback features are multiplied element by element, and the non-language gating value is calculated using the Sigmoid activation function. , where is the Sigmoid activation function, The audio feedback feature and the non-language cross-normalized feature are then concatenated and element-wise multiplied with the non-language gating value to output the non-language gating feature. ; Finally, the non-language gating features are further transformed using the feedforward neural network layer to determine the non-language output features of the audio modality, namely the audio coding features , where is the feed-forward neural network layer;

[0110] According to the process of determining the audio coding feature through the non-language encoder based on the audio feedback feature, the visual coding feature is constructed by using the visual feedback feature as the non-language input feature of the non-language encoder. .

[0111] Step 104: Use text encoding features and mixed encoding features to perform feature screening interaction to determine text screening features, mixed screening features, and cross screening features.

[0112] Step 104 includes the following sub-steps:

[0113] The text encoding features and the mixed encoding features are segmented respectively, and the corresponding first features and second features are obtained;

[0114] Each first feature is input into the LSTM layer for feature extraction and then multiplied element by element with the associated second feature to generate the third feature;

[0115] The Sigmoid activation function is used to map each third feature and construct the screening gate value;

[0116] Respectively multiply the text encoding feature and the mixed encoding feature by the corresponding screening gating values ​​element by element, and output the text screening feature and the mixed screening feature;

[0117] The text filtering feature projection is used as the query matrix, and the mixed filtering feature projection is used as the key matrix and the value matrix to perform cross-attention operation and determine the cross-filtering features.

[0118] It should be noted that after obtaining the encoding features of the language modality and the non-language modality, the language and non-language modality are mixed (NVV mixing). In this embodiment, the LSTM-Gated Module (LGM) and the CrossModal Attention layer are used for feature processing. The most useful features are selected based on the gating mechanism of the LGM block, and then the cross-attention layer is used for fusion, thereby enhancing the interaction and information sharing between different modalities:

[0119] The structure of the LGM block is as follows Figure 6 As shown in the figure, it includes LSTM layer and activation function layer (Activation), and the text encoding features are processed by LGM block to obtain text screening features. , process the mixed coding features to obtain mixed screening features , where For the LSTM gating block: the text encoding features and the mixed encoding features are used as the gated input features of the input LGM block respectively, and the gated input features are divided into two parts , where The first feature is For the second feature, is the gate input feature, The split operation represents the gated input feature being divided into two features by a linear function; the first feature of the text encoding feature and the mixed encoding feature is operated through the LSTM layer to extract the local features of the gated input feature, and multiplied element by element with the corresponding directly retained second feature to generate the third feature , where For the LSTM layer, the third feature of the text encoding feature and the mixed encoding feature is processed by the activation function layer and the corresponding output filtering gate value ; Then, the text encoding feature and the mixed encoding feature as the gated input feature are multiplied element by element with their corresponding screening gate values ​​to obtain the gated output feature as the LGM block Text filtering features and mixed screening features ;

[0120] After being processed by the gating mechanism of the LGM block, the text filtering features and mixed filtering features obtained by feature filtering are input into the cross attention layer for cross attention operation to construct the cross filtering features. , where is a cross attention layer, where the text filter features are projected into the query matrix and the mixed filter features are projected into the key matrix and the value matrix.

[0121] Step 105: The text screening features, mixed screening features and cross screening features are concatenated and input into a linear layer for sentiment classification, and the sentiment analysis results are output.

[0122] It should be noted that the text filtering features, mixed filtering features and cross filtering features are concatenated and combined, and then sentiment classification is performed using a linear layer (Regression) to obtain the final sentiment analysis result. , where is a linear layer, For splicing.

[0123] To verify the effectiveness of the above method, we conducted experiments on the CMU-MOSI and CMU-MOSEI datasets. The experimental results are shown in Table 1:

[0124] Table 1 Comparison of experimental results of different models

[0125]

[0126] In Table 1, TFN, LMF, MFN and MULT data are from MMSA. Our method is improved based on the MULT model. According to the scores measured on the MOSI dataset and the MOSEI dataset, it is significantly better than other models in terms of sentiment classification accuracy (ACC2). The higher F1 value shows that our method has a greater advantage in capturing subtle emotional changes. The correlation coefficient (Corr) reflects that the linear correlation between the predicted sentiment value and the actual sentiment value of this method is stronger. The results show that our method is superior to the existing methods and is of great significance to sentiment analysis, classroom sentiment analysis, teaching evaluation and other fields.

[0127] In an embodiment of the present invention, after feature extraction of multimodal sentiment data, information interaction and fusion between different modalities are performed, and the language modality, i.e., text modality, and non-language modalities, i.e., audio modality and visual modality, are respectively encoded and feature screened and fused. This allows the feature extraction methods of the language modality and non-language modality to adapt to their respective characteristics, prevent information mixing and interference between modalities, and help improve the reliability of sentiment analysis. When adapting to special situations such as classrooms, multi-level fusion methods can make full use of information from different modalities, thereby more accurately capturing sentiment changes or other key information in tasks such as sentiment classification, thereby improving the accuracy of multimodal sentiment analysis in the classroom.

[0128] See also Figure 7 , Figure 7 A structural block diagram of a multimodal sentiment analysis system provided in an embodiment of the present invention.

[0129] The present invention provides a multimodal sentiment analysis system, comprising:

[0130] The feature acquisition module 701 is used to acquire multimodal emotion data, perform feature extraction on the multimodal emotion data, and output text unimodal features, audio unimodal features, and visual unimodal features;

[0131] Feedback module 702, used for performing modal feedback interaction based on text unimodal features, audio unimodal features and visual unimodal features, and generating text feedback features, audio feedback features and visual feedback features;

[0132] The language and non-language encoding module 703 is used to encode the text feedback features through the language encoder to determine the text coding features, and input the audio feedback features and the visual feedback features into the non-language encoder respectively, construct the corresponding audio coding features and visual coding features, and then splice them into a mixed coding feature;

[0133] The language and non-language mixed module 704 is used to perform feature screening interaction using text coding features and mixed coding features to determine text screening features, mixed screening features and cross screening features;

[0134] The classification module 705 is used to concatenate the text screening features, the mixed screening features and the cross screening features and input them into the linear layer for sentiment classification, and output the sentiment analysis results.

[0135] Optionally, the feature acquisition module 701 includes:

[0136] A data feature extraction unit is used to obtain text emotion data, audio emotion data and visual emotion data; extract the original text features of the text emotion data based on the text feature extraction model; extract the original audio features of the audio emotion data through the audio feature extraction model; and extract the original visual features of the visual emotion data using the visual feature extraction model;

[0137] The unimodal feature extraction unit is used to input the original text features, the original audio features and the original visual features into the unimodal feature extractor in sequence for feature extraction, and output the text unimodal features, the audio unimodal features and the visual unimodal features.

[0138] Optionally, the feedback module 702 is specifically configured to:

[0139] The Sigmoid activation function is used to perform nonlinear mapping on the text unimodal features, audio unimodal features and visual unimodal features respectively to generate text feedback masks, audio feedback masks and visual feedback masks;

[0140] The feedback masks of any two modalities are averaged and then multiplied element-wise with the unimodal features of the third modality to determine the text feedback features, audio feedback features, and visual feedback features.

[0141] Optionally, encoding the text feedback feature by a language encoder to determine the text encoding feature includes:

[0142] After encoding the absolute position of the text feedback feature, it is added element by element to the text feedback feature to determine the text position feature;

[0143] After layer normalization of the text position features, they are mapped into text query matrix, text key matrix and text value matrix, and multi-head self-attention operation is performed to output text attention features;

[0144] The text position feature is residually connected with the text attention feature and then normalized to obtain the text normalized feature.

[0145] The text normalization features are transformed through the feedforward neural network layer to output text encoding features.

[0146] Optionally, the processing of the non-verbal encoder includes:

[0147] Linearly projecting the non-language input features of the input non-language encoder into a first non-language query matrix, a first non-language key matrix, and a first non-language value matrix, and performing a multi-head self-attention operation to determine a non-language multi-head attention feature;

[0148] The non-language input features are residually connected with the non-language multi-head attention features, and then layer normalization is performed to generate non-language multi-head normalized features;

[0149] Mapping the text feedback features into a second non-language query matrix, mapping the non-language multi-head normalized features into a second non-language key matrix and a second non-language value matrix, and then performing a cross-attention operation to construct a non-language cross-attention feature;

[0150] Perform residual connection on the text feedback features and the non-language cross-attention features and then perform layer normalization to generate non-language cross-normalized features;

[0151] After element-wise multiplication of the non-language input features and the non-language cross-normalized features, the non-language gating value is calculated using the Sigmoid activation function;

[0152] After concatenating the non-language input features with the non-language cross-normalized features, they are multiplied element-by-element with the non-language gating value to output the non-language gating features.

[0153] The non-language gating features are transformed through a feed-forward neural network layer to determine the non-language output features.

[0154] Optionally, the language and non-language mixing module 704 is specifically used for:

[0155] The text encoding features and the mixed encoding features are segmented respectively, and the corresponding first features and second features are obtained;

[0156] Each first feature is input into the LSTM layer for feature extraction and then multiplied element by element with the associated second feature to generate the third feature;

[0157] The Sigmoid activation function is used to map each third feature and construct the screening gate value;

[0158] Respectively multiply the text encoding feature and the mixed encoding feature by the corresponding screening gating values ​​element by element, and output the text screening feature and the mixed screening feature;

[0159] The text filtering feature projection is used as the query matrix, and the mixed filtering feature projection is used as the key matrix and the value matrix to perform cross-attention operation and determine the cross-filtering features.

[0160] An embodiment of the present invention also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory; when the computer program is executed by the processor, the processor executes the steps of the multimodal sentiment analysis method as described in any of the above embodiments.

[0161] An embodiment of the present invention further provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the steps of the multimodal sentiment analysis method as described in any of the above embodiments are implemented.

[0162] An embodiment of the present invention also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implements the steps of the multimodal sentiment analysis method as described in any of the above embodiments.

[0163] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system and module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0164] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0165] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0166] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0167] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0168] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal sentiment analysis method, characterized in that: include: Acquire multimodal emotion data, perform feature extraction on the multimodal emotion data, and output text unimodal features, audio unimodal features, and visual unimodal features; Performing modal feedback interaction based on the text unimodal feature, the audio unimodal feature, and the visual unimodal feature to generate text feedback features, audio feedback features, and visual feedback features; The text feedback feature is encoded by a language encoder to determine a text encoding feature, and the audio feedback feature and the visual feedback feature are respectively input into a non-language encoder to construct corresponding audio encoding features and visual encoding features and then splice them into a mixed encoding feature; Using the text encoding feature and the mixed encoding feature to perform feature screening interaction, and determine text screening features, mixed screening features and cross screening features; The text screening feature, the mixed screening feature and the cross screening feature are concatenated and input into a linear layer for sentiment classification, and a sentiment analysis result is output.

2. The multimodal sentiment analysis method according to claim 1, characterized in that: The method of acquiring multimodal emotion data, performing feature extraction on the multimodal emotion data, and outputting text unimodal features, audio unimodal features, and visual unimodal features includes: Obtain text sentiment data, audio sentiment data, and visual sentiment data; Extracting original text features of the text sentiment data based on a text feature extraction model; Extracting original audio features of the audio emotion data through an audio feature extraction model; Using a visual feature extraction model to extract the original visual features of the visual emotion data; The original text features, the original audio features and the original visual features are sequentially input into a unimodal feature extractor for feature extraction, and text unimodal features, audio unimodal features and visual unimodal features are output.

3. The multimodal sentiment analysis method according to claim 1, characterized in that: The performing modal feedback interaction based on each of the text unimodal feature, the audio unimodal feature and the visual unimodal feature to generate text feedback features, audio feedback features and visual feedback features includes: Using a Sigmoid activation function to perform nonlinear mapping on the text unimodal feature, the audio unimodal feature, and the visual unimodal feature, respectively, to generate a text feedback mask, an audio feedback mask, and a visual feedback mask; The feedback masks of any two modalities are averaged and then multiplied element-wise with the unimodal features of the third modality to determine the text feedback features, audio feedback features, and visual feedback features.

4. The multimodal sentiment analysis method according to claim 1, characterized in that: The step of encoding the text feedback feature by a language encoder to determine the text encoding feature comprises: After encoding the absolute position of the text feedback feature, the feature is added element by element to the text feedback feature to determine the text position feature; After layer normalization processing is performed on the text position features, they are mapped into a text query matrix, a text key matrix, and a text value matrix, and multi-head self-attention operations are performed to output text attention features; Performing a residual connection between the text position feature and the text attention feature and then performing layer normalization to obtain a text normalization feature; The text normalization features are subjected to feature conversion through a feedforward neural network layer to output text encoding features.

5. The multimodal sentiment analysis method according to claim 1, characterized in that: The processing of the non-language encoder includes: Linearly projecting the non-language input features input into the non-language encoder into a first non-language query matrix, a first non-language key matrix, and a first non-language value matrix, and performing a multi-head self-attention operation to determine a non-language multi-head attention feature; Performing residual connection between the non-language input feature and the non-language multi-head attention feature and then performing layer normalization to generate a non-language multi-head normalized feature; Mapping the text feedback feature into a second non-language query matrix and mapping the non-language multi-head normalized feature into a second non-language key matrix and a second non-language value matrix, and then performing a cross-attention operation to construct a non-language cross-attention feature; Performing a residual connection on the text feedback feature and the non-language cross-attention feature and then performing layer normalization to generate a non-language cross-normalized feature; After performing element-by-element multiplication of the non-language input feature and the non-language cross-normalized feature, a non-language gating value is calculated using a Sigmoid activation function; After concatenating the non-language input feature and the non-language cross-normalized feature, multiplying the non-language gating value element by element to output a non-language gating feature; The non-language gating feature is subjected to feature transformation through a feed-forward neural network layer to determine a non-language output feature.

6. The multimodal sentiment analysis method according to claim 1, characterized in that: The adopting of the text encoding feature and the mixed encoding feature to perform feature screening interaction to determine the text screening feature, the mixed screening feature and the cross screening feature includes: The text encoding feature and the mixed encoding feature are segmented respectively to obtain corresponding first features and second features; Input each of the first features into the LSTM layer for feature extraction and then multiply the first features element by element with the associated second features to generate a third feature; A Sigmoid activation function is used to map each of the third features to construct a screening gating value; Respectively multiplying the text encoding feature and the mixed encoding feature by the corresponding screening gating values ​​element by element, and outputting the text screening feature and the mixed screening feature; A cross-attention operation is performed with the text screening feature projection as a query matrix and the mixed screening feature projection as a key matrix and a value matrix to determine the cross-screening feature.

7. A multimodal sentiment analysis system, characterized in that: include: A feature acquisition module, used to acquire multimodal emotion data, perform feature extraction on the multimodal emotion data, and output text unimodal features, audio unimodal features, and visual unimodal features; A feedback module, configured to perform modal feedback interaction based on the text unimodal feature, the audio unimodal feature and the visual unimodal feature, and generate text feedback features, audio feedback features and visual feedback features; A language and non-language encoding module, used to encode the text feedback features through a language encoder to determine text encoding features, and input the audio feedback features and the visual feedback features into a non-language encoder respectively, construct corresponding audio encoding features and visual encoding features, and then splice them into a mixed encoding feature; A language and non-language mixed module, used to use the text coding feature and the mixed coding feature to perform feature screening interaction, and determine text screening features, mixed screening features and cross screening features; The classification module is used to splice the text screening features, the mixed screening features and the cross screening features, and then input them into the linear layer for sentiment classification, and output the sentiment analysis results.

8. A computer device, characterized in that: It includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the multimodal sentiment analysis method as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the multimodal sentiment analysis method as described in any one of claims 1-6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the multimodal sentiment analysis method as described in any one of claims 1-6 are implemented.

Images