A multimodal sentiment analysis method and system

Abstract

The application discloses a kind of multimodal sentiment analysis method and system, it is related to sentiment analysis technical field, method includes: to the text unimodal feature, audio unimodal feature and visual unimodal feature of multimodal sentiment data feature extraction are modality feedback interaction, generate text feedback feature, audio feedback feature and visual feedback feature;Text feedback feature, audio feedback feature and visual feedback feature are encoded by language encoder and non-language encoder, determine the mixed encoding feature of text feedback feature and the splicing of audio encoding feature and visual encoding feature;Text encoding feature and mixed encoding feature are used to carry out feature screening interaction, determine text screening feature, mixed screening feature and cross screening feature;Based on text screening feature, mixed screening feature and cross screening feature output sentiment analysis result.Based on the above scheme, it can prevent information mixed and interference between modes, help to improve sentiment analysis reliability.

Description

Technical Field

[0001] The present invention relates to the field of sentiment analysis technology, and in particular to a multimodal sentiment analysis method and system. Background Art

[0002] The method based on multimodal sentiment analysis combines data sources from multiple modalities, including verbal and nonverbal. In classroom communication, students' text and language expressions (such as questions and answers) belong to verbal modalities, while facial expressions, body language and tones in audio belong to nonverbal modalities. By combining verbal and nonverbal sentiment information analysis, teachers can understand students' emotional changes more comprehensively and accurately. These emotional changes reflect students' participation, understanding, concentration and other aspects of learning status, which can provide strong support for teachers to timely adjust teaching strategies, improve classroom interaction and enhance teaching effectiveness.

[0003] In traditional multimodal sentiment analysis methods, features are usually extracted from each modality first, then fused and the sentiment classification results are output through a classifier. However, in the special context of the classroom, students' language modal information often comes from after-class homework and other forms, and non-language modal information is mainly obtained directly from current classroom interactions. When the above traditional methods are applied to the special context of the classroom, they are prone to problems such as insufficient extraction of common information between modalities and noise influence, resulting in low reliability of sentiment analysis. Summary of the Invention

[0004] The present invention provides a multimodal sentiment analysis method and system, which improves the technical problem of low sentiment analysis reliability of traditional multimodal sentiment analysis methods.

[0005] The first aspect of the present invention provides a multimodal sentiment analysis method, comprising:

[0006] Acquiring multimodal emotion data, performing feature extraction on the multimodal emotion data, and outputting text unimodal features, audio unimodal features, and visual unimodal features;

[0007] Performing modal feedback interaction based on the text unimodal feature, the audio unimodal feature, and the visual unimodal feature to generate text feedback features, audio feedback features, and visual feedback features;

[0008] Encoding the text feedback feature through a language encoder to determine a text coding feature, and inputting the audio feedback feature and the visual feedback feature into a non-language encoder respectively, constructing corresponding audio coding features and visual coding features, and then splicing them into a mixed coding feature;

[0009] Using the text encoding feature and the mixed encoding feature to perform feature screening interaction to determine text screening features, mixed screening features and cross screening features;

[0010] The text screening features, the mixed screening features and the cross screening features are spliced ​​and input into a linear layer for sentiment classification, and a sentiment analysis result is output.

[0011] Optionally, the acquiring of multimodal emotion data, performing feature extraction on the multimodal emotion data, and outputting text unimodal features, audio unimodal features, and visual unimodal features includes:

[0012] Obtain text emotion data, audio emotion data, and visual emotion data;

[0013] Extracting original text features of the text sentiment data based on a text feature extraction model;

[0014] Extracting original audio features of the audio emotion data through an audio feature extraction model;

[0015] Using a visual feature extraction model to extract the original visual features of the visual emotion data;

[0016] The text original features, the audio original features and the visual original features are sequentially input into a unimodal feature extractor for feature extraction, and text unimodal features, audio unimodal features and visual unimodal features are output.

[0017] Optionally, performing modal feedback interaction based on each of the text unimodal feature, the audio unimodal feature, and the visual unimodal feature to generate text feedback features, audio feedback features, and visual feedback features includes:

[0018] A Sigmoid activation function is used to perform nonlinear mapping on the text unimodal feature, the audio unimodal feature, and the visual unimodal feature to generate a text feedback mask, an audio feedback mask, and a visual feedback mask;

[0019] The feedback masks of any two modalities are averaged and then multiplied element-wise with the unimodal features of the third modality to determine the text feedback features, audio feedback features, and visual feedback features.

[0020] Optionally, encoding the text feedback feature by a language encoder to determine the text encoding feature includes:

[0021] After performing absolute position encoding on the text feedback feature, the feature is added element by element to the text feedback feature to determine a text position feature;

[0022] The text position feature is subjected to layer normalization processing, mapped into a text query matrix, a text key matrix and a text value matrix, and subjected to multi-head self-attention operation to output a text attention feature;

[0023] The text position feature is subjected to layer normalization processing, mapped into a text query matrix, a text key matrix and a text value matrix, and subjected to multi-head self-attention operation to output a text attention feature;

[0024] The text normalization feature is subjected to feature conversion by a feedforward neural network layer to output a text encoding feature.

[0025] Optionally, the processing procedure of the non-verbal encoder comprises:

[0026] The non-verbal input feature input into the non-verbal encoder is linearly projected into a first non-verbal query matrix, a first non-verbal key matrix and a first non-verbal value matrix, and subjected to multi-head self-attention operation to determine a non-verbal multi-head attention feature;

[0027] The non-verbal input feature is subjected to layer normalization processing after being subjected to residual connection with the non-verbal multi-head attention feature to generate a non-verbal multi-head normalization feature;

[0028] The text feedback feature is mapped into a second non-verbal query matrix, and the non-verbal multi-head normalization feature is mapped into a second non-verbal key matrix and a second non-verbal value matrix, and subjected to cross-attention operation to construct a non-verbal cross-attention feature;

[0029] The text feedback feature is subjected to layer normalization processing after being subjected to residual connection with the non-verbal cross-attention feature to generate a non-verbal cross-normalization feature;

[0030] The non-verbal input feature and the non-verbal cross-normalization feature are subjected to element-wise multiplication, and a non-verbal gating value is calculated by a Sigmoid activation function;

[0031] The non-verbal input feature and the non-verbal cross-normalization feature are spliced, and subjected to element-wise multiplication with the non-verbal gating value to output a non-verbal gating feature;

[0032] The non-verbal gating feature is subjected to feature conversion by a feedforward neural network layer to determine a non-verbal output feature.

[0033] Optionally, the feature screening interaction of the text encoding feature and the mixed encoding feature to determine a text screening feature, a mixed screening feature and a cross-screening feature comprises:

[0034] The text encoding feature and the mixed encoding feature are respectively segmented to obtain corresponding first features and second features;

[0035] Input each of the first features into the LSTM layer for feature extraction and then multiply the first features element-wise with the associated second features to generate a third feature;

[0036] A Sigmoid activation function is used to map each of the third features to construct a screening gating value;

[0037] Multiplying the text encoding feature and the mixed encoding feature by the corresponding screening gating values ​​element by element, and outputting a text screening feature and a mixed screening feature;

[0038] A cross-attention operation is performed with the text screening feature projection as a query matrix and the mixed screening feature projection as a key matrix and a value matrix to determine the cross-screening feature.

[0039] A second aspect of the present invention provides a multimodal sentiment analysis system, comprising:

[0040] A feature acquisition module is used to acquire multimodal emotion data, perform feature extraction on the multimodal emotion data, and output text unimodal features, audio unimodal features, and visual unimodal features;

[0041] A feedback module, configured to perform modal feedback interaction based on the text unimodal feature, the audio unimodal feature, and the visual unimodal feature to generate text feedback features, audio feedback features, and visual feedback features;

[0042] A language and non-language encoding module is used to encode the text feedback features through a language encoder to determine text coding features, and input the audio feedback features and the visual feedback features into a non-language encoder respectively, construct corresponding audio coding features and visual coding features, and then splice them into a mixed coding feature;

[0043] A language and non-language mixed module, configured to perform feature screening interaction using the text encoding feature and the mixed encoding feature to determine text screening features, mixed screening features, and cross-screening features;

[0044] The classification module is used to splice the text screening features, the mixed screening features and the cross screening features, and then input them into the linear layer for sentiment classification, and output the sentiment analysis results.

[0045] A third aspect of the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the multimodal sentiment analysis method as described in any one of the above items.

[0046] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the multimodal sentiment analysis method as described in any one of the above items.

[0047] A fifth aspect of the present invention provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the multimodal sentiment analysis method as described in any one of the above items.

[0048] It can be seen from the above technical solutions that the present invention has the following advantages:

[0049] The above-mentioned scheme of the present invention provides a multimodal sentiment analysis method, including: obtaining multimodal sentiment data, performing feature extraction on the multimodal sentiment data, and outputting text unimodal features, audio unimodal features, and visual unimodal features; performing modal feedback interaction based on the text unimodal features, audio unimodal features, and visual unimodal features to generate text feedback features, audio feedback features, and visual feedback features; encoding the text feedback features through a language encoder to determine text coding features, and inputting the audio feedback features and visual feedback features into a non-language encoder respectively, constructing corresponding audio coding features and visual coding features, and then splicing them into mixed coding features; using text coding features and mixed coding features to perform feature screening interaction to determine text screening features, mixed screening features, and cross-screening features; splicing the text screening features, mixed screening features, and cross-screening features, and then inputting them into a linear layer for sentiment classification, and outputting sentiment analysis results. Based on the above scheme, after feature extraction of multimodal sentiment data, information interaction and fusion between different modalities are performed, and the language modality, i.e., text modality, and non-language modalities, i.e., audio modality and visual modality, are encoded and feature filtered and fused respectively. This allows the feature extraction methods of the language modality and non-language modality to adapt to their respective characteristics, prevents information mixing and interference between modalities, and helps improve the reliability of sentiment analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0051] Figure 1 A flowchart of the steps of a multimodal sentiment analysis method provided by an embodiment of the present invention;

[0052] Figure 2 A network architecture diagram of a multimodal sentiment analysis network provided by an embodiment of the present invention;

[0053] Figure 3 A flowchart of a multi-modal sentiment analysis network provided by an embodiment of the present application is shown in FIG. 1.

[0054] Figure 4 A structural diagram of a language encoder provided by an embodiment of the present application is shown in FIG. 2.

[0055] Figure 5 A structural diagram of a non-language encoder provided by an embodiment of the present application is shown in FIG. 3.

[0056] Figure 6 A structural diagram of an LSTM gating block provided by an embodiment of the present application is shown in FIG. 4.

[0057] Figure 7 A structural block diagram of a multi-modal sentiment analysis system provided by an embodiment of the present application is shown in FIG. 5. DETAILED DESCRIPTION

[0058] The embodiment of the present application provides a multi-modal sentiment analysis method and system, which is used for improving the technical problem of low sentiment analysis reliability of a conventional multi-modal sentiment analysis method.

[0059] In order to make the technical scheme of the present application clearer and easier to understand, the technical scheme in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the embodiments described below are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0060] Please refer to Figure 1 , Figure 1 A step flowchart of a multi-modal sentiment analysis method provided by an embodiment of the present application is shown in FIG. 6.

[0061] The multi-modal sentiment analysis method provided by the present application comprises the following steps.

[0062] Step 101: obtaining multi-modal sentiment data, performing feature extraction on the multi-modal sentiment data, and outputting text single-modal features, audio single-modal features and visual single-modal features.

[0063] Step 101 comprises the following sub-steps.

[0064] S11: obtaining text sentiment data, audio sentiment data and visual sentiment data.

[0065] It should be noted that multimodal sentiment data can be obtained from the CMU-MOSI and CMU-MOSEI datasets; the CMU-MOSI dataset is a widely used benchmark dataset in multimodal sentiment analysis (MSA) research. The dataset contains a series of YouTube monologues in which the narrators express their personal opinions on topics such as movies. There are a total of 93 videos covering 89 different speakers. MOSI contains 2198 subjective sentence-video clips that have been manually annotated with a rating range of [ ,3], where + indicates strong negative sentiment, and +3 indicates strong positive sentiment. The CMU-MOSEI dataset is an improvement on MOSI, with more sentences, richer samples, and a diversity of speakers and topics. The dataset contains 23,453 annotated video clips (sentences) from 5,000 videos, covering 1,000 different speakers and 250 different topics.

[0066] In this embodiment, the multimodal emotion data mainly involves the text emotion data of the text mode (Text, t) as the language mode, the audio emotion data of the audio mode (Audio, a) as the non-language mode, and the visual emotion data of the visual mode (Vision, v). The multimodal emotion data is input as follows: Figure 2 The trained multimodal sentiment analysis network is processed as shown above. First, the original features of the sentiment data are extracted using the feature extraction model. , where the data modality , is the sequence length, is the sequence dimension.

[0067] Optionally, the training process of the multimodal sentiment analysis network includes: obtaining training multimodal sentiment data and real sentiment analysis results; inputting the training multimodal sentiment data into an untrained multimodal sentiment analysis network, and outputting predicted sentiment analysis results; using the real sentiment analysis results and the predicted sentiment analysis results to calculate the loss function value, iteratively optimizing the untrained multimodal sentiment analysis network according to the loss function value until the loss function value converges, and determining the trained multimodal sentiment analysis network; this embodiment performs classification tasks based on the real sentiment analysis results and the predicted sentiment analysis results to calculate the loss function value, and adopts a binary cross entropy loss function based on mean square error for calculation in the specific implementation, and for the batch The loss function value is calculated as follows:

[0068] ;

[0069] Where, is the loss function value, is the batch size when using batch training method, For the training samples, is the real sentiment analysis result, To predict sentiment analysis results.

[0070] S12. Extract the original text features of the text sentiment data based on the text feature extraction model.

[0071] It should be noted that all videos in the dataset have both Chinese and English versions. This embodiment only uses the Chinese version. Figure 2 As shown, the text feature extraction model can use a pretrained BERT model (Pretrained BERT), such as the Chinese BERT-base-uncased model; the native sequence as text sentiment data is recorded as , add two unique tags "[CLS]" and "[SEP]" to indicate the beginning and end of each sentence, and then use the pre-trained Chinese BERT-base-uncased model to encode the input text sentiment data to obtain the corresponding text original features ,in, For text feature extraction model, Extract model parameters for text features. In this way, word vectors can be quickly extracted from the sequence, and each word can be represented as a 768-dimensional word vector. In the CMU-MOSI dataset and CMU-MOSEI dataset, the original text features of the text sentiment data are ,in , .

[0072] S13. Extracting original audio features of the audio emotion data through an audio feature extraction model.

[0073] It should be noted that, in this embodiment, for audio emotion data, the COVERAP acoustic analysis framework can be used as an audio feature extraction model to extract manual features of the audio, including 12 Mel-frequency cepstral coefficients (MFCCs), pitch, volume, glottal source parameters, and other features related to emotion and intonation, thereby obtaining the original audio features of the audio emotion data; in the CMU-MOSI dataset, the original audio features are ,in , , while in the CMU-MOSEI dataset .

[0074] S14. Use a visual feature extraction model to extract the original visual features of the visual emotion data.

[0075] It should be noted that the visual emotion data in the dataset mainly comes from facial expressions, head shaking and other information. In the specific implementation, the OpenFace 2.0 (Baltrusaitis et al., 2018) model can be used to extract facial features at the frame level. These facial features are encoded based on the Facial Action Coding System (Ekman and Rosenberg, 1997), thereby obtaining the visual raw features of the visual emotion data; in the CMU-MOSI dataset, the visual raw features are ,in , , while in the CMU-MOSEI dataset .

[0076] S15. Input the original text features, the original audio features, and the original visual features into the unimodal feature extractor in sequence for feature extraction, and output text unimodal features, audio unimodal features, and visual unimodal features.

[0077] It should be noted that the feature extraction model preprocesses and precodes the sentiment data of each modality to generate a unified and quantifiable original feature, which provides a basis for subsequent sentiment classification or regression tasks, such as Figure 2 and Figure 3 As shown in the figure, after the data feature extraction is completed, the scalar long short-term memory network (sLSTM) is used as a single-modal feature extractor to perform deeper feature extraction on each original feature, thereby obtaining the corresponding single-modal feature , where the data modality , is a unimodal feature extractor, It is a single-modal feature extractor parameter. On the one hand, it can perform feature mining on time series data. On the other hand, compared with traditional LSTM, sLSTM can support multiple heads, allowing independent memory storage and mixing operations in each head, and can process data in parallel, thereby improving processing efficiency.

[0078] Step 102: Perform modal feedback interaction based on the text unimodal features, the audio unimodal features, and the visual unimodal features to generate text feedback features, audio feedback features, and visual feedback features.

[0079] Step 102 includes the following sub-steps:

[0080] S21. Use the Sigmoid activation function to perform nonlinear mapping on the text unimodal features, audio unimodal features and visual unimodal features respectively to generate text feedback mask, audio feedback mask and visual feedback mask.

[0081] It should be noted that the single-modal feature of each modality is obtained by the Sigmoid activation function , and the feedback mask is obtained by the Sigmoid activation function , wherein , is the Sigmoid activation function.

[0082] S22, the feedback masks of any two modalities are subjected to mean operation, and then are multiplied element by element with the single-modal feature of the third modality to determine the text feedback feature, the audio feedback feature and the visual feedback feature.

[0083] It should be noted that after obtaining the feedback mask of each modality, the feedback mask is combined with the single-modal feature by element-wise multiplication:

[0084] ;

[0085] wherein the data modalities , ;

[0086] Under this operation, the feedback masks of any two modalities can be applied to the single-modal feature of the third modality, and the mutual feedback of information between different modalities can enhance the relationship expression between modalities, more dynamically weigh the importance of different modalities, and thus improve the performance of multi-modal fusion.

[0087] Step 103, encoding the text feedback feature by a language encoder to determine a text encoded feature, and inputting the audio feedback feature and the visual feedback feature into a non-language encoder respectively to construct corresponding audio encoded features and visual encoded features and then splicing them into a mixed encoded feature.

[0088] It should be noted that, as shown in Figure 3 , after completing the modal feedback interaction, language and non-language modal encoding (NVV encoding) is performed, as shown in Figure 2 , the language modal and the non-language modal are encoded separately, which not only can obtain more abundant information between modalities and within modalities by using self-attention and cross-attention mechanisms, but also can be well applied to the problem that the language modal and the non-language modal exist differences in the classroom context.

[0089] In one specific embodiment, the text encoded feature is determined by encoding the text feedback feature by a language encoder, comprising:

[0090] After the text feedback feature is subjected to absolute position encoding, the text feedback feature is added element by element to determine the text position feature;

[0091] After layer normalization processing on the text position feature, the text position feature is mapped into a text query matrix, a text key matrix and a text value matrix and multi-head self-attention operation is performed to output a text attention feature;

[0092] After residual connection between the text position feature and the text attention feature, layer normalization is performed to obtain a text normalized feature;

[0093] The text normalized feature is converted by a feedforward neural network layer to output a text encoding feature.

[0094] It should be noted that for the verbal modality, i.e., the text modality, the embodiment adopts a language encoder as shown in Figure 4 for feature processing, and the language encoder includes a normalization layer (Norm), a multi-head self-attention layer (MultiHead Self Attention) and a feedforward neural network layer (FeedForward):

[0095] Text feedback feature First, absolute position encoding (Positional Encoding) is performed, and then the text feedback feature is added to determine a text position feature , wherein is the absolute position encoding;

[0096] Then, after layer normalization processing on the text position feature by the normalization layer, the text position feature is mapped into a text query matrix , a text key matrix and a text value matrix , wherein is a text query linear projection matrix, is a text key linear projection matrix, is a text value linear projection matrix, and the multi-head self-attention layer is inputted to perform multi-head self-attention operation to output a text attention feature , wherein is a softmax activation function, is a transpose of the text key matrix, is a scaling factor;

[0097] Further, after residual connection between the text position feature and the text attention feature, the normalization layer is inputted to perform layer normalization to obtain a text normalized feature , wherein is the normalization layer; finally, the text normalized feature is inputted to the feedforward neural network layer to convert the feature, and a text encoding feature is outputted.

[0098] In one embodiment, the processing of the non-speech encoder includes:

[0099] Linearly projecting the non-language input features of the input non-language encoder into a first non-language query matrix, a first non-language key matrix, and a first non-language value matrix, and performing a multi-head self-attention operation to determine a non-language multi-head attention feature;

[0100] The non-language input features are residually connected with the non-language multi-head attention features, and then layer normalization is performed to generate non-language multi-head normalized features;

[0101] Mapping the text feedback features into a second non-language query matrix, mapping the non-language multi-head normalized features into a second non-language key matrix and a second non-language value matrix, and then performing a cross-attention operation to construct a non-language cross-attention feature;

[0102] Perform residual connection on the text feedback features and the non-language cross-attention features and then perform layer normalization to generate non-language cross-normalized features;

[0103] After element-wise multiplication of the non-language input features and the non-language cross-normalized features, the non-language gating value is calculated using the Sigmoid activation function;

[0104] After concatenating the non-language input features with the non-language cross-normalized features, they are element-wise multiplied with the non-language gating value to output the non-language gating features.

[0105] The non-language gating features are transformed through a feed-forward neural network layer to determine the non-language output features.

[0106] It should be noted that for non-verbal modalities, namely audio and vision, this embodiment uses a non-verbal encoder for feature extraction. The two modalities are input into the non-verbal encoder separately, and the parameters of the two are not shared. Figure 5 As shown in the figure, the non-language encoder includes a multi-head self-attention layer (MultiHead SelfAttention), a residual connection and normalization layer (Add&Norm), a cross-attention layer (CrossModal Attention), a gating layer (Gate), and a feedforward neural network layer (FeedForward). Take the audio feedback feature as the non-language input feature of the non-language encoder as an example:

[0107] First, the audio feedback feature Linear projection is the first non-linguistic query matrix , the first non-linguistic bond matrix and the first non-linguistic value matrix , where is a linear projection matrix of the first non-verbal query, is a linear projection matrix of the first non-verbal key, is a linear projection matrix of the first non-verbal value, and then a multi-head self-attention layer is inputted to perform multi-head self-attention operation to obtain non-verbal multi-head attention feature , wherein, is a transpose of the first non-verbal key matrix, thereby enhancing the intra-modal feature of the audio modal; then, after residual connection and normalization layer, the non-verbal input feature and the non-verbal multi-head attention feature are connected in residual connection, and then layer normalization is performed to generate non-verbal multi-head normalized feature ;

[0108] Secondly, the text feedback feature of the text modal is linearly mapped as a second non-verbal query matrix , the non-verbal multi-head normalized feature of the audio modal is mapped as a second non-verbal key matrix , and a second non-verbal value matrix , wherein, is a linear projection matrix of the second non-verbal query, is a linear projection matrix of the second non-verbal key, is a linear projection matrix of the second non-verbal value; then a cross-attention layer is inputted to perform cross-attention operation to obtain corresponding non-verbal multi-head attention feature , wherein, is a transpose of the second non-verbal key matrix, thereby enabling the encoding of the non-verbal modal to screen out more useful features for multi-modal fusion under the supervision of the language modal; the text feedback feature after residual connection and the non-verbal cross-attention feature are inputted into a normalization layer to perform layer normalization processing, and the non-verbal cross-normalized feature is outputted ;

[0109] Further, a gating layer based on a gate mechanism (Gate Mechanism) is used to control the fusion of the non-verbal cross-normalized feature and the audio feedback feature to prevent loss of intra-modal information: the non-verbal cross-normalized feature and the audio feedback feature are multiplied element by element, and then a sigmoid activation function is used to calculate a non-verbal gating value , wherein, is a sigmoid activation function, is an element-by-element multiplication, then the audio feedback feature and the non-verbal cross-normalized feature are spliced and multiplied element by element with the non-verbal gating value to output a non-verbal gating feature ; finally, a feedforward neural network layer is used to further transform the non-verbal gating feature to determine the non-verbal output feature of the audio modal, i.e. the audio encoding feature , wherein, is a feedforward neural network layer;

[0110] determining the audio encoding features by the non-verbal encoder according to the audio feedback features, to construct visual encoding features taking the visual feedback features as the non-verbal input features of the non-verbal encoder .

[0111] Step 104: performing feature screening interaction on the text encoding features and the mixed encoding features to determine text screening features, mixed screening features and cross screening features.

[0112] Step 104 includes the following sub-steps:

[0113] segmenting the text encoding features and the mixed encoding features respectively to obtain corresponding first features and second features;

[0114] inputting each first feature into an LSTM layer for feature extraction, and then performing element-by-element multiplication with the associated second feature to generate a third feature;

[0115] mapping each third feature using a Sigmoid activation function to construct a screening gate value;

[0116] element-by-element multiplying the text encoding features and the mixed encoding features with the corresponding screening gate values respectively to output text screening features and mixed screening features;

[0117] performing cross attention operation on the text screening features projection as a query matrix, the mixed screening features projection as a key matrix and a value matrix to determine cross screening features.

[0118] It should be noted that after obtaining the encoding features of the language modal and the non-verbal modal, language and non-verbal modal mixing (NVV mixing) is performed, and in this embodiment, feature processing is performed through an LSTM gated module (LSTM-Gated Module, LGM) and a cross attention layer (CrossModal Attention), the most useful features are screened based on the gating mechanism of the LGM block, and then the cross attention layer is used for fusion, so as to enhance the interaction and information sharing between different modalities:

[0119] The structure of the LGM block is shown in Figure 6 , which includes an LSTM layer and an activation function layer (Activation), and the text screening features are obtained by processing the text encoding features through the LGM block, and the mixed screening features are obtained by processing the mixed encoding features through the LGM block, wherein is the LSTM gated module: taking the text encoding features and the mixed encoding features as the gating input features of the LGM block, and dividing the gating input features into two parts , wherein As the first feature, For the second feature, is the gate input feature, The split operation represents the gated input feature being divided into two features by a linear function; the first feature of the text encoding feature and the mixed encoding feature is operated through the LSTM layer to extract the local features of the gated input feature, and multiplied element-by-element with the corresponding directly retained second feature to generate the third feature , where For the LSTM layer, the third feature of the text encoding feature and the mixed encoding feature is processed by the activation function layer and the corresponding output filtering gate value is obtained. ; Then, the text encoding features and mixed encoding features as gated input features are multiplied element-wise with their corresponding screening gate values ​​to obtain the gated output features as the LGM block Text filtering features and mixed screening features ;

[0120] After being processed by the gating mechanism of the LGM block, the text filtering features and the mixed filtering features obtained by feature filtering are input into the cross attention layer for cross attention operation to construct the cross filtering features. , where is a cross attention layer, where the text filter features are projected into the query matrix and the mixed filter features are projected into the key matrix and the value matrix.

[0121] Step 105: The text screening features, mixed screening features, and cross-screening features are concatenated and input into a linear layer for sentiment classification, and the sentiment analysis results are output.

[0122] It should be noted that the text filtering features, mixed filtering features and cross-filtering features are combined and then sentiment classification is performed using a linear layer (Regression) to obtain the final sentiment analysis results. , where is a linear layer, For splicing.

[0123] To verify the effectiveness of the above method, we conducted experiments on the CMU-MOSI and CMU-MOSEI datasets. The experimental results are shown in Table 1:

[0124] Table 1 Comparison of experimental results of different models

[0125]

[0126] In Table 1, TFN, LMF, MFN, and MULT data are from MMSA. Our method is an improvement based on the MULT model. According to the scores measured on the MOSI and MOSEI datasets, our method is significantly better than other models in terms of sentiment classification accuracy (ACC2). The higher F1 value indicates that our method has a greater advantage in capturing subtle emotional changes. The correlation coefficient (Corr) reflects the stronger linear correlation between the predicted and actual emotional values ​​of our method. The results show that our method is superior to existing methods and is of great significance in fields such as sentiment analysis, classroom sentiment analysis, and teaching evaluation.

[0127] In an embodiment of the present invention, after feature extraction of multimodal emotional data, information interaction and fusion between different modalities are performed, and the language modality, i.e., text modality, and the non-language modalities, i.e., audio modality and visual modality, are encoded and feature-filtered and fused respectively. This allows the feature extraction methods of the language modality and the non-language modality to adapt to their respective characteristics, prevents information mixing and interference between modalities, and helps to improve the reliability of emotional analysis. When adapting to special situations such as classrooms, multi-level fusion methods can make full use of information from different modalities, thereby more accurately capturing emotional changes or other key information in tasks such as emotion classification, thereby improving the accuracy of multimodal emotional analysis in the classroom.

[0128] See also Figure 7 , Figure 7 A structural block diagram of a multimodal sentiment analysis system provided by an embodiment of the present invention.

[0129] The present invention provides a multimodal sentiment analysis system, comprising:

[0130] A feature acquisition module 701 is used to acquire multimodal emotion data, perform feature extraction on the multimodal emotion data, and output text unimodal features, audio unimodal features, and visual unimodal features;

[0131] Feedback module 702, configured to perform modal feedback interaction based on text unimodal features, audio unimodal features, and visual unimodal features, and generate text feedback features, audio feedback features, and visual feedback features;

[0132] The language and non-language encoding module 703 is used to encode the text feedback features through the language encoder to determine the text coding features, and input the audio feedback features and visual feedback features into the non-language encoder respectively, construct the corresponding audio coding features and visual coding features, and then splice them into a mixed coding feature;

[0133] The language and non-language mixing module 704 is used to perform feature screening interaction using text coding features and mixed coding features to determine text screening features, mixed screening features, and cross-screening features;

[0134] The classification module 705 is configured to perform sentiment classification on the spliced text screening feature, mixed screening feature and cross screening feature input into a linear layer, and output a sentiment analysis result.

[0135] Optionally, the feature acquisition module 701 comprises:

[0136] The data feature extraction unit is configured to acquire text sentiment data, audio sentiment data and visual sentiment data; extract text original features of the text sentiment data based on a text feature extraction model; extract audio original features of the audio sentiment data through an audio feature extraction model; and extract visual original features of the visual sentiment data using a visual feature extraction model.

[0137] The single-modal feature extraction unit is configured to sequentially input the text original features, the audio original features and the visual original features into a single-modal feature extractor for feature extraction, and output text single-modal features, audio single-modal features and visual single-modal features.

[0138] Optionally, the feedback module 702 is specifically configured to:

[0139] The Sigmoid activation function is used to perform nonlinear mapping on the text single-modal features, the audio single-modal features and the visual single-modal features respectively, to generate text feedback masks, audio feedback masks and visual feedback masks.

[0140] The feedback masks of any two modalities are subjected to mean operation, and then are multiplied element by element with the single-modal features of the third modality to determine text feedback features, audio feedback features and visual feedback features.

[0141] Optionally, the text feedback features are encoded by a language encoder to determine text encoded features, comprising:

[0142] The text feedback features are subjected to absolute position encoding, and then are added element by element with the text feedback features to determine text position features.

[0143] The text position features are subjected to layer normalization processing, and then are mapped into a text query matrix, a text key matrix and a text value matrix and subjected to multi-head self-attention operation to output text attention features.

[0144] The text position features and the text attention features are subjected to residual connection, and then are subjected to layer normalization to obtain text normalized features.

[0145] The text normalized features are subjected to feature conversion by a feedforward neural network layer to output text encoded features.

[0146] Optionally, the processing process of the non-language encoder comprises:

[0147] Linearly projecting the non-language input features of the input non-language encoder into a first non-language query matrix, a first non-language key matrix, and a first non-language value matrix, and performing a multi-head self-attention operation to determine a non-language multi-head attention feature;

[0148] The non-language input features are residually connected with the non-language multi-head attention features, and then layer normalization is performed to generate non-language multi-head normalized features;

[0149] Mapping the text feedback features into a second non-language query matrix, mapping the non-language multi-head normalized features into a second non-language key matrix and a second non-language value matrix, and then performing a cross-attention operation to construct a non-language cross-attention feature;

[0150] Perform residual connection on the text feedback features and the non-language cross-attention features and then perform layer normalization to generate non-language cross-normalized features;

[0151] After element-wise multiplication of the non-language input features and the non-language cross-normalized features, the non-language gating value is calculated using the Sigmoid activation function;

[0152] After concatenating the non-language input features with the non-language cross-normalized features, they are element-wise multiplied with the non-language gating value to output the non-language gating features.

[0153] The non-language gating features are transformed through a feed-forward neural network layer to determine the non-language output features.

[0154] Optionally, the language and non-language mixing module 704 is specifically configured to:

[0155] The text encoding features and the mixed encoding features are segmented respectively to obtain the corresponding first features and second features;

[0156] Each first feature is input into the LSTM layer for feature extraction and then multiplied element-wise with the associated second feature to generate the third feature;

[0157] The Sigmoid activation function is used to map each third feature and construct the screening gate value;

[0158] Multiply the text encoding feature and the mixed encoding feature by the corresponding screening gate values ​​element by element, and output the text screening feature and the mixed screening feature;

[0159] A cross-attention operation is performed with the text filtering feature projection as the query matrix and the mixed filtering feature projection as the key matrix and value matrix to determine the cross-filtering features.

[0160] The embodiment of the present application further provides a computer device, comprising a memory and a processor, the memory stores a computer program; the computer program is executed by the processor, so that the processor executes the steps of the multi-modal sentiment analysis method according to any one of the above embodiments.

[0161] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program / instruction, and the computer program / instruction is executed by a processor to realize the steps of the multi-modal sentiment analysis method according to any one of the above embodiments.

[0162] The embodiment of the present application further provides a computer program product, which comprises a computer program / instruction, and the computer program / instruction is executed by a processor to realize the steps of the multi-modal sentiment analysis method according to any one of the above embodiments.

[0163] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system and module can refer to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0164] In several embodiments provided in the present application, it should be understood that the disclosed system and method can be implemented in other ways. For example, the system embodiments described above are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, system or unit, and can be electrical, mechanical or other forms.

[0165] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0166] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit.

[0167] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the entire or part of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0168] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multimodal sentiment analysis method, characterized in that: include: Acquiring multimodal emotion data, performing feature extraction on the multimodal emotion data, and outputting text unimodal features, audio unimodal features, and visual unimodal features; Performing modal feedback interaction based on the text unimodal feature, the audio unimodal feature, and the visual unimodal feature to generate text feedback features, audio feedback features, and visual feedback features; Encoding the text feedback feature through a language encoder to determine a text coding feature, and inputting the audio feedback feature and the visual feedback feature into a non-language encoder respectively, constructing corresponding audio coding features and visual coding features, and then splicing them into a mixed coding feature; Using the text encoding feature and the mixed encoding feature to perform feature screening interaction to determine text screening features, mixed screening features and cross screening features; The text screening feature, the mixed screening feature and the cross screening feature are spliced ​​and input into a linear layer for sentiment classification, and a sentiment analysis result is output; The performing modal feedback interaction based on the text unimodal feature, the audio unimodal feature, and the visual unimodal feature to generate text feedback features, audio feedback features, and visual feedback features includes: A Sigmoid activation function is used to perform nonlinear mapping on the text unimodal feature, the audio unimodal feature, and the visual unimodal feature to generate a text feedback mask, an audio feedback mask, and a visual feedback mask; The feedback masks of any two modalities are averaged and then multiplied element-wise with the unimodal features of the third modality to determine the text feedback features, audio feedback features, and visual feedback features; The interactive feature screening using the text encoding feature and the mixed encoding feature to determine the text screening feature, the mixed screening feature and the cross screening feature includes: Segmenting the text encoding feature and the mixed encoding feature respectively to obtain corresponding first features and second features; Input each of the first features into the LSTM layer for feature extraction and then multiply the first features element-wise with the associated second features to generate a third feature; A Sigmoid activation function is used to map each of the third features to construct a screening gating value; Multiplying the text encoding feature and the mixed encoding feature by the corresponding screening gating values ​​element by element, and outputting a text screening feature and a mixed screening feature; A cross-attention operation is performed with the text screening feature projection as a query matrix and the mixed screening feature projection as a key matrix and a value matrix to determine the cross-screening feature.

2. The multimodal sentiment analysis method according to claim 1, characterized in that The acquiring of multimodal emotion data, performing feature extraction on the multimodal emotion data, and outputting text unimodal features, audio unimodal features, and visual unimodal features includes: Obtain text emotion data, audio emotion data, and visual emotion data; Extracting original text features of the text sentiment data based on a text feature extraction model; Extracting original audio features of the audio emotion data through an audio feature extraction model; Using a visual feature extraction model to extract the original visual features of the visual emotion data; The text original features, the audio original features and the visual original features are sequentially input into a unimodal feature extractor for feature extraction, and text unimodal features, audio unimodal features and visual unimodal features are output.

3. The multimodal sentiment analysis method according to claim 1, characterized in that The step of encoding the text feedback feature by a language encoder to determine a text encoding feature includes: After performing absolute position encoding on the text feedback feature, the feature is added element by element to the text feedback feature to determine a text position feature; After layer normalization processing is performed on the text position features, they are mapped into a text query matrix, a text key matrix, and a text value matrix, and a multi-head self-attention operation is performed to output text attention features; Performing a residual connection between the text position feature and the text attention feature and then performing layer normalization to obtain a text normalized feature; The text normalization features are subjected to feature conversion through a feedforward neural network layer to output text encoding features.

4. The multimodal sentiment analysis method according to claim 1, characterized in that The processing of the non-language encoder includes: Linearly projecting the non-language input features input to the non-language encoder into a first non-language query matrix, a first non-language key matrix, and a first non-language value matrix, and performing a multi-head self-attention operation to determine a non-language multi-head attention feature; Performing a residual connection between the non-language input feature and the non-language multi-head attention feature and then performing layer normalization to generate a non-language multi-head normalized feature; Mapping the text feedback feature into a second non-language query matrix and mapping the non-language multi-head normalized feature into a second non-language key matrix and a second non-language value matrix, and then performing a cross-attention operation to construct a non-language cross-attention feature; Performing a residual connection on the text feedback feature and the non-language cross-attention feature and then performing layer normalization to generate a non-language cross-normalized feature; After performing element-wise multiplication of the non-language input feature and the non-language cross-normalized feature, a non-language gating value is calculated using a Sigmoid activation function; After concatenating the non-language input feature and the non-language cross-normalized feature, multiplying the concatenated feature element-by-element with the non-language gating value to output a non-language gating feature; The non-language gating feature is subjected to feature transformation through a feed-forward neural network layer to determine a non-language output feature.

5. A multimodal sentiment analysis system, characterized in that: include: A feature acquisition module is used to acquire multimodal emotion data, perform feature extraction on the multimodal emotion data, and output text unimodal features, audio unimodal features, and visual unimodal features; A feedback module, configured to perform modal feedback interaction based on the text unimodal feature, the audio unimodal feature, and the visual unimodal feature to generate text feedback features, audio feedback features, and visual feedback features; A language and non-language encoding module is used to encode the text feedback features through a language encoder to determine text coding features, and input the audio feedback features and the visual feedback features into a non-language encoder respectively, construct corresponding audio coding features and visual coding features, and then splice them into a mixed coding feature; A language and non-language mixed module, configured to perform feature screening interaction using the text encoding feature and the mixed encoding feature to determine text screening features, mixed screening features, and cross-screening features; A classification module is used to splice the text screening features, the mixed screening features and the cross screening features, and then input them into a linear layer for sentiment classification, and output a sentiment analysis result; The feedback module is specifically used for: A Sigmoid activation function is used to perform nonlinear mapping on the text unimodal feature, the audio unimodal feature, and the visual unimodal feature to generate a text feedback mask, an audio feedback mask, and a visual feedback mask; The feedback masks of any two modalities are averaged and then multiplied element-wise with the unimodal features of the third modality to determine the text feedback features, audio feedback features, and visual feedback features; The language and non-language mixing module is specifically used for: Segmenting the text encoding feature and the mixed encoding feature respectively to obtain corresponding first features and second features; Input each of the first features into the LSTM layer for feature extraction and then multiply the first features element-wise with the associated second features to generate a third feature; A Sigmoid activation function is used to map each of the third features to construct a screening gating value; Multiplying the text encoding feature and the mixed encoding feature by the corresponding screening gating values ​​element by element, and outputting a text screening feature and a mixed screening feature; A cross-attention operation is performed with the text screening feature projection as a query matrix and the mixed screening feature projection as a key matrix and a value matrix to determine the cross-screening feature.

6. A computer device, characterized in that: It includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor performs the steps of the multimodal sentiment analysis method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the multimodal sentiment analysis method according to any one of claims 1 to 4 are implemented.

8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the multimodal sentiment analysis method according to any one of claims 1 to 4 are implemented.

Images