Multimodal sentiment analysis methods, systems, computer devices, and readable storage media
By processing multimodal data through feature extraction, denoising, enhancement, and interaction modules, the problems of noise interference and ignoring contextual information are solved, improving the accuracy of multimodal sentiment analysis, especially the sentiment detection effect in ironic contexts.
Patent Information
- Application Number
- CN202311826801.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-28
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2043-12-28
AI Technical Summary
Existing multimodal sentiment analysis models are affected by interference noise during cross-modal contextual interaction and sentiment prediction, and ignore irrelevant contextual information between modalities, resulting in inaccurate sentiment prediction results.
Employing modules for feature extraction, representation denoising, sentiment enhancement, and contextual interaction, this approach removes noise and mines relevant and irrelevant contextual information between modalities, utilizing self-, relevant, and irrelevant attention mechanisms for cross-modal sentiment prediction.
It improves the accuracy of multimodal sentiment prediction, especially the sentiment detection capability in ironic contexts.
Smart Images

Figure CN117786492B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of sentiment prediction technology, and in particular to a multimodal sentiment analysis method, system, computer device, and readable storage medium. Background Technology
[0002] With the booming development of social media, traditional text sentiment analysis can no longer meet people's needs. Therefore, multimodal sentiment analysis, which connects various modalities and extracts attitudes, opinions, and emotional information from them, has begun to attract attention. The advantages of this multi-source data play a decisive role in sentiment analysis. For example, in sentiment prediction in ironic contexts, the combined use of multiple data sources can easily detect the true emotions in ironic contexts.
[0003] In related technologies, complex deep learning models are constructed by leveraging complementary sentiment information between multimodal data to obtain multimodal representations that have a reliable mapping to sentiment polarity.
[0004] However, related technologies tend to focus more on the commonalities and individualities of sentiment information across modalities, neglecting the significant noise inherent in the data from each modality. This makes the models susceptible to interference noise when performing cross-modal contextual interactions and sentiment prediction. Furthermore, while related technologies also employ Transformer-based cross-modal contextual interactions, the multi-head attention mechanism used in Transformers tends to assign greater weight to relevant intermodal contextual interactions, while ignoring irrelevant intermodal contextual information. This prevents the model from acquiring contextual interaction information that expresses different sentiments between modalities, thus affecting the final sentiment prediction results. Summary of the Invention
[0005] This application provides a multimodal sentiment analysis method, system, computer device, and readable storage medium, which solves the technical problem that the existing technology contains a large amount of noise in the data of each modality, which makes the model affected by the interference noise when performing cross-modal context interaction and sentiment prediction. Moreover, the existing technology ignores the unrelated contextual information between modalities, which makes the model unable to obtain the contextual interaction information expressing different emotions between modalities, thus affecting the final sentiment prediction results.
[0006] To achieve the above objectives, this application adopts the following technical solution:
[0007] In a first aspect, a multimodal sentiment analysis method is provided, characterized in that the method includes:
[0008] The raw video data is input into the feature extraction module to obtain context-level modal representations of the text modality, the acoustic modality, and the visual modality.
[0009] The contextual modal representations of the text modality, the acoustic modality, and the visual modality are input into the representation denoising module to obtain the noise-removed text modal representation, the noise-removed acoustic modal representation, and the noise-removed visual modal representation.
[0010] The noise-removed text modal representation, the noise-removed acoustic modal representation, and the noise-removed visual modal representation are input into the emotion enhancement module to obtain the emotion-enhanced acoustic modal representation and the emotion-enhanced visual modal representation.
[0011] The noise-removed text modal representation, the emotion-enhanced acoustic modal representation, and the emotion-enhanced visual modal representation are input into the context interaction module to obtain context interaction-enhanced text modal representation, context interaction-enhanced acoustic modal representation, and context interaction-enhanced visual modal representation.
[0012] The context-enhanced text modal representation, the context-enhanced acoustic modal representation, and the context-enhanced visual modal representation are input into the sentiment prediction module, and the sentiment prediction result is output.
[0013] Optionally, obtain context-level modal representations, including:
[0014] Divide a piece of raw data into multiple small video segments;
[0015] Extract discourse-level representations of the text modality, the acoustic modality, and the visual modality from the multiple small video clips;
[0016] Based on the correlation between the discourse-level representations of the text modality, the discourse-level representations of the acoustic modality, the discourse-level representations of the visual modality, and the discourse-level representations of other video segments, the discourse-level representations of the text modality, the discourse-level representations of the acoustic modality, and the discourse-level representations of the visual modality are fused to obtain the context-level modal representations of the text modality, the context-level modal representations of the acoustic modality, and the context-level modal representations of the visual modality.
[0017] Optionally, obtaining the noise-removed modal characterization includes:
[0018] The contextual modal representations of the text modality, the acoustic modality, and the visual modality are input into the common encoder to obtain the common representations of each contextual modality.
[0019] The common modal representations are removed from the context-level modal representations to obtain the non-common representations of each context-level modality;
[0020] The non-common representations of each context-level modality are input into the individual encoder to obtain the individual representations of each context-level modality;
[0021] The non-common representations of each context-level modality are decomposed into individual representations of each context-level modality and noise representations of each context-level modality, and the noise representations of each context-level modality are separated from the non-common representations of each context-level modality.
[0022] The common representations of each context-level modality and the individual representations of the corresponding modality are fused to obtain the noise-removed modal representations.
[0023] Optionally, the sum of the Euclidean distances between the common representations of any two modalities can be used as the commonality loss;
[0024] The sum of orthogonal constraints between the personality representations of any two modalities is taken as the personality loss;
[0025] The sum of the squared L2 norms of each mode is used as the noise loss;
[0026] The commonality loss, individuality loss, and noise loss are used to evaluate the effectiveness of the modal representations learned by the model.
[0027] Optionally, obtaining the emotion-enhanced acoustic modal representation and the emotion-enhanced visual modal representation includes:
[0028] The noise-removed modal representations are input into the emotion enhancement module;
[0029] The weight vectors between the noise-removed text modal representation and the noise-removed acoustic modal representation, as well as between the noise-removed text visual modal representation and the noise-removed visual modal representation, are calculated respectively.
[0030] The weight vectors are weighted and summed with the noise-removed text modal representations to obtain the enhanced acoustic modal representation and the enhanced visual modal representation.
[0031] The enhanced acoustic modal representation, the enhanced visual modal representation, the noise-removed acoustic modal representation, and the noise-removed visual modal representation are concatenated to obtain the emotion-enhanced acoustic modal representation and the emotion-enhanced visual modal representation.
[0032] Optionally, obtain the modal representation of each modal context interaction enhancement modality, including:
[0033] The noise-removed text modal representation, the emotion-enhanced acoustic modal representation, and the emotion-enhanced visual modal representation are input into the context interaction module;
[0034] The contextual sentiment association information in the noise-removed text modal representation, the sentiment-enhanced acoustic modal representation, and the sentiment-enhanced visual modal representation is extracted using a self-attention mechanism.
[0035] Cross-modal contextual sentiment association information among the noise-removed text modal representation, the sentiment-enhanced acoustic modal representation, and the sentiment-enhanced visual modal representation is mined using relevant attention mechanisms;
[0036] Cross-modal contextual sentiment-dissociated information among the noise-removed text modal representation, the sentiment-enhanced acoustic modal representation, and the sentiment-enhanced visual modal representation is mined using an irrelevant attention mechanism;
[0037] The cross-modal context sentiment association information, the cross-modal context sentiment non-association information, and the corresponding modal representations are summed to obtain the modal representations of each modal context interaction enhancement modality.
[0038] Secondly, a multimodal sentiment prediction system is provided, the multimodal sentiment prediction system comprising: a text representation extraction module, an acoustic representation extraction module, a visual representation extraction module, a representation denoising module, a sentiment enhancement module, a contextual interaction module, and a sentiment prediction module;
[0039] in,
[0040] The text representation extraction module is used to obtain text context-level modal representations;
[0041] The acoustic representation extraction module is used to obtain acoustic context-level modal representations;
[0042] The visual representation extraction module is used to obtain visual context-level modal representations;
[0043] The characterization denoising module is used to obtain the denoised text representation, the denoised acoustic representation, and the denoised visual representation.
[0044] The emotion enhancement module is used to acquire emotion enhancement acoustic modal representation and emotion enhancement visual modal representation;
[0045] The context interaction module is used to obtain context interaction enhanced text modal representation, context interaction enhanced acoustic modal representation, and context interaction enhanced visual modal representation;
[0046] The sentiment prediction module is used to obtain sentiment prediction results.
[0047] Optionally, the sentiment prediction module is a multimodal sentiment prediction network, which has a three-layer network structure, with the first two layers being feedforward layers and the third layer outputting sentiment polarity.
[0048] Thirdly, a computer device is provided, comprising: a memory and a processor, the memory being used to store a computer program; the processor being used to execute the method as described in any one of the first aspects when the computer program is invoked.
[0049] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of the first aspects.
[0050] This application provides a multimodal sentiment analysis method, the method comprising: inputting raw video data into a feature extraction module to obtain context-level modal representations of text modality, acoustic modality, and visual modality; inputting the context-level modal representations of text modality, acoustic modality, and visual modality into a representation denoising module to obtain noise-removed text modal representations, noise-removed acoustic modal representations, and noise-removed visual modal representations; and inputting the noise-removed text modal representations, noise-removed acoustic modal representations, and visual modal representations into a representation denoising module. The noise-removed visual modal representation is input into the sentiment enhancement module to obtain sentiment-enhanced acoustic and visual modal representations. The noise-removed text modal representation, the sentiment-enhanced acoustic and visual modal representations are then input into the context interaction module to obtain context-interactive enhanced text, acoustic, and visual modal representations. Finally, these context-interactive enhanced text, acoustic, and visual modal representations are input into the sentiment prediction module to output the sentiment prediction result. By obtaining modal representations that simultaneously consider commonalities, individual characteristics, and noise, and by mining relevant and irrelevant contextual interaction information between modalities, the accuracy of multimodal sentiment prediction can be improved. Attached Figure Description
[0051] Figure 1 A system schematic diagram of a multimodal sentiment analysis system provided in this application embodiment;
[0052] Figure 2 A schematic flowchart illustrating a multimodal sentiment analysis method provided in this application embodiment;
[0053] Figure 2A A schematic flowchart of a characterization denoising method provided in an embodiment of this application is shown;
[0054] Figure 2B A schematic flowchart of an emotion enhancement method provided in an embodiment of this application is shown;
[0055] Figure 2CThis diagram illustrates a schematic flowchart of a context interaction method provided by an embodiment of the present invention.
[0056] Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation
[0057] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of the principles and methods of well-known multimodal emotion learning models and electronic devices are omitted so as not to obscure the description of this application with unnecessary detail.
[0058] The terminology used in the following embodiments is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “the,” “the,” and “the” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise.
[0059] With the booming development of social media platforms such as Twitter, TikTok, and YouTube, traditional text sentiment analysis can no longer meet people's needs. Therefore, multimodal sentiment analysis, which connects various modalities and extracts attitudes, opinions, and emotional information from them, has begun to attract attention. The advantages of this multi-source data are crucial for sentiment analysis. For example, in sentiment prediction within ironic contexts, using multiple data sources together can easily detect the true emotions within those ironic situations.
[0060] Specifically, complex deep learning models can be constructed by leveraging complementary sentiment information between multimodal data to obtain multimodal representations that have a reliable mapping to sentiment polarity.
[0061] Correspondingly, constructing learning models using complementary sentiment information from multimodal data takes into account both common and unique sentiment information across modalities, neglecting the significant noise inherent in the data. This makes the model susceptible to interference noise when performing cross-modal contextual interactions and sentiment prediction. Furthermore, while learning models using complementary sentiment information from multimodal data also perform Transformer-based cross-modal contextual interactions, the multi-head attention mechanism used in Transformers tends to assign greater weight to relevant intermodal contextual interactions, while ignoring irrelevant intermodal contextual information. This prevents the model from acquiring contextual interaction information expressing different sentiments between modalities, thus affecting the final sentiment prediction results.
[0062] Therefore, this application proposes a multimodal sentiment analysis method, the method comprising: inputting raw video data into a feature extraction module to obtain context-level modal representations of text modality, acoustic modality, and visual modality; inputting the context-level modal representations of text modality, acoustic modality, and visual modality into a representation denoising module to obtain noise-removed text modal representations, noise-removed acoustic modal representations, and noise-removed visual modal representations; and inputting the noise-removed text modal representations, noise-removed acoustic modal representations, and visual modal representations into a representation denoising module to obtain noise-removed text modal representations, noise-removed acoustic modal representations, and visual modal representations; and inputting the noise-removed text modal representations, noise-removed acoustic modal representations, and visual modal representations into a representation denoising module. The noise-removed visual modal representation is input into the sentiment enhancement module to obtain sentiment-enhanced acoustic and visual modal representations. The noise-removed text modal representation, the sentiment-enhanced acoustic and visual modal representations are then input into the context interaction module to obtain context-interactive enhanced text, acoustic, and visual modal representations. Finally, these context-interactive enhanced text, acoustic, and visual modal representations are input into the sentiment prediction module to output the sentiment prediction result. By obtaining modal representations that simultaneously consider commonalities, individual characteristics, and noise, and by mining relevant and irrelevant contextual interaction information between modalities, the accuracy of multimodal sentiment prediction can be improved.
[0063] The following section first introduces the multimodal sentiment analysis system provided in the embodiments of this application. See also... Figure 1 , Figure 1 A system schematic diagram of a multimodal sentiment analysis system provided in this application embodiment is shown below. Figure 1As shown, the multimodal sentiment analysis system with hierarchical denoising, representation deentanglement, and dual-channel cross-modal interaction may include: a text representation extraction module 101, an acoustic representation extraction module 102, a visual representation extraction module 103, a representation denoising module 104, a sentiment enhancement module 105, a contextual interaction module 106, and a sentiment prediction module 107.
[0064] The original video data is input into the text representation extraction module 101, the acoustic representation extraction module 102, and the visual representation extraction module 103, respectively. After processing by these modules, corresponding text, acoustic, and visual representations are obtained. Specifically, the original video data can be divided into multiple small video segments. Then, the text, acoustic, and visual modal representations of the text, acoustic, and visual modalities are extracted from each small video segment using the text, acoustic, and visual representation modules 101, 102, and 103, respectively. Each sentence can be used as a segmentation point to divide the original video data.
[0065] The obtained text representation, acoustic representation, and visual representation are input into the representation denoising module 104 to obtain the noise-removed text representation, acoustic representation, and visual representation.
[0066] Furthermore, the obtained noise-removed text representation, acoustic representation, and visual representation are input into the emotion enhancement module 105 to obtain emotion-enhanced acoustic modal representation and visual modal representation.
[0067] Furthermore, the noise-removed text representation, the obtained emotion-enhanced acoustic representation, and the visual representation are input into the context interaction module 106 to obtain context-enhanced text representation, acoustic representation, and visual representation.
[0068] Furthermore, the obtained context-enhanced textual, acoustic, and visual representations are input into the sentiment prediction module 107 to obtain the final multimodal sentiment prediction result.
[0069] Figure 2 This schematic flowchart illustrates a multimodal sentiment analysis method provided in this application embodiment. It is intended as an example and not a limitation. This method can be applied to the aforementioned multimodal sentiment analysis system with hierarchical denoising, representation deentanglement, and dual-channel cross-modal context interaction. (See also...) Figure 2 The method includes:
[0070] Step 201: Input the raw video data into the feature extraction module to obtain context-level modal representations of each modality.
[0071] Specifically, inputting raw video data into the feature extraction module can segment a raw video data into multiple smaller video segments, using each sentence as a segmentation point. Then, feature extraction methods are used to extract discourse-level modal representations of text, acoustic, and visual modalities from each small video segment. Specifically, the GloVe model, COVAREP model, and Facet model can be used to encode the modal signals such as subtitles, audio, and images in the video segment, respectively, ultimately obtaining a text modal representation f of size 1*k. t Acoustic modal characterization f a and visual modal representation f v This application does not impose specific limitations on the model for obtaining modal characterization in its embodiments.
[0072] Furthermore, the text modality representation f t Acoustic modal characterization f a and visual modal representation f v The context between different video segments is mined by inputting them into the Bi-LSTM model. The highly relevant modal representations are then weighted and summed to obtain the context-level modal representations F of the text modality, acoustic modality, and visual modality. t F a and F v Their size is 1*d, where d∈R. In the application embodiment, R can be R={64,128,256}, but the application embodiment does not impose specific restrictions on the value of d.
[0073] Step 202: Input the context-level modal representation obtained in step 201 into the representation denoising module to obtain the noise-removed text modal representation, acoustic modal representation and visual modal representation.
[0074] Specifically, the representation denoising process is divided into two layers. In the first layer, the context-level modal representation is input into the commonality encoder to obtain the modal commonality representation. Then, the modal commonality representation is removed from the context-level modal representation to obtain the modal non-commonality representation. In the second layer, the modal non-commonality representation is input into the individuality encoder to obtain the modal individuality representation. Then, the modal individuality representation is removed from the modal non-commonality representation to obtain the modal noise representation. Different loss functions are used to constrain these representations, thereby learning modal representations that take into account individuality, commonality, and noise. Finally, the modal commonality representation and modal individuality representation are summed to obtain the noise-removed modal representations.
[0075] The characterization denoising process is described in detail below; see [link to relevant documentation]. Figure 2A . Figure 2A A schematic flowchart of a characterization denoising method provided in an embodiment of this application is shown. Figure 2 The characterization and denoising methods shown include:
[0076] Step 2021: Obtain the common representation of each mode through the common encoder. Common representation C n As shown in formula (1):
[0077]
[0078] in, This represents a common encoder that uses the same set of parameters θ when extracting common representations for each mode. c , n∈{t,a,v}.
[0079] Step 2022: Represent the contextual modality F n The decomposition involves common and non-common representations, and the non-common representations are separated from the context-level modal representations. Therefore, the non-common representations can be separated from the context-level modal features F. n It is separated from the middle, as shown in formula (2).
[0080]
[0081]
[0082] Among them, common characteristics C n It contains information expressing shared sentiments across modalities, rather than common representations. It is not useless for sentiment analysis; it contains information that reflects the individual sentiment between modalities. In contexts such as irony, this information can be helpful in predicting ironic sentiment.
[0083] Step 2023: Obtain the individual representation of each context-level modality through the individual encoder, as shown in formula (3):
[0084]
[0085] in, The personality encoder assigns a unique set of parameters when extracting personality representations for each context-level modality.
[0086] Step 2024: Decompose the non-common representation into individual representation and noise representation, and separate the noise representation. Among these, the non-common representation... It contains information on intermodal individual sentiment that has a positive effect on sentiment analysis. Therefore, it includes non-common representations. It is decomposed into two parts: individual representation and noise representation. Then, the noise representation is separated from the non-common representation. It is separated from the middle, as shown in formula (4):
[0087]
[0088]
[0089] Among them, personality representation P n It contains information expressing the individual emotions between different modalities. Noise characterization N... n It does not contain any emotional information, such as cluttered backgrounds and changing noise, which have no positive effect on sentiment analysis.
[0090] Step 2025: Characterize the common features C n and personality traits P n The two modal representations are then merged into a new modal representation (a noise-removed modal representation). As shown in formula (5):
[0091]
[0092] The constraints on the modal commonality representation, the modal individuality representation, and the modal noise representation are as follows:
[0093] It should be noted that the common loss γ cl This represents the difference between common representations of different modalities. The smaller the value, the more representative the common representations learned by the model are. Euclidean distance (ED) can be used to evaluate the difference between two representations. Euclidean distance measures the straight-line distance between two vectors in Euclidean space, that is, the length of the straight line connecting the two vectors. In two-dimensional space, Euclidean distance can be understood as the straight-line distance between two points; in higher-dimensional space, Euclidean distance is the straight-line distance in the corresponding multi-dimensional space. The sum of the Euclidean distances between the common representations of any two modalities can be used as the commonality loss, as shown in formula (6), where α is the weight coefficient of the commonality loss.
[0094] γ cl =α(ED(C) t C a )+ED(C t C v )+ED(C a C v (6)
[0095] It should also be noted that the loss of individuality (γ) pl This loss is used to measure the redundancy between personality representations across different modalities. It assesses whether the personality representations learned by the model can capture the personality sentiment of a modality. Orthogonality constraints can be used to calculate the personality loss. Assuming A and B are two representation matrices, with rows representing personality representation vectors, the orthogonality constraint can be expressed as... in It is the squared Frobenius norm. The smaller the value of OC, the more orthogonal the two representations of A and B are, and the greater the difference between A and B. The sum of the orthogonal constraints between the personality representations of any two modalities can be used as the personality loss, as shown in formula (7), where β is the weighting coefficient of the personality loss.
[0096] γ pl =β(OC(P) t ,P a )+OC(P t ,P ν )+OC(P a ,P ν (7)
[0097] It should be further noted that the noise loss γ nl The L2 norm is used to evaluate the magnitude of noise features; the smaller the value, the less noise is contained in the modal feature. The sum of the squared L2 norms of each mode is used as the noise loss, as shown in Equation (8), where γ is the weighting coefficient of the noise loss.
[0098]
[0099] The smaller the values of the commonality loss, the individuality loss, and the noise loss, the more representative the representation learned by the model is.
[0100] Step 203: Input the noise-removed modal representations into the emotion enhancement module to obtain the various modal representations for emotion enhancement. Specifically, calculate the weight vectors between the noise-removed text modal representation and the noise-removed acoustic modal representation, as well as between the noise-removed text modal representation and the noise-removed visual modal representation. Weight the weight vectors and the noise-removed text modal representations to obtain two enhanced modal representations. Concatenate the two enhanced modal representations with the noise-removed acoustic modal representation and the noise-removed visual modal representation, respectively, to obtain the various modal representations for emotion enhancement.
[0101] The following section details the methods for achieving emotion enhancement; see [link to relevant documentation]. Figure 2B . Figure 2B A schematic flowchart illustrating an emotion enhancement method provided in an embodiment of this application is shown. Figure 2B The emotion enhancement methods shown include:
[0102] Step 2031: The emotion enhancement module includes a TEA (Text-Enhanced Acoustic) module and a TEV (Text-Enhanced Visual) module. The TEA receives text and acoustic features as input, while the TEV receives text and visual features as input. First, the text modality features are concatenated with the acoustic modality features and the visual modality features respectively to obtain... and Use them to generate two weight vectors ω ta and ω tv As shown in formula (9):
[0103]
[0104] Where m∈{a,v}, θ tm These are network parameters, ReLU is a non-linear activation function, and L is a fully connected layer. These weights are used to extract emotional information from text features to enhance acoustic and visual features.
[0105] Step 2032: Move F t h The emotion vector F is obtained by fusing it with its respective weight vector to enhance acoustics and vision. ta and F tv As shown in Formula 10:
[0106] F tm =ω tm *L(F t h ;θ m (10)
[0107] Where m∈{a,v}, θ m These are network parameters, and L represents a fully connected layer network.
[0108] Step 2033: Concatenate and fuse the noise-removed acoustic modal representation and the noise-removed visual modal representation with the emotion vectors used to enhance acoustic and visual representations to obtain the emotion-enhanced acoustic and visual modal representations. and Furthermore, in order to keep the emotional vector within an ideal range, we use a scaling factor. Apply constraints as shown in formula (11):
[0109]
[0110]
[0111] Where m∈{a,v}, μ is the hyperparameter selected through cross-validation, ||*||2 is the L2 norm, and Dropout and LN are the dropout layer and normalization layer, respectively.
[0112] Step 204: Input the noise-removed text modal representation, the emotion-enhanced acoustic and visual modal representations into the context interaction module to obtain the interaction-enhanced modal representations after contextual interaction for each modality. Specifically, contextual sentiment association information in the modal representations can be mined using a self-attention mechanism; cross-modal contextual sentiment association information and cross-modal contextual sentiment non-association information of the noise-removed text modality, the emotion-enhanced acoustic modality, and the emotion-enhanced visual modality can be mined using relevant and irrelevant attention mechanisms, respectively, and these information and modal representations are summed; finally, the contextual interaction-enhanced modal representations for each modality are obtained.
[0113] The following section details the implementation methods for contextual interaction; see [link to relevant documentation]. Figure 2C , Figure 2C A schematic flowchart of a context interaction method provided by an embodiment of the present invention is shown; as follows: Figure 2C The context interaction methods shown include:
[0114] Step 2041: Acoustic features Transformed into a query Q through a fully connected layer a Key K a Sum V a Using Q a and K a Calculate the similarity weight matrix using vectors, and then sum it with vector V. a A new feature vector is obtained by weighted summation. Each computation is treated as a separate head, and the intra-modal sentiment features are obtained by concatenating the outputs of multiple heads. As shown in formula (12):
[0115]
[0116] Among them, W Q W K W V W0 represents the corresponding weight, d represents the dimension, and h represents the number of heads.
[0117] Step 2042: In the first channel, using acoustic features and the combined visual and textual features Get query key Sum Then, the intermodal similarity emotion features are calculated. As shown in formula (13):
[0118]
[0119]
[0120]
[0121] in, These are the corresponding weights.
[0122] Step 2043: In another channel, acoustic features are used. and the combined visual and textual features Get query key Sum However, after obtaining the similarity weight matrix, it is inverted to ensure that the weighted summation focuses on the sentiment information of intermodal differences. Then, the sentiment features of intermodal differences are calculated. As shown in formula (14):
[0123]
[0124]
[0125]
[0126] in, These are the corresponding weights, and negate(*) represents the negation operation.
[0127] Step 2044: Finally, the obtained intermodal similarity and difference sentiment features are... and After fusion, the result is fed into the FFN layer. The output of each layer is processed by residual transformation and normalization, as shown in formula (15):
[0128]
[0129] FFN is a feedforward neural network layer.
[0130] It should be noted that the above embodiments provide a detailed description of the method for obtaining context-interactive enhanced modal representations by taking the acquisition of context-interactive enhanced acoustic modal representations as an example. The methods for obtaining context-interactive enhanced text modal representations and context-interactive enhanced visual modal representations are similar and will not be described again here.
[0131] Step 205: Input the modal representations of each interaction enhancement into the sentiment prediction network and output the sentiment prediction result. Specifically, the modal representations of each interaction enhancement are input into a multilayer perceptron (MLP) to achieve sentiment classification. The MLP contains a three-layer network structure, the first two layers are feedforward layers, each layer is followed by a ReLU activation function, and the last layer outputs the sentiment polarity, as shown in formula (16):
[0132]
[0133] in, It is a value between -3 and +3, representing the intensity of emotion.
[0134] Table (1) compares the results of the method provided in this embodiment with those of various representative methods. This comparison was conducted on the CMU-MOSI public dataset in the field of multimodal sentiment analysis, mainly testing the model's sentiment 2-classification performance, i.e., determining whether the sentiment category of a sample is "positive" or "negative". The test selected the accuracy and F1 score of sentiment 2-classification as evaluation indicators. Accuracy refers to the ratio of the number of correctly classified sentiments in the sample prediction results to the total number of samples. The higher the value, the better the prediction effect. The F1 score is the harmonic mean of precision and recall. The higher the value, the better the prediction effect. As shown in Table 1, compared with several classic models in the field of multimodal sentiment analysis, this embodiment of the invention achieved better performance in the sentiment 2-classification test, improving the accuracy and F1 score by 1.9% and 1.9% respectively compared with the suboptimal model TMPSA.
[0135] Table 1
[0136]
[0137] In summary, the multimodal sentiment analysis method provided in this application includes: inputting raw video data into a feature extraction module to obtain context-level modal representations of text, acoustic, and visual modalities; inputting the context-level modal representations of text, acoustic, and visual modalities into a representation denoising module to obtain noise-removed text, acoustic, and visual modal representations; and inputting the noise-removed text and acoustic modal representations into a representation denoising module to obtain noise-removed text, acoustic, and visual modal representations; and inputting the noise-removed text and acoustic modal representations into a representation denoising module. The noise-removed visual modal representation is input into the emotion enhancement module to obtain emotion-enhanced acoustic and visual modal representations. The noise-removed text modal representation, the emotion-enhanced acoustic and visual modal representations are then input into the context interaction module to obtain context-interactive enhanced text, acoustic, and visual modal representations. Finally, these context-interactive enhanced text, acoustic, and visual modal representations are input into the emotion prediction module to output the emotion prediction result. By obtaining modal representations that simultaneously consider commonalities, individual characteristics, and noise, and by mining relevant and irrelevant contextual interaction information between modalities, the accuracy of multimodal emotion prediction can be improved.
[0138] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0139] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0140] Based on the same inventive concept, this application also provides a computer device. Figure 3 This application provides a schematic diagram of the structure of a computer device, as shown in the embodiment of the present application. Figure 3 As shown, the computer device provided in this embodiment may include: a memory 31 and a processor 32. The memory 31 is used to store a computer program 33; the processor 32 is used to execute the method described in the above method embodiment when the computer program 33 is invoked.
[0141] The computer device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so they will not be described again here.
[0142] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the methods described in the above-described method embodiments.
[0143] This application also provides a computer program product that, when run on a computer device, enables the computer device to implement the methods described in the above-described method embodiments.
[0144] If the integrated units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium can include at least: any entity or device capable of carrying computer program code to a photographic device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.
[0145] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0146] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0147] In the embodiments provided in this application, it should be understood that the disclosed apparatus / devices and methods can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0148] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.
[0149] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0150] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."
[0151] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0152] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.
[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.
Claims
1. A multimodal sentiment analysis method, characterized in that, The method includes: The raw video data is input into the feature extraction module to obtain context-level modal representations of the text modality, the acoustic modality, and the visual modality. The contextual modal representations of the text modality, the acoustic modality, and the visual modality are input into the representation denoising module to obtain the noise-removed text modal representation, the noise-removed acoustic modal representation, and the noise-removed visual modal representation. The noise-removed text modal representation, the noise-removed acoustic modal representation, and the noise-removed visual modal representation are input into the emotion enhancement module to obtain the emotion-enhanced acoustic modal representation and the emotion-enhanced visual modal representation. The noise-removed text modal representation, the emotion-enhanced acoustic modal representation, and the emotion-enhanced visual modal representation are input into the context interaction module to obtain context interaction-enhanced text modal representation, context interaction-enhanced acoustic modal representation, and context interaction-enhanced visual modal representation. The context-enhanced text modal representation, the context-enhanced acoustic modal representation, and the context-enhanced visual modal representation are input into the sentiment prediction module, and the sentiment prediction result is output. Specifically, the contextual sentiment association information in the noise-removed text modal representation, the sentiment-enhanced acoustic modal representation, and the sentiment-enhanced visual modal representation is mined through a self-attention mechanism; Cross-modal contextual sentiment association information among the noise-removed text modal representation, the sentiment-enhanced acoustic modal representation, and the sentiment-enhanced visual modal representation is mined using relevant attention mechanisms; Cross-modal contextual sentiment-dissociated information among the noise-removed text modal representation, the sentiment-enhanced acoustic modal representation, and the sentiment-enhanced visual modal representation is mined using an irrelevant attention mechanism; The cross-modal context sentiment association information, the cross-modal context sentiment non-association information, and the corresponding modal representations are summed to obtain the modal representations of each modal context interaction enhancement modality. Specifically, mining cross-modal contextual sentiment-dissociated information among the noise-removed text modal representation, the sentiment-enhanced acoustic modal representation, and the sentiment-enhanced visual modal representation using an irrelevant attention mechanism includes: using acoustic features and the combined visual and textual features Get query ,key Sum However, after obtaining the similarity weight matrix, the similarity weight matrix is inverted, and then the intermodal difference sentiment features are calculated. The details are as follows: in, These are the corresponding weights. This indicates the negation operation.
2. The method according to claim 1, characterized in that, Obtaining context-level modal representations includes: Divide a piece of raw data into multiple small video segments; Extract discourse-level representations of the text modality, the acoustic modality, and the visual modality from the multiple small video clips; Based on the correlation between the discourse-level representations of the text modality, the discourse-level representations of the acoustic modality, the discourse-level representations of the visual modality, and the discourse-level representations of other video segments, the discourse-level representations of the text modality, the discourse-level representations of the acoustic modality, and the discourse-level representations of the visual modality are fused to obtain the context-level modal representations of the text modality, the context-level modal representations of the acoustic modality, and the context-level modal representations of the visual modality.
3. The method according to claim 1, characterized in that, Obtaining noise-removed modal characterization includes: The contextual modal representations of the text modality, the acoustic modality, and the visual modality are input into the common encoder to obtain the common representations of each contextual modality. The common modal representations are removed from the context-level modal representations to obtain the non-common representations of each context-level modality; The non-common representations of each context-level modality are input into the individual encoder to obtain the individual representations of each context-level modality; The non-common representations of each context-level modality are decomposed into individual representations of each context-level modality and noise representations of each context-level modality, and the noise representations of each context-level modality are separated from the non-common representations of each context-level modality. The common representations of each context-level modality and the individual representations of the corresponding modality are fused to obtain the noise-removed modal representations.
4. The method according to claim 3, characterized in that, The sum of the Euclidean distances between the common representations of any two modalities is taken as the common loss; The sum of orthogonal constraints between the personality representations of any two modalities is taken as the personality loss; The sum of the squared L2 norms of each mode is used as the noise loss; The commonality loss, individuality loss, and noise loss are used to evaluate the effectiveness of the modal representations learned by the model.
5. The method according to claim 1, characterized in that, The acquisition of emotion-enhanced acoustic modal representation and emotion-enhanced visual modal representation includes: The noise-removed modal representations are input into the emotion enhancement module; The weight vectors between the noise-removed text modal representation and the noise-removed acoustic modal representation, as well as between the noise-removed text modal representation and the noise-removed visual modal representation, are calculated respectively. The weight vectors are weighted and summed with the noise-removed text modal representations to obtain the enhanced acoustic modal representation and the enhanced visual modal representation. The enhanced acoustic modal representation, the enhanced visual modal representation, the noise-removed acoustic modal representation, and the noise-removed visual modal representation are concatenated to obtain the emotion-enhanced acoustic modal representation and the emotion-enhanced visual modal representation.
6. A multimodal sentiment prediction system, characterized in that, include: The module includes a text representation extraction module, an acoustic representation extraction module, a visual representation extraction module, a representation denoising module, a sentiment enhancement module, a contextual interaction module, and a sentiment prediction module. in, The text representation extraction module is used to obtain text context-level modal representations; The acoustic representation extraction module is used to obtain acoustic context-level modal representations; The visual representation extraction module is used to obtain visual context-level modal representations; The characterization denoising module is used to obtain the denoised text representation, the denoised acoustic representation, and the denoised visual representation. The emotion enhancement module is used to acquire emotion enhancement acoustic modal representation and emotion enhancement visual modal representation; The context interaction module is used to obtain context interaction enhanced text modal representation, context interaction enhanced acoustic modal representation, and context interaction enhanced visual modal representation; The sentiment prediction module is used to obtain sentiment prediction results; The context interaction module extracts contextual sentiment association information from the noise-removed text modal representation, the emotion-enhanced acoustic modal representation, and the emotion-enhanced visual modal representation through a self-attention mechanism. Cross-modal contextual sentiment association information among the noise-removed text modal representation, the sentiment-enhanced acoustic modal representation, and the sentiment-enhanced visual modal representation is mined using relevant attention mechanisms; Cross-modal contextual sentiment-dissociated information among the noise-removed text modal representation, the sentiment-enhanced acoustic modal representation, and the sentiment-enhanced visual modal representation is mined using an irrelevant attention mechanism; The cross-modal context sentiment association information, the cross-modal context sentiment non-association information, and the corresponding modal representations are summed to obtain the modal representations of each modal context interaction enhancement modality. Specifically, mining cross-modal contextual sentiment-dissociated information among the noise-removed text modal representation, the sentiment-enhanced acoustic modal representation, and the sentiment-enhanced visual modal representation using an irrelevant attention mechanism includes: using acoustic features and the combined visual and textual features Get query ,key Sum However, after obtaining the similarity weight matrix, the similarity weight matrix is inverted, and then the intermodal difference sentiment features are calculated. The details are as follows: in, These are the corresponding weights. This indicates the negation operation.
7. The multimodal sentiment prediction system according to claim 6, characterized in that, The sentiment prediction module is a multimodal sentiment prediction network, which has a three-layer network structure. The first two layers are feedforward layers, and the third layer outputs the sentiment polarity.
8. A computer device, characterized in that, include: A memory and a processor, wherein the memory is used to store computer programs; The processor is configured to perform the method as described in any one of claims 1 to 5 when the computer program is invoked.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 5.
