Multi-modal sentiment analysis method and product based on sentiment perception characterization decoupling and offset
By adopting the decoupling and offsetting method of emotion-perceptual representation in multimodal sentiment analysis, decoupling and offset non-text modal features, the problem of neglecting modal interaction influence is solved, and the accuracy of feature extraction and fusion is improved , reducing the difficulty and labeling cost of model fine-tuning.
Patent Information
- Application Number
- CN202411479801.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-23
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-10-23
AI Technical Summary
The existing multimodal sentiment analysis methods ignore the interaction and mutual influence between modals in the process of feature extraction and fusion, resulting in redundant information shifting to text modes, increasing the difficulty of fine-tuning downstream models, and single-modal Annotation is expensive and may contain noise.
Using a multimodal emotion analysis method based on emotion perception representation decoupling and offset, the interactive emotion perception representation decoupling module and attention cross-modal representation offset decoupling module are extracted and decoupled respectively. , using the cross attention autoencoder and projection module to project non-text modal features into the text representation space, perform emotional polarity analysis and sentiment analysis, and generate emotional offset text vectors for fine-tuning of pre-trained language models.
Effectively utilize the complementarity of multimodal information, reduce the offset of redundant information, reduce the difficulty of fine-tuning downstream models, and reduce the cost of single-modal annotation through an unsupervised label training system.
Smart Images

Figure CN119004391A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to the fields of artificial intelligence and emotional computing technology, and in particular to a multimodal sentiment analysis method and product based on decoupling and offset of sentiment perception representation. Background Art
[0002] With the rapid development and widespread popularity of mobile devices and social networks, people are increasingly fond of generating and posting short videos on video social platforms to share their experiences and express their emotions. Analyzing the emotions and feelings contained in these user-generated videos plays an important role in various relevant practical applications, such as human-computer interaction and mental health monitoring.
[0003] Compared with early sentiment analysis that only relies on language, multimodal sentiment analysis (MSA) has attracted more and more attention due to its advantages such as data complementarity, model robustness and superior performance.
[0004] Multimodal sentiment analysis basically follows the process of feature extraction, feature fusion, and feature-to-sentiment mapping. The main challenge facing multimodal sentiment analysis at present is how to extract the discriminative features of each modality and fuse multimodal representations. Among the related technologies, MSA methods mainly focus on designing complex fusion mechanisms to integrate the features extracted from multiple modalities. Early fusion techniques are basically model-free, including feature-level (early) fusion, decision-level (late) fusion, and hybrid fusion. Although these methods are easy to understand, they cannot model the interaction of multimodal features well.
[0005] In recent years, model-based fusion techniques have emerged, which can be divided into the following types: (1) Tensor-level multimodal feature fusion. This method mainly uses vector outer products, such as recurrent neural networks (RNNs) and long short-term memory networks (LSTMs), to fuse the extracted features. Specifically, the tensor fusion model (TFN) uses triple Cartesian products to fuse multimodal features; the memory fusion network (MFN) uses LSTM to extract unimodal features and uses Delta memory attention networks and multi-view gate memory for modal interaction; the low-rank multimodal fusion method uses modality-specific low-rank factors for multimodal fusion, which speeds up the fusion process. (2) Word-level multimodal feature fusion. This method requires the use of forced alignment to align words between speech and audio to obtain word-level features. Specifically, the recursive attention variation embedding network (RAVEN) uses an attention gating module to fuse word-level features and dynamically use non-linguistic features to offset word embeddings; a multimodal adaptation gate is introduced in BERT to replace the original word embedding with the offset word-level features. In the MSA method proposed in the related art, features of different modalities are usually extracted independently, ignoring the phenomenon that unimodal emotions may be affected by other modal information. Summary of the invention
[0006] The embodiments of the present invention provide a multimodal sentiment analysis method and product based on the decoupling and offset of sentiment perception representation.
[0007] A first aspect of an embodiment of the present invention provides a multimodal sentiment analysis method based on sentiment perception representation decoupling and offset, which is applied to a sentiment perception representation decoupling and offset system. The method includes: Acquire a multimodal feature, wherein the multimodal feature includes: a text modal feature and a plurality of non-text modal features; Through the representation decoupling module of interactive emotion perception, each non-text modal feature is taken as input, and the text modal features and other non-text modal features are used as guidance to perform sentiment polarity analysis, and obtain the sentiment polarity representation corresponding to each non-text modal feature. Each non-text modal feature is taken as input, and the text modal features, other non-text modal features, and the sentiment polarity representation corresponding to each non-text modal feature are used as guidance to perform sentiment analysis, and obtain the sentiment representation corresponding to each non-text modal feature. Through the attention cross-modal representation offset decoupling module, the sentiment representations corresponding to multiple non-text modal features are projected into the text representation space, and the sentiment offset text vector is obtained according to the offset scaling factors corresponding to each of the multiple non-text modal features, the sentiment representations corresponding to the projected non-text modal features, and the text modal features. The sentiment-shifted text vector is input into the fine-tuned pre-trained language model to obtain the sentiment classification result.
[0008] Optionally, the representation decoupling module of the interactive emotion perception includes: a coarse-grained cross-attention autoencoder and a fine-grained cross-attention autoencoder, the coarse-grained cross-attention autoencoder is used for emotion polarity analysis, and the fine-grained cross-attention autoencoder is used for emotion analysis; The coarse-grained cross-attention autoencoder includes: a coarse-grained emotion polarity encoder, a coarse-grained non-emotion polarity encoder, and a coarse-grained emotion polarity decoder. The coarse-grained sentiment polarity encoder takes each non-text modality feature, text modality feature and other non-text modality features as input, and outputs the sentiment polarity representation corresponding to each non-text modality feature; The coarse-grained non-sentiment polarity encoder takes each non-text modality feature as input and outputs the non-sentiment polarity representation corresponding to each non-text modality feature; The coarse-grained sentiment polarity decoder takes the output of the coarse-grained sentiment polarity encoder, the output of the coarse-grained non-sentiment polarity encoder, text modality features and other non-text modality features as input, and outputs the reconstructed coarse-grained non-text modality features.
[0009] Optionally, the fine-grained cross-attention autoencoder includes: a fine-grained emotion encoder, a fine-grained non-emotion encoder and a fine-grained emotion decoder, The fine-grained sentiment encoder takes each non-text modality feature, text modality feature, other non-text modality features and the sentiment polarity representation corresponding to each non-text modality feature as input, and takes the sentiment representation corresponding to each non-text modality feature as output; The fine-grained non-emotional encoder takes each non-text modal feature and the sentiment polarity representation corresponding to each non-text modal feature as input, and takes the non-emotional representation corresponding to each non-text modal feature as output; The fine-grained sentiment decoder takes the output of the fine-grained sentiment encoder, the output of the fine-grained non-sentiment encoder, text modality features and other non-text modality features as input, and outputs the reconstructed fine-grained non-text modality features.
[0010] Optionally, the reconstruction loss of the coarse-grained criss-cross attention autoencoder and the fine-grained criss-cross attention autoencoder is: ; in, represents a batch of samples, represents the reconstruction loss of the coarse-grained criss-cross attention autoencoder, represents the reconstruction loss of the fine-grained criss-cross attention autoencoder; Indicates i Non-text modality features of samples , Represents the reconstructed coarse-grained non-text modality features of the i-th sample , Indicates i Reconstructed fine-grained non-text modality features of samples ; , ; in, is a coarse-grained sentiment polarity decoder, is a fine-grained emotion decoder; ; ; is the sentiment polarity representation corresponding to each non-text modality feature, is the non-emotional polarity representation corresponding to each non-text modality feature; is the sentiment representation corresponding to each non-text modality feature is the non-emotional representation corresponding to each non-text modality feature; is a non-text modality feature, are other non-text modality features, is a coarse-grained sentiment polarity encoder, is a coarse-grained non-sentiment polarity encoder, is a fine-grained sentiment encoder, It is a fine-grained non-emotional semantic encoder; The contrast loss of the coarse-grained criss-cross attention autoencoder and the fine-grained criss-cross attention autoencoder is: ; , ; in, is the contrastive loss of the coarse-grained criss-cross attention autoencoder, is the contrastive loss of the fine-grained criss-cross attention autoencoder, yes All positive samples in Or all negative samples ; include or , and Differentiate by emotional polarity, or , and Differentiate by emotion category. or , and Distinguish by non-text modal features; X , Indicates the first i The samples obtained , express Except for i The samples other than the samples ; The orthogonal loss of the coarse-grained criss-cross attention autoencoder and the fine-grained criss-cross attention autoencoder is: ; in, is the orthogonal loss of the coarse-grained criss-cross attention autoencoder, is the orthogonal loss of the fine-grained criss-cross attention autoencoder, They are i The sentiment polarity representation and non-sentiment polarity representation corresponding to the non-text modal features obtained by samples, They are i The sentiment and non-sentiment representations corresponding to the non-text modal features obtained from samples.
[0011] Optionally, the attention cross-modal representation offset decoupling module includes a projection module and an offset module, wherein the projection module projects the sentiment representations corresponding to the multiple non-text modal features into the text representation space, and the offset module controls the offset degree of the sentiment offset text vector based on the offset scaling factors corresponding to the multiple non-text modal features, wherein the offset scaling factor is: ; in, Indicates m The offset scaling factor corresponding to the non-text modality features, , represents the sentiment representation corresponding to the projected non-text modality features, is a hyperparameter.
[0012] Optionally, the loss function of the pre-trained language model is: or ; ; It is i The true value of the sentiment of samples, It is i The sentiment prediction value of samples, It is a pre-trained language model; is the number of emotion types, is a binary indicator function, Indicates that based on i The sentiment shifted text vector obtained by samples.
[0013] Optionally, the emotion perception representation decoupling and offset system is trained based on the following steps: Get the pre-trained language model and freeze the parameters of the pre-trained language model; Using the multimodal analysis dataset, the first equilibrium coefficient was used Training the emotion perception representation decoupling and offset system, and updating various parameters of the interactive emotion perception representation decoupling module and the attention cross-modal representation offset decoupling module; Using the sentiment classification dataset, we use the second balance coefficient Update the parameters of the projection module and pre-trained language model; ; Indicates the first balance coefficient or the second balance coefficient.
[0014] A second aspect of an embodiment of the present invention provides a multimodal sentiment analysis device based on sentiment perception representation decoupling and offset, which is applied to a sentiment perception representation decoupling and offset system, and the device includes: An acquisition module, used for acquiring multimodal features, wherein the multimodal features include: a text modal feature and a plurality of non-text modal features; A representation module is used for performing sentiment polarity analysis using each non-text modal feature as input and guided by text modal features and other non-text modal features through an interactive sentiment perception representation decoupling module to obtain a sentiment polarity representation corresponding to each non-text modal feature; a representation module is used for performing sentiment analysis using each non-text modal feature as input and guided by text modal features, other non-text modal features and the sentiment polarity representation corresponding to each non-text modal feature to obtain a sentiment representation corresponding to each non-text modal feature; The sentiment offset text vector determination module is used to project the sentiment representations corresponding to multiple non-text modal features into the text representation space through the attention cross-modal representation offset decoupling module, and obtain the sentiment offset text vector according to the offset scaling factors corresponding to each of the multiple non-text modal features, the sentiment representations corresponding to the projected non-text modal features, and the text modal features; The classification module is used to input the sentiment-shifted text vector into the fine-tuned pre-trained language model to obtain the sentiment classification result.
[0015] A third aspect of an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executed, implements the multimodal sentiment analysis method based on decoupling and offset of sentiment perception representation as described in the first aspect of the present invention.
[0016] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multimodal sentiment analysis method based on decoupling and offset of emotion perception representation as described in the first aspect of the present invention.
[0017] A fifth aspect of an embodiment of the present invention provides a computer program product, including a computer program / instruction, which is implemented by a processor to implement the steps in the multimodal sentiment analysis method based on decoupling and offset of emotion perception representation as described in the first aspect of the present invention.
[0018] In an embodiment of the present invention, in the process of obtaining the emotion representation corresponding to each non-text modal feature, information of different modalities is taken into consideration. Specifically, through the representation decoupling module of interactive emotion perception, specific emotion representations are extracted under the interaction and mutual influence of information of multiple modalities, and redundant features irrelevant to emotion in each non-text modal feature are removed. The emotion representation corresponding to each decoupled non-text modal feature is used to offset the original text modal feature, so that a large amount of redundant and irrelevant information will not be offset to the text modal feature; the offset operation will not destroy the original text modal feature, thereby reducing the difficulty of fine-tuning the downstream pre-trained language model.
[0019] In addition, in an embodiment of the present invention, the sentiment classification label and the sentiment polarity label can be obtained based on the sentiment label of the data corresponding to the multimodal features. Based on these two labels, the sentiment perception representation decoupling and offset system can be trained without labeling the unimodal features. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0021] Figure 1 It is a flowchart of the steps of a multimodal sentiment analysis method based on sentiment perception representation decoupling and offset provided by an embodiment of the present invention; Figure 2 It is a structural schematic diagram of an emotion perception representation decoupling and offset system in a multimodal emotion analysis method based on emotion perception representation decoupling and offset provided by an embodiment of the present invention; Figure 3 It is a structural diagram of a coarse-grained cross-attention autoencoder in a representation decoupling module of interactive emotion perception in a multimodal emotion analysis method based on emotion perception representation decoupling and offset provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0022] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0023] Although some methods proposed in the related art consider the differences within each modality, the decoupling features between the modalities are performed separately. These methods often only consider the mutual influence of multimodal information in the feature fusion process, but ignore this point in the feature extraction process. The inventors have explored and found that these methods have the following disadvantages: (1) The interaction and mutual influence of multimodal information are ignored in the feature extraction process; (2) The lack of information filtering for non-language modalities causes a large amount of redundant and irrelevant information to be shifted to the text modality. The shift operation destroys the original text feature space and increases the difficulty of fine-tuning the downstream pre-trained language model. (3) Unimodal annotation is expensive, and unsupervised generated unimodal labels may contain noise.
[0024] In the related art, multimodal sentiment computing methods mainly focus on designing complex feature fusion strategies to integrate the representations of different modalities extracted separately, ignoring the phenomenon that a single modal sentiment may be affected by other modal information. The embodiment of the present invention proposes to explore the interaction between different modalities in the feature extraction and feature fusion stages to more deeply utilize the complementary information of multimodal sentiment data.
[0025] Specifically, the embodiment of the present invention proposes a multimodal sentiment analysis method based on the decoupling and offset of sentiment perception representation, such as Figure 1 As shown, it shows a flowchart of the steps of a multimodal sentiment analysis method based on sentiment perception representation decoupling and offset proposed in an embodiment of the present invention. The method is applied to a sentiment perception representation decoupling and offset system, and the method includes the following steps: S101, obtaining a multimodal feature, wherein the multimodal feature includes: a text modal feature and a plurality of non-text modal features; S102, using the representation decoupling module of interactive emotion perception, taking each non-text modal feature as input, guided by the text modal feature and other non-text modal features, performing sentiment polarity analysis to obtain the sentiment polarity representation corresponding to each non-text modal feature, taking each non-text modal feature as input, guided by the text modal feature, other non-text modal features and the sentiment polarity representation corresponding to each non-text modal feature, performing sentiment analysis to obtain the sentiment representation corresponding to each non-text modal feature; S103, projecting the sentiment representations corresponding to the multiple non-text modal features into the text representation space through the attention cross-modal representation offset decoupling module, and obtaining the sentiment offset text vector according to the offset scaling factors corresponding to the multiple non-text modal features, the sentiment representations corresponding to the projected non-text modal features, and the text modal features; S104, inputting the sentiment offset text vector into the fine-tuned pre-trained language model to obtain a sentiment classification result.
[0026] For ease of understanding, the embodiment of the present invention also provides a structural schematic diagram of the emotion perception representation decoupling and offset system in the multimodal emotion analysis method based on emotion perception representation decoupling and offset, such as Figure 2 shown.
[0027] Specifically, in the embodiment of the present invention, the multimodal features are features of multiple modes of the same data. For example, for video data, its multimodal features include text modal features, video modal features, and audio modal features.
[0028] In the embodiment of the present invention, the representation decoupling module of interactive emotion perception aims to learn emotion-specific feature representations through a cross-attention autoencoder (CAAE). The encoder and decoder in CAAE are composed of multiple layers of Transformer-style cross-attention (CA) layers, which consider information from other modalities when extracting emotion-related features, thereby capturing the true expression of emotions in a single modality in a multimodal scenario.
[0029] In an embodiment of the present invention, a coarse-to-fine strategy is adopted to train the cross-attention autoencoder, that is, firstly, the coarse-grained cross-attention autoencoder is trained to learn the representation of emotion polarity, and then these representations are used to guide the fine-grained cross-attention autoencoder to learn the emotion representation, and finally a representation decoupling module of interactive emotion perception is obtained.
[0030] In the embodiment of the present invention, considering the important influence of text semantics on sentiment analysis, the embodiment of the present invention proposes to use the sentiment representations corresponding to the decomposed multiple non-text modal features to offset the original text modal features. Specifically, the sentiment representations corresponding to the multiple non-text modal features are first projected into the text representation space. Through this offset operation, the change of the original semantics can be minimized, and the influence of non-language modalities on sentiment can be explored, which also makes the fused word embedding easier for the downstream language model to fine-tune.
[0031] In the embodiments of the present invention, emotion refers to a variety of specific emotions, such as happiness, sadness, fear, irony, surprise, etc., and emotion analysis refers to analyzing the corresponding specific emotions based on input features. Emotion polarity refers to the polarity classification corresponding to emotions, such as positive, negative, neutral, etc. For example, the emotion polarity corresponding to happiness is positive, and the emotion polarity corresponding to sadness, irony, and fear is negative. Emotion polarity analysis refers to analyzing the corresponding emotion polarity based on input features.
[0032] In an embodiment of the present invention, sentiment polarity analysis is first performed to obtain the sentiment polarity representation corresponding to each non-text modal feature, and then sentiment analysis is performed based on the sentiment polarity representation corresponding to the non-text modal feature to obtain the sentiment representation corresponding to each non-text modal feature, thereby realizing sentiment analysis from coarse-grained to fine-grained.
[0033] In an embodiment of the present invention, when analyzing the sentiment polarity, the currently targeted non-text modal features, text modal features and other non-text modal features are taken into consideration to obtain a sentiment polarity representation of the currently targeted non-text modal features, and then based on the sentiment polarity representation and the non-text modal features, text modal features and other non-text modal features, the sentiment representation corresponding to the currently targeted non-text modal features is obtained.
[0034] For example, for a certain video, in which a person is speaking happily, a sentiment polarity analysis can be performed based on its video modal features, text modal features, and audio modal features to obtain a sentiment polarity representation representing positivity, and then a sentiment analysis can be performed based on the sentiment polarity representation, video modal features, text modal features, and audio modal features to obtain a sentiment representation representing happiness.
[0035] For another example, for a certain video, in which a character is smiling and remarking on a certain point of view, we can perform sentiment polarity analysis based on its video modal features, text modal features, and audio modal features to obtain a sentiment polarity representation representing negativity. Then, we can perform sentiment analysis based on the sentiment polarity representation, video modal features, text modal features, and audio modal features to obtain a sentiment representation representing irony without being disturbed by the smiling expression.
[0036] In an embodiment of the present invention, the representation decoupling module of interactive emotion perception includes: a coarse-grained cross-attention autoencoder and a fine-grained cross-attention autoencoder, the coarse-grained cross-attention autoencoder is used for emotion polarity analysis, and the fine-grained cross-attention autoencoder is used for emotion analysis; The coarse-grained cross-attention autoencoder includes: a coarse-grained emotion polarity encoder, a coarse-grained non-emotion polarity encoder, and a coarse-grained emotion polarity decoder. The coarse-grained sentiment polarity encoder takes each non-text modality feature, text modality feature and other non-text modality features as input, and outputs the sentiment polarity representation corresponding to each non-text modality feature; The coarse-grained non-sentiment polarity encoder takes each non-text modality feature as input and outputs the non-sentiment polarity representation corresponding to each non-text modality feature; The coarse-grained sentiment polarity decoder takes the output of the coarse-grained sentiment polarity encoder, the output of the coarse-grained non-sentiment polarity encoder, text modality features and other non-text modality features as input, and outputs the reconstructed coarse-grained non-text modality features.
[0037] The fine-grained cross-attention autoencoder includes: a fine-grained emotion encoder, a fine-grained non-emotion encoder and a fine-grained emotion decoder. The fine-grained sentiment encoder takes each non-text modality feature, text modality feature, other non-text modality features and the sentiment polarity representation corresponding to each non-text modality feature as input, and takes the sentiment representation corresponding to each non-text modality feature as output; The fine-grained non-emotional encoder takes each non-text modal feature and the sentiment polarity representation corresponding to each non-text modal feature as input, and takes the non-emotional representation corresponding to each non-text modal feature as output; The fine-grained sentiment decoder takes the output of the fine-grained sentiment encoder, the output of the fine-grained non-sentiment encoder, text modality features and other non-text modality features as input, and outputs the reconstructed fine-grained non-text modality features.
[0038] In the embodiment of the present invention, a method similar to an autoencoder is used for feature extraction to accurately obtain the emotion representation of non-language modalities. In the embodiment of the present invention, a stacked cross-attention layer is used to construct an emotion feature encoder, and the input guiding the decoupling of the emotion representation is used as the key (K) and value (V) in the cross-attention mechanism.
[0039] Specifically, an embodiment of the present invention also provides a schematic diagram of the structure of a coarse-grained cross-attention autoencoder in a representation decoupling module for interactive emotion perception, such as Figure 3 As shown, specifically, in the embodiment of the present invention, the coarse-grained cross-attention autoencoder and the fine-grained cross-attention autoencoder can adopt a similar network structure. Taking the coarse-grained cross-attention autoencoder as an example, in the embodiment of the present invention, a six-layer cross-attention stacked coarse-grained sentiment polarity encoder is designed, which takes non-language modal features as input, wherein the first three layers use other non-language modal features as input to the CA layer, and the last three layers use text modal features as input to the CA layer to obtain the final output: the sentiment polarity representation corresponding to the non-text modal features. For the coarse-grained non-sentiment polarity encoder, a self-attention mechanism encoder is adopted, which takes non-language modal features as input and outputs the non-sentiment polarity representation corresponding to the non-text modal features. For the coarse-grained sentiment polarity decoder, a nine-layer cross-attention stacked coarse-grained sentiment polarity decoder is designed, which takes the sentiment polarity representation corresponding to the non-text modal features as input. The first three layers use the non-sentiment polarity representation as the input of the CA layer, the middle three layers use the text modal features to input into the CA layer, and the last three layers use the non-text modal features to input into the CA layer to obtain the final output: the reconstructed coarse-grained non-text modal features.
[0040] For the fine-grained cross-attention autoencoder, compared with the coarse-grained cross-attention autoencoder, the bottom layers of the fine-grained emotion encoder, fine-grained non-emotion encoder and fine-grained emotion decoder respectively have three more CA layers using emotion polarity corresponding to non-text modal features.
[0041] In the embodiment of the present invention, after obtaining the data to be analyzed, multimodal feature extraction can be performed on the data to be analyzed to obtain the original non-linguistic features. and text modality features These features are input into the representation decoupling module of interactive emotion perception to obtain the corresponding emotion polarity representation and non-emotional polarity representation : (1) in is the original feature of the non-text modal feature to be decoupled, are other non-text modality features, is a coarse-grained sentiment polarity encoder, is a coarse-grained non-sentiment polarity encoder.
[0042] In the embodiment of the present invention, the sentiment polarity corresponding to the non-text modal feature is represented as Together with the multimodal features, they are fed into a Transformer-based fine-grained cross-attention autoencoder. The sentiment polarity representation is used to extract the modality through CA guidance. The fine-grained sentiment representation in . The sentiment representation can be obtained by and non-emotional expressions : (2) in is a fine-grained sentiment encoder, is a fine-grained non-sentiment encoder.
[0043] In an embodiment of the present invention, considering that some features of non-text modal features are related to sentiment polarity (or emotion) and some features are irrelevant to sentiment polarity (or emotion), it is proposed to extract features related to sentiment polarity (or emotion) and all semantic features irrelevant to sentiment polarity (or emotion) based on a coarse-grained sentiment polarity encoder and a coarse-grained non-sentiment polarity encoder (or a fine-grained sentiment encoder and a fine-grained non-sentiment encoder).
[0044] For example, for a human face, features related to the corners of the mouth or the eyes may be more relevant to emotions, but features related to the nose or the head contour are irrelevant to emotions.
[0045] In an embodiment of the present invention, in order to minimize the loss of feature information related to the extracted emotion polarity (or emotion) during the encoding process and ensure the effectiveness of the encoding, a decoder is used to reconstruct the original features, and the reconstruction loss between the original features and the reconstructed features is used to minimize the information loss during the encoding process and ensure the effectiveness of the encoding.
[0046] Specific, coarse-grained non-textual modality features and fine-grained non-text modality features Respectively expressed as: , (3) in, is a coarse-grained sentiment polarity decoder, is a fine-grained emotion decoder.
[0047] Specifically, the reconstruction loss of the coarse-grained cross-attention autoencoder and the fine-grained cross-attention autoencoder is: (4) in, represents a batch of samples, represents the reconstruction loss of the coarse-grained criss-cross attention autoencoder, represents the reconstruction loss of the fine-grained criss-cross attention autoencoder; Indicates i Non-text modality features of samples , Represents the reconstructed coarse-grained non-text modality features of the i-th sample , Indicates i Reconstructed fine-grained non-text modality features of samples .
[0048] In the embodiment of the present invention, for a batch of samples, the reconstruction loss between the reconstructed features and the original features may be calculated based on each sample.
[0049] In the embodiment of the present invention, in order to enable the encoder to accurately extract emotional information, it is proposed that the emotion-related representations of the same emotion should be more similar than the representations of different emotions, and the emotion-independent representations from the same context and characters (i.e., different sentence samples from the same video) should be more similar than the representations from different contexts and characters. Therefore, in the embodiment of the present invention, a new contrast loss is also proposed, as shown below: (5) , (6) in, is the contrastive loss of the coarse-grained criss-cross attention autoencoder, is the contrastive loss of the fine-grained criss-cross attention autoencoder, yes All positive samples in Or all negative samples ; include or , and Differentiate by emotional polarity, or , and Differentiate by emotion category. or , and Distinguish by non-text modal features; X is For the corresponding , Indicates the first i The samples obtained , express Except for i The samples other than the samples . In, when When it is a positive sample, calculate The similarity with all other positive samples is When it is a negative sample, calculate And the similarity with all other negative samples.
[0050] In the embodiment of the present invention, for the coarse-grained emotion polarity encoder and the coarse-grained non-emotion polarity encoder in the coarse-grained cross-attention autoencoder, and for the fine-grained emotion encoder and the fine-grained non-emotion encoder in the fine-grained cross-attention autoencoder, the contrast loss needs to be calculated. Formula (6) is used to calculate the contrast loss of any encoder (coarse-grained emotion polarity encoder, coarse-grained non-emotion polarity encoder, fine-grained emotion encoder or fine-grained non-emotion encoder). For example, for the coarse-grained emotion polarity encoder, X in formula (6) is , for ,but = , based on formula (6) Represents the sample in sample set D i For positive samples, With all other positive samples What you get similarity, or i For negative samples With all other negative samples What you get The similarity of Representation samplei With all other positive samples What you get The similarity with all other negative samples What you get The sum of similarities.
[0051] Specifically, for the coarse-grained sentiment polarity encoder, the positive and negative samples are divided according to the sentiment polarity of the data corresponding to the multimodal features, and based on the contrast loss, the sentiment polarity representations of the same sentiment polarity are more similar than the sentiment polarity representations of different sentiment polarities. For the coarse-grained non-sentiment polarity encoder, the positive and negative samples are divided according to the sentiment polarity of the current non-text modal representation, and based on the contrast loss, the non-sentiment polarity representations of the same sentiment polarity are more similar than the non-sentiment polarity representations of different sentiment polarities. For the fine-grained sentiment encoder, the positive and negative samples are divided according to the sentiment polarity of the data corresponding to the multimodal features, and based on the contrast loss, the sentiment representations of the same sentiment are more similar than the sentiment representations of different sentiments. For the fine-grained non-sentiment encoder, the positive and negative samples are divided according to the sentiment of the current non-text modal representation, and based on the contrast loss, the non-sentiment representations of the same sentiment are more similar than the non-sentiment representations of different sentiments.
[0052] In the embodiment of the present invention, considering that effective sentiment polarity representation (or sentiment representation) and non-sentiment polarity representation (or non-sentiment representation) are mutually exclusive, in order to completely disentangle these representations, the embodiment of the present invention proposes that each decoupled representation should be as orthogonal as possible to reduce the information redundancy of each representation. Therefore, in the embodiment of the present invention, during the training process of the coarse-grained cross-attention autoencoder and the fine-grained cross-attention autoencoder, it is also necessary to calculate the orthogonal loss.
[0053] Specifically, the orthogonal loss of the coarse-grained cross-attention autoencoder and the fine-grained cross-attention autoencoder is: (7) in, is the orthogonal loss of the coarse-grained criss-cross attention autoencoder, is the orthogonal loss of the fine-grained criss-cross attention autoencoder, They are i The sentiment polarity representation and non-sentiment polarity representation corresponding to the non-text modal features obtained by samples, They are i The sentiment and non-sentiment representations corresponding to the non-text modal features obtained from samples.
[0054] In an embodiment of the present invention, the attention cross-modal representation offset decoupling module includes a projection module and an offset module. The projection module projects the emotion representations corresponding to the multiple non-text modal features into the text representation space. The offset module controls the offset degree of the emotion offset text vector based on the offset scaling factors corresponding to the multiple non-text modal features. The offset scaling factor is: (8) in, Indicates m The offset scaling factor corresponding to the non-text modality features, , represents the sentiment representation corresponding to the projected non-text modality features, is a hyperparameter, It is a text modality feature.
[0055] In the embodiment of the present invention, the representation spaces of different modalities are heterogeneous. In the embodiment of the present invention, it is hoped that the sentiment representation corresponding to the non-text modal features can be in the same representation space as the text modal features, so that the pre-trained language model can be easily fine-tuned for sentiment analysis. Based on this, the embodiment of the present invention proposes a projection module, which is composed of multiple linear layers. The projection module maps the sentiment representation corresponding to the decoupled non-text modal features to the text representation space so that the pre-trained language model can easily understand it.
[0056] In the embodiment of the present invention, the sentiment representation corresponding to the projected non-text modal feature contains the sentiment information of other modalities and can be directly used to offset the text modal feature. However, in order to avoid excessive offset and destroy the original text semantics, the embodiment of the present invention proposes to control the offset degree of the sentiment offset text vector based on the offset scaling factor.
[0057] In an embodiment of the present invention, each non-text modality may correspond to a different offset scaling factor, and finally each non-text modality may be used to offset the text modality feature based on the offset scaling factor, and then the final sentiment offset text vector may be calculated.
[0058] For example, for multimodal features including text modality features, video modality features, and speech modality features, the final sentiment shift text vector is generated as follows: (9) in, represents the emotion representation corresponding to the projected video modality features, represents the emotion representation corresponding to the projected speech modality feature, Indicates the offset scaling factor corresponding to the video mode, Indicates the offset scaling factor corresponding to the speech mode.
[0059] In the embodiment of the present invention, by By inputting the pre-trained language model (PLM), we can get the prediction results: (10) According to the multimodal sentiment calculation task, you can choose cross entropy loss or mean square error loss to train the pre-trained language model as shown below: or (11) It is a pre-trained language model; is the number of emotion types, is a binary indicator function, It is i The true value of the sentiment of samples, It is i The sentiment prediction value of samples.
[0060] In an embodiment of the present invention, a training strategy of pre-training (freezing) and then fine-tuning is used. Specifically, a pre-trained language model is first obtained, which is trained based on text modal features and can understand the semantic information contained in the text modal features. The parameters of the pre-trained language model are frozen, and the emotion perception representation decoupling and offset system is trained using a multimodal analysis dataset. Then, the projection layer and the pre-trained language model are fine-tuned using the emotion classification dataset. Both training iterations use loss, but using different balance coefficients .
[0061] In the embodiment of the present invention, the multimodal analysis dataset can be any public dataset containing multimodal features and can be used for multimodal analysis. The sentiment classification dataset can be any public dataset containing multimodal features and carrying sentiment classification labels, such as CMU-MOSI, CMU-MOSEI and CH-SIMS.
[0062] Specifically, the emotion perception representation decoupling and offset system is trained based on the following steps: Get the pre-trained language model and freeze the parameters of the pre-trained language model; Using the multimodal analysis dataset, the first equilibrium coefficient was used Training the emotion perception representation decoupling and offset system, and updating various parameters of the interactive emotion perception representation decoupling module and the attention cross-modal representation offset decoupling module; Using the sentiment classification dataset, we use the second balance coefficient Update the parameters of the projection module and pre-trained language model; ; Indicates the first balance coefficient or the second balance coefficient.
[0063] Based on the same inventive concept, an embodiment of the present invention further provides a multimodal sentiment analysis device based on sentiment perception representation decoupling and offset, which is applied to a sentiment perception representation decoupling and offset system, and the device includes: An acquisition module, used for acquiring multimodal features, wherein the multimodal features include: a text modal feature and a plurality of non-text modal features; A representation module is used for performing sentiment polarity analysis using each non-text modal feature as input and guided by text modal features and other non-text modal features through an interactive sentiment perception representation decoupling module to obtain a sentiment polarity representation corresponding to each non-text modal feature; a representation module is used for performing sentiment analysis using each non-text modal feature as input and guided by text modal features, other non-text modal features and the sentiment polarity representation corresponding to each non-text modal feature to obtain a sentiment representation corresponding to each non-text modal feature; The sentiment offset text vector determination module is used to project the sentiment representations corresponding to multiple non-text modal features into the text representation space through the attention cross-modal representation offset decoupling module, and obtain the sentiment offset text vector according to the offset scaling factors corresponding to each of the multiple non-text modal features, the sentiment representations corresponding to the projected non-text modal features, and the text modal features; The classification module is used to input the sentiment-shifted text vector into the fine-tuned pre-trained language model to obtain the sentiment classification result.
[0064] Based on the same inventive concept, an embodiment of the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes, the steps in the multimodal sentiment analysis method based on decoupling and offset of sentiment perception representation as described in any of the above embodiments are implemented.
[0065] Based on the same inventive concept, an embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the multimodal sentiment analysis method based on emotion perception representation decoupling and offset described in any of the above embodiments are implemented.
[0066] Based on the same inventive concept, an embodiment of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the multimodal sentiment analysis method based on emotion perception representation decoupling and offset described in any of the above embodiments.
[0067] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0068] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0069] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (apparatus), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0070] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0071] These computer program instructions can also be loaded onto a computer or other programmable terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0072] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.
[0073] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.
[0074] The above is a detailed introduction to the multimodal sentiment analysis method based on decoupling and offset of emotion perception representation provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A multimodal sentiment analysis method based on decoupling and offset of sentiment perception representation, characterized in that: Applied to the emotion perception representation decoupling and offset system, the method comprises: Acquire a multimodal feature, wherein the multimodal feature includes: a text modal feature and a plurality of non-text modal features; Through the representation decoupling module of interactive emotion perception, each non-text modal feature is taken as input, and the text modal features and other non-text modal features are used as guidance to perform sentiment polarity analysis, and obtain the sentiment polarity representation corresponding to each non-text modal feature. Each non-text modal feature is taken as input, and the text modal features, other non-text modal features, and the sentiment polarity representation corresponding to each non-text modal feature are used as guidance to perform sentiment analysis, and obtain the sentiment representation corresponding to each non-text modal feature. Through the attention cross-modal representation offset decoupling module, the sentiment representations corresponding to multiple non-text modal features are projected into the text representation space, and the sentiment offset text vector is obtained according to the offset scaling factors corresponding to each of the multiple non-text modal features, the sentiment representations corresponding to the projected non-text modal features, and the text modal features. The sentiment offset text vector is input into the fine-tuned pre-trained language model to obtain the sentiment classification result.
2. The multimodal sentiment analysis method based on emotion perception representation decoupling and offset according to claim 1 is characterized in that: The interactive emotion perception representation decoupling module includes: a coarse-grained cross-attention autoencoder and a fine-grained cross-attention autoencoder, the coarse-grained cross-attention autoencoder is used for emotion polarity analysis, and the fine-grained cross-attention autoencoder is used for emotion analysis; The coarse-grained cross-attention autoencoder includes: a coarse-grained emotion polarity encoder, a coarse-grained non-emotion polarity encoder, and a coarse-grained emotion polarity decoder. The coarse-grained sentiment polarity encoder takes each non-text modality feature, text modality feature and other non-text modality features as input, and outputs the sentiment polarity representation corresponding to each non-text modality feature; The coarse-grained non-emotional polarity encoder takes each non-text modal feature as input and outputs the non-emotional polarity representation corresponding to each non-text modal feature; The coarse-grained sentiment polarity decoder takes the output of the coarse-grained sentiment polarity encoder, the output of the coarse-grained non-sentiment polarity encoder, text modality features and other non-text modality features as input, and outputs the reconstructed coarse-grained non-text modality features.
3. The multimodal sentiment analysis method based on emotion perception representation decoupling and offset according to claim 2 is characterized in that: The fine-grained cross-attention autoencoder includes: a fine-grained emotion encoder, a fine-grained non-emotion encoder and a fine-grained emotion decoder. The fine-grained sentiment encoder takes each non-text modality feature, text modality feature, other non-text modality features and the sentiment polarity representation corresponding to each non-text modality feature as input, and takes the sentiment representation corresponding to each non-text modality feature as output; The fine-grained non-emotional encoder takes each non-text modal feature and the sentiment polarity representation corresponding to each non-text modal feature as input, and takes the non-emotional representation corresponding to each non-text modal feature as output; The fine-grained sentiment decoder takes the output of the fine-grained sentiment encoder, the output of the fine-grained non-sentiment encoder, text modality features and other non-text modality features as input, and outputs the reconstructed fine-grained non-text modality features.
4. The multimodal sentiment analysis method based on emotion perception representation decoupling and offset according to claim 3 is characterized in that: The reconstruction loss of the coarse-grained criss-cross attention autoencoder and the fine-grained criss-cross attention autoencoder is: ; in, represents a batch of samples, represents the reconstruction loss of the coarse-grained criss-cross attention autoencoder, represents the reconstruction loss of the fine-grained criss-cross attention autoencoder; Indicates i Non-text modality features of samples , Indicates i Reconstructed coarse-grained non-text modality features of samples , Indicates i Reconstructed fine-grained non-text modality features of samples ; , ; in, is a coarse-grained sentiment polarity decoder, is a fine-grained emotion decoder; ; ; is the sentiment polarity representation corresponding to each non-text modality feature, is the non-emotional polarity representation corresponding to each non-text modality feature; is the sentiment representation corresponding to each non-text modality feature is the non-emotional representation corresponding to each non-text modality feature; is a non-text modality feature, are other non-text modality features, is a coarse-grained sentiment polarity encoder, is a coarse-grained non-sentiment polarity encoder, is a fine-grained sentiment encoder, It is a fine-grained non-emotional semantic encoder; The contrast loss of the coarse-grained criss-cross attention autoencoder and the fine-grained criss-cross attention autoencoder is: ; , ; in, is the contrastive loss of the coarse-grained criss-cross attention autoencoder, is the contrastive loss of the fine-grained criss-cross attention autoencoder, yes All positive samples in Or all negative samples ; include or , and Differentiate by emotional polarity, or , and Differentiate by emotion category. or , and Distinguish by non-text modal features; X , Indicates the first i The samples obtained , express Except for i The samples other than the samples ; The orthogonal loss of the coarse-grained criss-cross attention autoencoder and the fine-grained criss-cross attention autoencoder is: ; in, is the orthogonal loss of the coarse-grained criss-cross attention autoencoder, is the orthogonal loss of the fine-grained criss-cross attention autoencoder, They are i The sentiment polarity representation and non-sentiment polarity representation corresponding to the non-text modal features obtained by samples, They are i The sentiment and non-sentiment representations corresponding to the non-text modal features obtained from samples.
5. The multimodal sentiment analysis method based on emotion perception representation decoupling and offset according to claim 4 is characterized in that: The attention cross-modal representation offset decoupling module includes a projection module and an offset module. The projection module projects the emotion representations corresponding to multiple non-text modal features into the text representation space. The offset module controls the offset degree of the emotion offset text vector based on the offset scaling factors corresponding to each of the multiple non-text modal features. The offset scaling factor is: ; in, Indicates m The offset scaling factor corresponding to the non-text modality features, , represents the sentiment representation corresponding to the projected non-text modality features, is a hyperparameter, It is a text modality feature.
6. The multimodal sentiment analysis method based on emotion perception representation decoupling and offset according to claim 5 is characterized in that: The loss function of the pre-trained language model is: or ; ; It is i The true value of the sentiment of samples, It is i The sentiment prediction value of samples, It is a pre-trained language model; is the number of emotion types, is a binary indicator function, Indicates that based on i The sentiment shifted text vector obtained by samples.
7. The multimodal sentiment analysis method based on emotion perception representation decoupling and offset according to claim 6 is characterized in that: The emotion perception representation decoupling and offset system is trained based on the following steps: Get the pre-trained language model and freeze the parameters of the pre-trained language model; Using the multimodal analysis dataset, the first equilibrium coefficient was used Training the emotion perception representation decoupling and offset system, and updating various parameters of the interactive emotion perception representation decoupling module and the attention cross-modal representation offset decoupling module; Using the sentiment classification dataset, we use the second balance coefficient Update the parameters of the projection module and pre-trained language model; ; Indicates the first balance coefficient or the second balance coefficient.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the multimodal sentiment analysis method based on emotion perception representation decoupling and offset as described in any one of claims 1-7 is implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the multimodal sentiment analysis method based on the decoupling and offset of sentiment perception representation as described in any one of claims 1 to 7.
10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps in the multimodal sentiment analysis method based on decoupling and offset of emotion perception representation as described in any one of claims 1-7 are implemented.
Citation Information
Patent Citations
Facial expression action unit adversarial synthesis method based on local attention model
CN110458003A
Adaptive fusion multi-modal sentiment classification method based on attention mechanism
CN113435496A
KR20230128812A
