Multi-modal sentiment analysis method and system based on sentiment polarity and sentiment intensity

Abstract

The invention discloses a multi-modal sentiment analysis method and system based on sentiment polarity and sentiment intensity, and the method comprises the steps: collecting initial video data, and extracting initial modal features; extracting high-level semantic features in the initial modal features; extracting interaction information in high-level semantic feature modals and between the modals; extracting sentiment polarity and sentiment intensity vectors in the interaction information; and completing character sentiment analysis in the video data based on the sentiment polarity and the sentiment intensity vector. According to the method, a Transform encoder and an MLP encoder are adopted to extract high-level semantic features, and interaction information in modals and interaction information between the modals are extracted through a multi-granularity cross-modal interaction module. By means of the design, the model can deeply mine local features and detail information between different modes, and understanding and analysis of emotion expression are enhanced. The method also completes the analysis of the character emotion in the video data through a cross-modal attention mechanism and a feedforward neural network.

Description

Technical Field

[0001] The present invention relates to the technical fields of natural language processing, computer vision and speech processing, and in particular to a multimodal sentiment analysis method and system based on sentiment polarity and sentiment intensity. Background Art

[0002] Multimodal sentiment analysis integrates data from multiple modalities, such as text, images, and speech, to accurately identify human emotions. This method provides a more detailed and comprehensive way to understand emotions by integrating the unique information of each modality and digging deep into emotional expressions from different perspectives. With the rise of social media, the generation of multimodal data is increasing, spanning different fields and languages, providing rich materials for sentiment analysis. Compared with traditional sentiment analysis that relies only on text, multimodal sentiment analysis can utilize emotional signals in images and speech, thanks to the latest advances in computer vision and speech recognition technology. This analysis not only broadens the source of emotional data, but also improves the accuracy of emotion recognition. Despite this, the field of multimodal sentiment analysis still faces many challenges, including how to effectively integrate data from different modalities, deal with possible semantic differences between modalities, and deal with the uneven distribution of different emotion categories in the data. These challenges require researchers to conduct more in-depth explorations in model design and algorithm innovation.

[0003] In the study of multimodal data analysis, although fusion methods based on attention mechanisms have been widely adopted, these methods do not deal well with the imbalance problem of different data modalities in expressing emotions. For example, text information can often clearly reflect the positive and negative tendencies and intensity of emotions; although audio signals can convey the intensity of emotions well, they are not accurate enough in distinguishing the specific types of emotions; and visual information, such as facial expressions, although relatively effective in identifying the positive and negative emotions, is slightly insufficient in capturing the intensity of emotions. This imbalance in emotional expression between different modalities, if not properly handled when fusing multimodal data, may cause the features of different modalities to be confused with each other, thereby affecting the accuracy of sentiment analysis. Summary of the invention

[0004] Faced with the fact that single-text sentiment analysis cannot cope with complex sentiment mechanisms and today's abundant multimodal data, the present invention designs a sentiment analysis model that receives data from multiple modalities, so as to comprehensively utilize the multi-faceted descriptions of the data to infer the emotions of the characters in the video clips. In addition, in order to solve the problem of different discriminative capabilities of each modality in multimodal interaction, firstly, coarse-grained interactions are performed at a relatively macro level to capture the global relationship between different modalities, and then fine-grained interactions are further performed on this basis to deeply explore the local features and detailed information between modalities. The features obtained by the multi-granular cross-modal interaction module are then used to generate a robust and highly generalizable sentiment polarity feature and sentiment intensity feature representation that integrates the rich information of multimodal data, so that the model can more effectively understand and utilize the complementary information of polarity and intensity of multimodal emotions when processing multimodal tasks, thereby improving the performance and robustness of the model.

[0005] To achieve the above object, the present invention provides a multimodal sentiment analysis method based on sentiment polarity and sentiment intensity, the steps comprising:

[0006] Collect initial video data and extract initial modal features;

[0007] Extracting high-level semantic features from the initial modal features;

[0008] Extracting interactive information within and between modalities of the high-level semantic features;

[0009] Extracting sentiment polarity and sentiment intensity vectors from the interaction information;

[0010] Based on the emotion polarity and the emotion intensity vector, character emotion analysis in the video data is completed.

[0011] Preferably, the extracted initial modal features include acoustic features and visual features;

[0012] The acoustic features include: Mel frequency cepstral coefficients, root mean square energy and spectrum centroid, fundamental frequency and frame energy; firstly, the audio content is converted into text using speech recognition technology; then, the pre-trained BERT model is used to extract text features of the transcribed text; and the acoustic features are extracted from the audio using an audio signal feature processor;

[0013] The visual features include: expression recognition, facial feature point detection, scene recognition, body posture estimation, and facial action tracking; the video is deeply processed using the computer vision library OpenFace to extract the visual features.

[0014] Preferably, the method for extracting the high-level semantic features comprises: extracting the high-level semantic features from each modality data using three Transformer encoders and two MLP encoders respectively; dividing the three Transformer encoders and the two MLP encoders into two stages, the first stage consisting of a Transformer encoding layer, and the second stage consisting of an MLP encoder, and the overall calculation process comprises:

[0015] b m =TransEncoder m (h m )

[0016] c n =MLPEncoder m (b n )

[0017] In the formula, TransEncoder m and MLPEncoder m are the Transformer encoder and MLP encoder of the first and second stages respectively; m∈{v,a,t], n∈{v,a}, where v represents the initial visual modality, a represents the acoustic modality, and t represents the text modality; h m b represents the feature representation of the modality; m represents the feature representation of modality m obtained by the first stage encoder; b n represents the feature representation of modality n obtained by the first stage encoder, c n Represents the feature representation of mode n obtained after the second stage.

[0018] Preferably, the method of extracting the interactive information within the high-level semantic feature modality and between modalities includes:

[0019]

[0020]

[0021]

[0022]

[0023] Among them, Cross-A represents the cross-modal attention mechanism; softmax is the normalized exponential function; Xm and Xn represent the modal features of the input; Represents the query weight matrix of the input modality Xn; Indicates input mode X m The key weight matrix of Indicates input mode X mThe value weight matrix; Conv represents a one-dimensional convolution operation; Z m and Z n Represents the feature representation of modalities m and n after the first Transformer encoder; Z nm Indicates that Z n and Z m Perform splicing operations; represents the coarse-grained interaction representation of modalities m and n, m∈{v,a,t}, n∈{v,a,t}; LN represents layer normalization; K represents the fine-grained interaction between modalities m and n; mn It represents the multi-granularity feature representation of modality m and modality n; concat represents the feature concatenation operation; FFN represents the feedforward neural network.

[0024] Preferably, the method for extracting the emotion polarity and the emotion intensity vector includes:

[0025]

[0026]

[0027]

[0028]

[0029]

[0030] loss p =ω1(F p + -y + ) 2 +ω2(F p - -y - ) 2

[0031]

[0032] Among them, K tm Multi-granular features representing text modality and n-modality; c n represents the features of modality n obtained by the second-stage MLP encoder; Accurate representation of multi-granular features representing textual modality and n-modality, Represents the precise representation of the features of modality n, m∈{v,a}, n∈{v,a}; FC represents the fully connected layer; softmax represents the normalized exponential function; Hadamard represents element-wise multiplication; sigmoid represents the activation function; Conv represents one-dimensional convolution; pm represents the obtained feature ratio value; Accurate representation of multi-granular features representing textual modality and v-modality; represents the exact representation of the characteristics of mode a; F p Represents the emotional polarity obtained, Accurate representation of multi-granular features representing text modality and a-modality; represents an accurate representation of the characteristics of the v mode; F i Represents the intensity of the emotion obtained; F p + The average polar vector of positive sentiment that represents the test results; F p - Represents the average polar vector of negative sentiment of the predicted results; y + Represents the positive sentiment mean pole vector of the true label; y - represents the average polar vector of negative sentiment of the true label; ω1 represents the weight of positive sentiment; ω2 represents the weight of negative sentiment; |y| represents the absolute value of the true label; ||F i ||1 represents the l1 norm of the predicted sentiment intensity vector; Cov represents the covariance calculation formula; exp represents the exponential function; Var represents the variance calculation formula; loss p Represents the emotional polarity loss we get; loss i Represents the loss of emotional intensity obtained.

[0033] Preferably, based on the emotion polarity and the emotion intensity vector, character emotion analysis in the video data is performed, the steps comprising:

[0034] F pi =concat(F p ,F i )

[0035] F=LN(Cross-A(F p , F pi )+F pi )+LN(Cross-A(F i , F pi )+F pi )

[0036]

[0037]

[0038] Among them, concat represents the feature concatenation operation, F p represents the sentiment polarity vector; F i represents the sentiment intensity vector; F pirepresents the obtained sentiment vector; Cross-A represents the cross-modal attention machine; LN represents layer normalization, F represents the mixed vector; FFN represents the feedforward neural network, represents the predicted value of the i-th sample; yi represents the true value of the i-th sample; loss pre represents the loss function of the predicted value; N represents the number of samples. The final loss function can be expressed as:

[0039]

[0040] In the formula, α, β, and ε represent hyperparameters, which determine the contribution of each loss term to the total loss; L train Represents the overall loss function.

[0041] The present invention also provides a multimodal sentiment analysis system based on sentiment polarity and sentiment intensity, the system is used to implement the above method, including: a preprocessing module, an extraction module, a multi-granularity cross-modal interaction module, a dynamic feature integration module and a sentiment prediction module;

[0042] The preprocessing module is used to collect initial video data and extract initial modal features;

[0043] The extraction module is used to extract high-level semantic features from the initial modal features;

[0044] The multi-granularity cross-modal interaction module is used to extract the interaction information within the high-level semantic feature modality and between modalities;

[0045] The dynamic feature integration module is used to extract the emotion polarity and emotion intensity vectors in the interaction information;

[0046] The emotion prediction module is used to complete the character emotion analysis in the video data based on the emotion polarity and the emotion intensity vector.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] The present invention uses Transformer encoder and MLP encoder to extract high-level semantic features, and extracts intra-modal and inter-modal interaction information through a multi-granularity cross-modal interaction module. This design enables the model to deeply explore local features and detailed information between different modalities, enhancing the understanding and analysis of emotional expression. The present invention also completes the analysis of character emotions in video data through a cross-modal attention mechanism and a feedforward neural network. This method not only improves the accuracy of sentiment analysis, but also optimizes the training process of the model by designing a suitable loss function. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0050] Figure 1 A schematic diagram of a method flow chart of an embodiment of the present invention;

[0051] Figure 2 A schematic diagram of a data preprocessing process according to an embodiment of the present invention;

[0052] Figure 3 A schematic diagram of a process for extracting interactive information according to an embodiment of the present invention;

[0053] Figure 4 A schematic diagram of a process for extracting sentiment polarity and sentiment intensity vectors according to an embodiment of the present invention;

[0054] Figure 5 A schematic diagram of the fusion process of an embodiment of the present invention;

[0055] Figure 6 Schematic diagram of the overall framework of the method of an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0058] Embodiment 1

[0059] As Figure 1 FIG. 1 is a schematic diagram of the method flow of this embodiment, and the steps include:

[0060] S1. Collect initial video data and extract initial modal features.

[0061] like Figure 2As shown, in the video data collection and preprocessing stage, the audio content is first converted into text using Transkriptor's advanced speech recognition technology, and then the pre-trained BERT model is used to extract text features of the transcribed text. Next, the audio signal feature processor Librosa is used to extract the acoustic features of the person's speech from the audio, including Mel-frequency cepstral coefficients (MFCC), root mean square (RMS) energy and spectrum centroid, fundamental frequency, and frame energy. In addition, this embodiment uses the computer vision library OpenFace to perform in-depth processing on the video and extract a series of key visual features, including expression recognition, facial feature point detection, scene recognition, body posture estimation, and facial action tracking. These preprocessing steps provide high-quality initial modal features for subsequent multimodal sentiment analysis.

[0062] S2. Extract high-level semantic features from initial modality features.

[0063] Three Transformer encoders and two MLP encoders are used to extract corresponding high-level semantic features from each modality data. In this process, the present embodiment divides the three Transformer encoders and the two MLP encoders into two stages, the first stage consists of the Transformer encoding layer, and the second stage consists of the MLP encoder (the MLP consists of two linear layers and a ReLU activation function).

[0064] The overall calculation process of step S2 is as follows:

[0065] b m =TransEncoder m (h m )

[0066] C n =MLPEncoder m (b n )

[0067] In the formula, TransEncoder m and MLPEncoder m are the Transformer encoder and MLP encoder of the first and second stages respectively; m∈{v,a,t}, n∈{v,a}, where v represents the initial visual modality, a represents the acoustic modality, and t represents the text modality; h m b represents the feature representation of the modality; m represents the feature representation of modality m obtained by the first stage encoder; b n represents the feature representation of modality n obtained by the first stage encoder, c n Represents the feature representation of mode n obtained after the second stage.

[0068] S3. Extract high-level semantic features from intra-modal and inter-modal interactive information.

[0069] like Figure 3 As shown, this step aims to extract the interaction information within and between modalities. This goal is achieved by sequentially performing coarse-grained and fine-grained cross-modal interactions. Specifically, the coarse-grained interaction is first processed by a one-dimensional convolution layer (the convolution kernel size is 3x3) to ensure that the features of all modalities are mapped to a unified dimensional space. Next, the cross-modal attention mechanism Cross-A is used to implement cross-modal attention, and then the representation features of the coarse-grained interaction are obtained through residual connections and layer normalization. In the fine-grained interaction stage, this embodiment combines the representation features obtained in the coarse-grained stage with the original features of each modality, and applies the cross-modal attention mechanism again. After that, the fine-grained feature representation is obtained through residual connections and normalization. The feature representation is merged by vector addition operations, and then through a feedforward neural network layer, residual connections and normalization, the feature representation is spliced ​​to form a comprehensive multi-granularity feature representation, thereby providing rich information for subsequent analysis and processing.

[0070] The overall calculation process of step S3 is as follows:

[0071]

[0072]

[0073]

[0074]

[0075] Among them, Cross-A represents the cross-modal attention mechanism; softmax is the normalized exponential function; X m and X n Represents the modal characteristics of the input; Indicates input mode X n The query weight matrix of Indicates input mode X m The key weight matrix of Indicates input mode X m The value weight matrix; Conv represents a one-dimensional convolution operation; Z m and Z n Represents the feature representation of modalities m and n after the first Transformer encoder; Z nm Indicates that Z n and Z m Perform splicing operations; represents the coarse-grained interaction representation of modalities m and n, m∈{v,a,t}, n∈{v,a,t}; LN represents layer normalization; K represents the fine-grained interaction between modalities m and n; mn It represents the multi-granularity feature representation of modality m and modality n; concat represents the feature concatenation operation; FFN represents the feedforward neural network.

[0076] S4. Extract sentiment polarity and sentiment intensity vectors from interaction information.

[0077] Considering that text-acoustic features are more significant in capturing emotional intensity, while text-visual features are more important in judging emotional polarity. In order to take advantage of both, we combined the multi-granularity feature K ta and the visual modality feature c obtained by the second stage MLP encoder v , to extract multimodal sentiment intensity features, the steps are as follows Figure 4 shown.

[0078] Similarly, this embodiment also combines the multi-granularity feature K of text-vision ta and the acoustic modal features c obtained by the second-stage MLP encoder v , to extract multimodal sentiment polarity features; the two input features are processed through a fully connected layer, softmax activation and residual connection to obtain a preliminary feature representation. Finally, through the Hadamard product and the enhancement of another fully connected layer, a more accurate multimodal sentiment feature representation is obtained. This embodiment performs vector addition on this representation and then passes it through a one-dimensional convolution and a sigmoid activation function to obtain the feature ratio value p. At this point, the two accurate multimodal feature representations can be combined to obtain the sentiment polarity and sentiment intensity representations. In particular, this embodiment uses square error to design a loss function for sentiment polarity, and uses correlation coefficient distance to design a loss function for sentiment intensity.

[0079] The overall calculation process of step S4 is as follows:

[0080]

[0081]

[0082]

[0083]

[0084]

[0085] loss p =ω1(F p + -y+ ) 2 +ω2(F p - -y - ) 2

[0086]

[0087] Among them, K tm Multi-granular features representing text modality and n-modality; c n represents the features of modality n obtained by the second-stage MLP encoder; Accurate representation of multi-granular features representing textual modality and n-modality, Represents the precise representation of the features of modality n, m∈{v,a}, n∈{v,a}; FC represents the fully connected layer; softmax represents the normalized exponential function; Hadamard represents element-wise multiplication; sigmoid represents the activation function; Conv represents one-dimensional convolution; pm represents the obtained feature ratio value; Accurate representation of multi-granular features representing textual modality and v-modality; represents the exact representation of the characteristics of mode a; F p Represents the emotional polarity obtained, Accurate representation of multi-granular features representing text modality and a-modality; represents an accurate representation of the characteristics of the v mode; F i Represents the intensity of the emotion obtained; F p + The average polar vector of positive sentiment that represents the test results; F p - Represents the average polar vector of negative sentiment of the predicted results; y + Represents the positive sentiment mean pole vector of the true label; y - represents the average polar vector of negative sentiment of the true label; ω1 represents the weight of positive sentiment; ω2 represents the weight of negative sentiment; |y| represents the absolute value of the true label; ||F i ||1 represents the l1 norm of the predicted sentiment intensity vector; Cov represents the covariance calculation formula; exp represents the exponential function; Var represents the variance calculation formula; loss p Represents the emotional polarity loss we get; loss i Represents the loss of emotional intensity obtained.

[0088] S5. Based on the emotion polarity and emotion intensity vectors, complete the character emotion analysis in the video data.

[0089] like Figure 5 As shown, the sentiment intensity vector F obtained in the above steps is iand sentiment polarity vector F p Concatenate to get the sentiment vector representation F pi ; Then the obtained emotion representation is cross-modal attention interaction with the emotion polarity vector and the emotion intensity vector (same as Cross-A in the above step), followed by residual connection and normalization operations to obtain two interaction vectors, and then the two interaction vectors are vector added to obtain a mixed vector F, and finally the prediction value is obtained by using a fully connected layer after a feedforward neural network, residual connection, and normalization, and finally the emotion loss is calculated using the mean square error.

[0090] The overall calculation process of step S5 is as follows:

[0091] F pi =concat(F p , F i )

[0092] F=LN(Cross-A(F p , F pi )+F pi )+LN(Cross-A(F i , F pi )+F pi )

[0093]

[0094]

[0095] Among them, concat represents the feature concatenation operation, F p represents the sentiment polarity vector; F i represents the sentiment intensity vector; F pi represents the obtained sentiment vector; Cross-A represents the cross-modal attention machine; LN represents layer normalization, F represents the mixed vector; FFN represents the feedforward neural network, represents the predicted value of the i-th sample; yi represents the true value of the i-th sample; loss pre represents the loss function of the predicted value; N represents the number of samples. The final loss function can be expressed as:

[0096]

[0097] In the formula, α, β, and ε represent hyperparameters, which determine the contribution of each loss term to the total loss; L train Represents the overall loss function.

[0098] The overall framework of this embodiment is as follows Figure 6 shown.

[0099] Embodiment 2

[0100] This embodiment also provides a multimodal sentiment analysis system based on sentiment polarity and sentiment intensity, including: a preprocessing module, an extraction module, a multi-granularity cross-modal interaction module, a dynamic feature integration module and a sentiment prediction module; the preprocessing module is used to collect initial video data and extract initial modal features; the extraction module is used to extract high-level semantic features in the initial modal features; the multi-granularity cross-modal interaction module is used to extract interaction information within and between modalities of high-level semantic features; the dynamic feature integration module is used to extract sentiment polarity and sentiment intensity vectors in the interaction information; the sentiment prediction module is used to complete character sentiment analysis in video data based on sentiment polarity and sentiment intensity vectors.

[0101] The following will explain in detail how the present invention solves technical problems in real life in conjunction with this embodiment.

[0102] First, during the video data acquisition and preprocessing stage, the acquisition module first uses Transkriptor's advanced speech recognition technology to convert the audio content into text, and then uses the pre-trained BERT model to extract the text features of the transcribed text. Next, the audio signal feature processor Librosa is used to extract the acoustic features of the person's speech from the audio, including Mel-frequency cepstral coefficients (MFCC), root mean square (RMS) energy and spectrum centroid, fundamental frequency, and frame energy. In addition, this embodiment uses the computer vision library OpenFace to perform in-depth processing on the video and extract a series of key visual features, including expression recognition, facial feature point detection, scene recognition, body posture estimation, and facial action tracking. These preprocessing steps provide high-quality initial modal features for subsequent multimodal sentiment analysis.

[0103] Afterwards, the extraction module extracts high-level semantic features from the initial modality features.

[0104] Three Transformer encoders and two MLP encoders are used to extract corresponding high-level semantic features from each modality data. In this process, the present embodiment divides the three Transformer encoders and the two MLP encoders into two stages, the first stage consists of the Transformer encoding layer, and the second stage consists of the MLP encoder (the MLP consists of two linear layers and a ReLU activation function).

[0105] The workflow of the extraction module is as follows:

[0106] b m =TransEncoder m (h m )

[0107] Cn =MLPEncoder m (b n )

[0108] In the formula, TransEncoder m and MLPEncoder m are the Transformer encoder and MLP encoder of the first and second stages respectively; m∈{v,a,t}, n∈{v,a}, where v represents the initial visual modality, a represents the acoustic modality, and t represents the text modality; h m b represents the feature representation of the modality; m represents the feature representation of modality m obtained by the first stage encoder; b n represents the feature representation of modality n obtained by the first stage encoder, c n Represents the feature representation of mode n obtained after the second stage.

[0109] This embodiment also designs a multi-granularity cross-modal interaction module, which aims to extract interaction information within and between modalities. This module achieves this goal by performing coarse-grained and fine-grained cross-modal interactions in sequence. Specifically, the coarse-grained interaction is first processed by a one-dimensional convolution layer (the convolution kernel size is 3x3) to ensure that the features of all modalities are mapped to a unified dimensional space. Then, the cross-modal attention mechanism Cross-A is used to implement cross-modal attention, and then the representation features of the coarse-grained interaction are obtained through residual connections and layer normalization. In the fine-grained interaction stage, this embodiment combines the representation features obtained in the coarse-grained stage with the original features of each modality, and applies the cross-modal attention mechanism again. After that, the fine-grained feature representation is obtained through residual connections and normalization. The feature representation is merged by vector addition operations, and then through a feedforward neural network layer, residual connections and normalization, the feature representation is spliced ​​to form a comprehensive multi-granularity feature representation, thereby providing rich information for subsequent analysis and processing.

[0110] The workflow of the multi-granularity cross-modal interaction module is as follows:

[0111]

[0112]

[0113]

[0114]

[0115] Among them, Cross-A represents the cross-modal attention mechanism; softmax is the normalized exponential function; X m and Xn Represents the modal characteristics of the input; Indicates input mode X n The query weight matrix of Indicates input mode X m The key weight matrix of Indicates input mode X m The value weight matrix; Conv represents a one-dimensional convolution operation; Z m and Z n Represents the feature representation of modalities m and n after the first Transformer encoder; Z nm Indicates that Z n and Z m Perform splicing operations; represents the coarse-grained interaction representation of modalities m and n, m∈{v,a,t}, n∈{v,a,t}; LN represents layer normalization; K represents the fine-grained interaction between modalities m and n; mn It represents the multi-granularity feature representation of modality m and modality n; concat represents the feature concatenation operation; FFN represents the feedforward neural network.

[0116] Considering that text-acoustic features are more significant in capturing emotional intensity, while text-visual features are more important in judging emotional polarity. In order to take advantage of both advantages, this embodiment proposes a dynamic integration module that combines the multi-granularity features K of text-acoustic ta and the visual modality feature c obtained by the second stage MLP encoder v , to extract multimodal emotion intensity features; similarly, this embodiment also combines the multi-granularity feature K of text-vision ta and the acoustic modal features c obtained by the second-stage MLP encoder v , to extract multimodal sentiment polarity features; the two input features are processed through a fully connected layer, softmax activation and residual connection to obtain a preliminary feature representation. Finally, through the Hadamard product and the enhancement of another fully connected layer, a more accurate multimodal sentiment feature representation is obtained. This embodiment performs vector addition on this representation and then passes it through a one-dimensional convolution and a sigmoid activation function to obtain the feature ratio value p. At this point, the two accurate multimodal feature representations can be combined to obtain the sentiment polarity and sentiment intensity representations. In particular, this embodiment uses square error to design a loss function for sentiment polarity, and uses correlation coefficient distance to design a loss function for sentiment intensity.

[0117] The workflow of the dynamic integration module is as follows:

[0118]

[0119]

[0120]

[0121]

[0122]

[0123] loss p =ω1(F p + -y + ) 2 +ω2(F p - -y - ) 2

[0124]

[0125] Among them, K tm Multi-granular features representing text modality and n-modality; c n represents the features of modality n obtained by the second-stage MLP encoder; Accurate representation of multi-granular features representing textual modality and n-modality, Represents the precise representation of the features of modality n, m∈{v,a}, n∈{v,a}; FC represents the fully connected layer; softmax represents the normalized exponential function; Hadamard represents element-wise multiplication; sigmoid represents the activation function; Conv represents one-dimensional convolution; pm represents the obtained feature ratio value; Accurate representation of multi-granular features representing textual modality and v-modality; represents the exact representation of the characteristics of mode a; F p Represents the emotional polarity obtained, Accurate representation of multi-granular features representing text modality and a-modality; represents an accurate representation of the characteristics of the v mode; F i Represents the intensity of the emotion obtained; F p + The average polar vector of positive sentiment that represents the test results; F p - Represents the average polar vector of negative sentiment of the predicted results; y + Represents the positive sentiment mean pole vector of the true label; y - represents the average polar vector of negative sentiment of the true label; ω1 represents the weight of positive sentiment; ω2 represents the weight of negative sentiment; |y| represents the absolute value of the true label; ||F i||1 represents the l1 norm of the predicted sentiment intensity vector; Cov represents the covariance calculation formula; exp represents the exponential function; Var represents the variance calculation formula; loss p Represents the emotional polarity loss we get; loss i Represents the loss of emotional intensity obtained.

[0126] Finally, the emotion prediction module completes the character emotion analysis in the video data based on the emotion polarity and emotion intensity vectors.

[0127] In the emotion prediction module, the emotion intensity vector F obtained in the above process is i and sentiment polarity vector F p Concatenate to get the sentiment vector representation F pi ; Then the obtained emotion representation is cross-modal attention interaction with the emotion polarity vector and the emotion intensity vector (same as Cross-A in the above step), followed by residual connection and normalization operations to obtain two interaction vectors, and then the two interaction vectors are vector added to obtain a mixed vector F, and finally the prediction value is obtained by using a fully connected layer after a feedforward neural network, residual connection, and normalization, and finally the emotion loss is calculated using the mean square error.

[0128] The workflow of the sentiment prediction module is as follows:

[0129] F pi =concat(F p ,F i )

[0130] F=LN(Crposs-A(F p , F pi )+F pi )+LN(Cross-A(F i , F pi )+F pi )

[0131]

[0132]

[0133] Among them, concat represents the feature concatenation operation, F p represents the sentiment polarity vector; F i represents the sentiment intensity vector; F pi represents the obtained sentiment vector; Cross-A represents the cross-modal attention machine; LN represents layer normalization, F represents the mixed vector; FFN represents the feedforward neural network, represents the predicted value of the i-th sample; yi represents the true value of the i-th sample; loss prerepresents the loss function of the predicted value; N represents the number of samples. The final loss function can be expressed as:

[0134]

[0135] In the formula, α, β, and ε represent hyperparameters, which determine the contribution of each loss term to the total loss; L train Represents the overall loss function.

[0136] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A multimodal sentiment analysis method based on sentiment polarity and sentiment intensity, characterized in that the steps include: Collect initial video data and extract initial modal features; Extracting high-level semantic features from the initial modal features; Extracting interactive information within and between modalities of the high-level semantic features; Extracting sentiment polarity and sentiment intensity vectors from the interaction information; Based on the emotion polarity and the emotion intensity vector, character emotion analysis in the video data is completed.

2. The multimodal sentiment analysis method based on sentiment polarity and sentiment intensity according to claim 1 is characterized in that: The extracted initial modal features include acoustic features and visual features; The acoustic features include: Mel frequency cepstral coefficients, root mean square energy and spectrum centroid, fundamental frequency and frame energy; firstly, the audio content is converted into text using speech recognition technology; then, the pre-trained BERT model is used to extract text features of the transcribed text; and the acoustic features are extracted from the audio using an audio signal feature processor; The visual features include: expression recognition, facial feature point detection, scene recognition, body posture estimation, and facial action tracking; the video is deeply processed using the computer vision library OpenFace to extract the visual features.

3. The multimodal sentiment analysis method based on sentiment polarity and sentiment intensity according to claim 1, characterized in that: The method for extracting the high-level semantic features includes: using three Transformer encoders and two MLP encoders to extract the high-level semantic features from each modality data respectively; dividing the three Transformer encoders and the two MLP encoders into two stages, the first stage consists of a Transformer encoding layer, and the second stage consists of an MLP encoder. The overall calculation process includes: b m =TransEncoder m (h m ) c n =MLPEncoder m (b n ) In the formula, TransEncoder m and MLPEncoder m are the Transformer encoder and MLP encoder of the first and second stages respectively; m∈{v,a,t}, n∈{v,a}, where v represents the initial visual modality, a represents the acoustic modality, and t represents the text modality; h m b represents the feature representation of the modality; m represents the feature representation of modality m obtained by the first stage encoder; b n represents the feature representation of modality n obtained by the first stage encoder, c n Represents the feature representation of mode n obtained after the second stage.

4. The multimodal sentiment analysis method based on sentiment polarity and sentiment intensity according to claim 1, characterized in that: The method of extracting the interactive information within the high-level semantic feature modality and between modalities includes: Among them, Cross-A represents the cross-modal attention mechanism; softmax is the normalized exponential function; X m and X n Represents the modal characteristics of the input; Indicates input mode X n The query weight matrix of Indicates input mode X m The key weight matrix of Indicates input mode X m The value weight matrix; Conv represents a one-dimensional convolution operation; Z m and Z n Represents the feature representation of modalities m and n after the first Transformer encoder; Z nm Indicates that Z n and Z m Perform splicing operations; represents the coarse-grained interaction representation of modalities m and n, m∈{v,a,t}, n∈{v,a,t}; LN represents layer normalization; K represents the fine-grained interaction between modalities m and n; mn It represents the multi-granularity feature representation of modality m and modality n; concat represents the feature concatenation operation; FFN represents the feedforward neural network.

5. The multimodal sentiment analysis method based on sentiment polarity and sentiment intensity according to claim 1, characterized in that: The method for extracting the emotion polarity and the emotion intensity vector includes: loss p =ω1(F p + -y + ) 2 +ω 2 (F p - -y - ) 2 Among them, K tm Multi-granular features representing text modality and n-modality; c n represents the features of modality n obtained by the second-stage MLP encoder; Accurate representation of multi-granular features representing textual modality and n-modality, Represents the precise representation of the features of modality n, m∈{v,a}, n∈{v,a}; FC represents the fully connected layer; softmax represents the normalized exponential function; Hadamard represents element-wise multiplication; sigmoid represents the activation function; Conv represents one-dimensional convolution; pm represents the obtained feature ratio value; Accurate representation of multi-granular features representing textual modality and v-modality; represents the exact representation of the characteristics of mode a; F p Represents the emotional polarity obtained, Accurate representation of multi-granular features representing text modality and a-modality; represents an accurate representation of the characteristics of the v mode; F i Represents the intensity of the emotion obtained; F p + The average polar vector of positive sentiment that represents the test results; F p - Represents the average polar vector of negative sentiment of the predicted results; y + Represents the positive sentiment mean pole vector of the true label; y - represents the average polar vector of negative sentiment of the true label; ω1 represents the weight of positive sentiment; ω2 represents the weight of negative sentiment; |y| represents the absolute value of the true label; ||F i ||1 represents the l1 norm of the predicted sentiment intensity vector; Cov represents the covariance calculation formula; exp represents the exponential function; Var represents the variance calculation formula; loss p Represents the emotional polarity loss we get; loss i Represents the loss of emotional intensity obtained.

6. The multimodal sentiment analysis method based on sentiment polarity and sentiment intensity according to claim 1, characterized in that: Based on the emotion polarity and the emotion intensity vector, completing the character emotion analysis in the video data, the steps include: F pi =concat(F p ,F i ) F=LN(Cross-A(F p ,F pi )+F pi )+LN(Cross-A(F i ,F pi )+F pi ) Among them, concat represents the feature concatenation operation, F p represents the sentiment polarity vector; F i represents the sentiment intensity vector; F pi represents the obtained sentiment vector; Cross-A represents the cross-modal attention machine; LN represents layer normalization, F represents the mixed vector; FFN represents the feedforward neural network, represents the predicted value of the i-th sample; yi represents the true value of the i-th sample; loss pre represents the loss function of the predicted value; N represents the number of samples; the final loss function is expressed as: In the formula, α, β, and ε represent hyperparameters, which determine the contribution of each loss term to the total loss; L train Represents the overall loss function.

7. A multimodal sentiment analysis system based on sentiment polarity and sentiment intensity, the system being used to implement the method according to any one of claims 1 to 6, characterized in that: include: Preprocessing module, extraction module, multi-granularity cross-modal interaction module, dynamic feature integration module and sentiment prediction module; The preprocessing module is used to collect initial video data and extract initial modal features; The extraction module is used to extract high-level semantic features from the initial modal features; The multi-granularity cross-modal interaction module is used to extract the interaction information within the high-level semantic feature modality and between modalities; The dynamic feature integration module is used to extract the emotion polarity and emotion intensity vectors in the interaction information; The emotion prediction module is used to complete the character emotion analysis in the video data based on the emotion polarity and the emotion intensity vector.

Images