Multimodal sentiment analysis method based on word-level interaction and word-by-word sentiment quantity weighting

Abstract

The multi-modal sentiment analysis method based on word-level interaction and word-by-word sentiment content weighting belongs to the technical field of deep learning. The BERT and Wav2vec 2.0 pre-training models are used to realize the extraction of text modal and speech modal word-level feature vectors; the Transform encoder and linear layer are used to obtain the text-speech dual-modal word-level interaction sentence-level feature vectors; the Bi-LSTM model is used to realize the word-by-word sentiment content weighting of the word-level feature vectors, so as to obtain the text-speech dual-modal word-by-word sentiment content weighted sentence-level feature vectors. Finally, the text-speech dual-modal word-level interaction sentence-level feature vectors and the text-speech dual-modal word-by-word sentiment content weighted sentence-level feature vectors are fused to obtain the multi-modal multi-granularity information sentence-level feature vectors, and the full connection network is used for sentence sentiment recognition. The present application reduces the loss of sentiment information caused by the granularity difference.

Description

Technical Field

[0001] Based on deep learning technology, a multimodal sentiment recognition method with multi-granularity feature fusion is designed. First, BERT and Wav2vec 2.0 pre-trained models are used to extract word-level feature vectors for both text and speech modalities. A Transformer encoder and linear layers are used to obtain sentence-level feature vectors for bimodal text-speech word-level interaction. A Bi-LSTM model is used to weight the word-level feature vectors according to word-by-word sentiment content, resulting in a sentence-level feature vector for bimodal text-speech based on word-by-word sentiment content. Finally, the sentence-level feature vectors for bimodal text-speech word-level interaction and the sentence-level feature vectors weighted by word-by-word sentiment content are fused to obtain a multimodal sentence-level feature vector containing multi-granularity information. A fully connected network is then used for sentiment recognition of the sentences. This invention relates to the field of sentiment analysis, specifically to deep learning, multimodal fusion, and multi-granularity feature fusion techniques. Background Technology

[0002] Sentiment analysis is a crucial component of Natural Language Processing (NLP), aiming to transform human emotions into machine-recognizable features through algorithms, enabling machines to identify and distinguish human emotions. With the development of artificial intelligence, the demand for human emotion prediction is gradually increasing, with high demand for sentiment recognition and analysis technology in fields such as chatbot customer service, large language model construction, and big data recommendation. Human emotions are expressed in multiple modalities, such as verbal expression, written expression, body language, and facial expressions, and different modalities have complementary emotional features. Multimodal fusion sentiment analysis is a method that simultaneously utilizes features from multiple modalities to analyze human emotions, leveraging the complementarity between modalities to improve the accuracy of sentiment prediction and reduce the limitations of single modalities. The information expressed by human emotions varies at different granularities; the emotions expressed by individual words and the emotions expressed by complete sentences are both related and distinct. Utilizing complementary features at the word level and sentence level can improve the accuracy of human emotion prediction.

[0003] Traditional multimodal sentiment analysis methods include sentence-level multimodal fusion and word-level multimodal fusion. Sentence-level multimodal fusion requires extracting word-level feature vectors from each modality and calculating sentence-level feature vectors based on these vectors. The sentence-level feature vectors of identical sentences from multiple modalities are then concatenated to obtain the multimodal fused sentence-level feature vector. Deep learning is used to train a model on this multimodal fused sentence-level feature vector, and the trained model is used for sentiment recognition. Word-level multimodal fusion requires concatenating word-level feature vectors from different modalities, training a model using deep learning, and then using the trained model for sentiment recognition. Traditional sentence-level feature vector construction involves averaging the word-level feature vectors, using this average as the sentence-level feature vector. For word-level multimodal fusion, feature vector construction requires alignment between different modalities, using temporal or semantic alignment to concatenate the word-level feature vectors from different modalities to obtain the word-level multimodal fusion feature vector.

[0004] Traditional methods use sentence-level features and word-level features separately as the basis for sentiment analysis, without studying the correlation between the two. However, there is a connection between word-level and sentence-level features, and using only one granularity as the basis for sentiment recognition can lead to the loss of some features. Traditional sentence-level feature vector construction methods use the average of word-level feature vectors within a sentence. However, human emotional expression is complex, and the emotional content of each word in a sentence varies. Calculating using the average method increases the error impact of words with low emotional content on the complete sentence, and reduces the ability of words with high emotional content to represent the sentiment of the complete sentence, resulting in a model with low accuracy. When performing sentence-level multimodal fusion, for noisy words with poor emotional expression, the traditional solution is to construct a low-frequency vocabulary dictionary to remove noisy words from the dataset before feature extraction and model training. However, in multimodal fusion sentiment analysis, the ability of noisy data to express sentiment varies across different modalities. For example, a word may contain a lower level of sentiment in the text modality but a higher level of sentiment in the speech modality. Therefore, removing identical words from other modalities based on the noise vocabulary standard of a single modality will affect the complementary ability between multimodalities. Summary of the Invention

[0005] To address the feature loss problem caused by learning single-granularity features in traditional multimodal sentiment analysis methods and the problem of using average values ​​as sentence-level feature vectors in sentence-level multimodal fusion methods, this invention proposes a multimodal sentiment analysis method based on word-level interaction and word-by-word sentiment weighting. The proposed method consists of four parts (…). Figure 1The methodology includes: text-speech bimodal word-level feature extraction, multimodal temporal alignment and word-level granular multimodal fusion, word-by-word sentiment weighting and sentence-level granular multimodal fusion, and multimodal multi-granular sentiment analysis model training. For text-speech bimodal word-level feature extraction, a pre-trained model is used to extract feature vectors. Wav2vec 2.0 is used to extract word-level feature vectors for the speech modality, and BERT is used to extract word-level feature vectors for the text modality. In the multimodal temporal alignment and word-level granular multimodal fusion part, word-level feature vectors from different modalities of the same word are concatenated to obtain text-speech bimodal word-level feature vectors. These vectors are then input into a Transformer encoder for encoding. The encoder output is then passed through a linear layer for dimensionality reduction to obtain the sentence-level feature vector representing the text-speech bimodal word-level interaction of the sentence. In the word-by-word sentiment weighting and sentence-level granular multimodal fusion part, word-by-word feature vectors from both modalities are input into Bi-LSTM networks for training, resulting in separate Bi-LSTM models for each modality. Subsequently, the word-by-word feature vectors obtained from BERT and Wav2vec2.0 are input into the corresponding modal Bi-LSTM models. The models output the sentiment prediction probabilities for each word, and the variance of these probabilities is calculated as the weight of the sentiment content contained in that word. The word-by-word feature vectors of the corresponding modal are weighted using the obtained weights of the word-by-word features in different modalities to obtain sentence-level feature vectors with weighted word-by-word sentiment content for both speech and text modalities. The sentence-level feature vectors with weighted word-by-word sentiment content for the same utterance in both modalities are concatenated to obtain a sentence-level feature vector with weighted word-by-word sentiment content for both text and speech modalities. The multimodal, multi-granularity sentiment analysis model is trained using a fully connected network as the classification model structure. The sentence-level feature vectors of the speech-text bimodal word-level interaction for the same utterance are concatenated with the sentence-level feature vectors with weighted word-by-word sentiment content for both text and speech modalities to obtain a sentence-level feature vector containing multi-granularity information for both text and speech modalities. This feature vector is then input into the fully connected network for training to obtain a multi-classification model for sentiment prediction, which is used to perform sentence sentiment analysis.

[0006] To achieve the above objectives, the multimodal sentiment analysis method of the present invention, based on word-level interaction and word-by-word sentiment weighting, includes the following steps:

[0007] Preparation phase:

[0008] 1. Text-Speech Dual-Modal Word-Level Feature Extraction

[0009] (1-1) Extracting word-level feature vectors of speech modalities: Mark the start and end times of each word in a sentence. While reading the audio waveform, read the start and end times of the words contained in the audio. The calculation formulas for the audio start frame SFrm and end frame EFrm are as follows:

[0010] t=N÷fs (1)

[0011] SFrm=ST÷t (2)

[0012] EFrm=ET÷t (3)

[0013] In formula (1), t represents the length of an audio frame, indicating the duration of a frame. N represents the number of sampling points in an audio frame, fs represents the audio sampling rate, ST represents the start time of a word, and ET represents the end time of the same word. Formulas (1), (2), and (3) can be used to calculate the start frame SFrm and end frame EFrm of the word, thus obtaining the accurate frame range of each word in the audio within a sentence. Calculate the word-level audio data of all sentences in the dataset by applying the above steps to all sentences in the dataset. Input the audio waveform within the range corresponding to each word of a sentence into the Wav2vec2.0 pre-trained model to extract features, obtaining the word-level feature vector of a sentence. For example, if a sentence contains M words, and the feature vector extracted by Wav2vec 2.0 is L1-dimensional, then the sentence will ultimately obtain a feature matrix of shape L1×M, where each column of the matrix is ​​a speech modality word-level feature vector, the number of columns M is the number of words in the sentence, and the number of rows L1 is the length of the feature vector. Perform the above operations on all statements in the dataset to obtain the speech modal word-level feature vectors of the complete dataset and the speech modal feature matrix of each statement.

[0014] (1-2) Extracting Text Modality Word-Level Feature Vectors: Text feature extraction employs word-level feature extraction. The sentences to be trained are input into the BERT pre-trained model, and the word-level feature vectors output by BERT are used as word-by-word feature vectors. For example, if a sentence contains M words, BERT outputs a feature matrix of shape L2×M, where each column of the matrix is ​​a text modality word-level feature vector, the number of columns M is the number of words in the sentence, and the number of rows L2 is the length of the feature vector. The above operation is performed on all sentences in the dataset to obtain the text modality word-level feature vectors of the complete dataset and the text modality feature matrix of each sentence.

[0015] 2. Multimodal temporal alignment and word-level granular multimodal fusion

[0016] (2-1) Multimodal time alignment: The word-level feature vectors of the speech modality and text modality obtained in steps (1-1) and (1-2) correspond to each other. Figure 2The speech modality word-level feature vector (box 1) and text modality word-level feature vector (box 2) are concatenated to obtain a text-speech bimodal word-level feature vector, which in turn yields the text-speech bimodal word-level feature matrix for each sentence. For a sentence containing M words, step (1-1) yields a speech modality word-level feature matrix of shape L1×M, with each column representing the speech modality feature vector of a single word. Step (1-2) yields a text modality word-level feature matrix of shape L2×M, with each column representing the text modality feature vector of a single word. These two feature matrices are concatenated to obtain a time-aligned speech-text bimodal word-level feature matrix of shape (L1+L2)×M, where each column represents a time-aligned text-speech bimodal word-level feature vector. This text-speech bimodal concatenation operation is performed on the feature matrices of all sentences in the dataset to obtain the time-aligned text-speech bimodal word-level feature matrix for each sentence. Figure 2 (Box 3)

[0017] (2-2) Configure the Transformer network: The Transformer network structure is as follows: Figure 3 As shown, the network input is the time-aligned text-speech bimodal word-level feature matrix of shape L×M obtained in step (2-1), where L = L1 + L2. The model network consists of S Transformer encoders, each with L×M input and output. The last encoder layer is connected to a linear layer. The linear layer takes an L×M feature matrix as input and outputs an L-dimensional feature vector, which is... Figure 2 The sentence-level feature vector of the bimodal text-to-speech word-level interaction in box 5.

[0018] 3. Word-by-word sentiment weighting and sentence-level granular multimodal fusion

[0019] (3-1) Configure the Bi-LSTM network: Define two Bi-LSTM networks with the same structure for the audio and text modalities respectively (boxes 3 and 4 in Figure 4). The specific structure of the Bi-LSTM network is as follows: Figure 5As shown, the speech modality feature matrix with shape L1×M extracted in step (1-1) is input into the speech modality Bi-LSTM network, and the text modality feature matrix with shape L2×M extracted in step (1-2) is input into the text modality Bi-LSTM network. The feature matrices are then fed into the encoders of their respective modalities, with the number of RNN layers set to a predetermined value. The encoder output is then subjected to Dropout, randomly pruning neurons with probability p, where p is a specific value. During subsequent network training, this process is adjusted by more than 10 iterations, selecting the probability p that maximizes the accuracy on the test set as the Dropout probability. The pruned data is then input into a linear layer, whose output is the probability of each sentiment in the predicted feature vector. This probability is used as the output of the Bi-LSTM network. For example, if a sentence contains M words and N sentiments are predicted, the network output is an N×M sentiment probability matrix, where each column represents the N predicted sentiment probability vectors, M is the number of words in the sentence, and N is the number of predicted sentiments.

[0020] (3-2) Input the speech modality feature matrix and the text modality feature matrix into the corresponding Bi-LSTM network of step (3-1) for training, and use the cross-entropy loss function as the model loss calculation function. During training, backpropagate the model according to the loss until the accuracy of the training set increases and the accuracy of the test set reaches the optimal value and no longer increases, then stop training. By training the two modalities separately, the text modality Bi-LSTM model (boxes 3A and 3B in Figure 4) and the speech modality Bi-LSTM model (boxes 4A and 4B in Figure 4) are obtained.

[0021] (3-3) Input the speech modal feature matrix into the speech modal Bi-LSTM model for inference, and obtain Figure 4B The probability matrix for word-by-word sentiment prediction of the speech modality is shown in box 6. The text modality feature matrix is ​​then input into the text modality Bi-LSTM model for inference, yielding... Figure 4B Box 5 shows the word-by-word sentiment prediction probability matrix for the text modality. The probability matrices obtained from the two modalities have the following shapes: Figure 6 As shown, the number of columns M in the matrix represents the number of words in the predicted sentence, the number of rows N represents the number of predicted sentiments, and the columns of the matrix consist of word-by-word sentiment prediction probability vectors. Each element in each column corresponds to the probability of a predicted sentiment. The complete matrix represents a sentence containing M words, with each word containing N different sentiments and their probabilities. In the dataset, each sentence's two modalities are processed using a corresponding Bi-LSTM model, resulting in a word-by-word sentiment prediction probability matrix for each sentence.

[0022] (3-4) In this invention, the variance of the word-by-word sentiment prediction probability vector in each row of the word-by-word sentiment prediction probability matrix obtained in step (3-3) is used as a metric for evaluating the word-by-word sentiment content. The specific formula for calculating the variance is as follows:

[0023]

[0024] Formula (4) is used to calculate the average modal sentiment prediction probability of a word. Each x in the molecule i Let N be the predicted probabilities of each sentiment within the same modality for a given word, and let N be the predicted sentiments in total. Formula (5) is used to calculate the variance of the predicted sentiment probabilities for a given word within a given modality. S is the average value of the same word and the same modality calculated in formula (4). 2 Let Variance be the variance of the sentiment prediction probability for this word in this modality. Collect all the variances of sentiment prediction probabilities for the same modality within a sentence and form a set of sentiment prediction probability variances. Normalize all variances within the set so that the sum of the variances of all words equals 1. This yields the weight of the sentiment content contained in each word within the corresponding modality of the sentence. Sort the weights obtained for each modality according to the order of the corresponding words and form a vector of length M for the sentiment content weights, denoted as vector A. Calculate the sentiment content weight vector for both the text and speech modalities of all sentences in the dataset.

[0025] (3-5) Multiply the sentiment content weight vector A obtained in step (3-4) with the feature matrices of the corresponding sentences in the same modality from steps (1-1) and (1-2) to obtain the sentence-level feature vector of word-by-word sentiment content weighted within a modality. For example, in the speech modality, a sentence has M words, the speech modality feature matrix is ​​B, the shape of matrix B is L1×M, the matrix B has L1 rows and M columns, the sentiment content weight vector is A, the shape of vector A is M×1, the vector A has M rows and 1 column. The sentiment content weighted feature vector C of this sentence is B×A, the shape of vector C is L1×1, the vector C has L1 rows and 1 column. Calculate the word-by-word sentiment content weighted sentence-level feature vectors for the text modality and speech modality of all sentences in the dataset, and obtain the following results. Figure 4B Boxes 9 and 10 concatenate the sentence-level feature vectors, which are weighted by word-by-word sentiment content from the speech and text modalities of the same sentence, to obtain the final result. Figure 4B The L-dimensional, word-by-word sentiment-weighted text-speech bimodal sentence-level feature vector in box 11 is L = L1 + L2, where L1 is the length of the word-by-word feature vector in the speech modality and L2 is the length of the word-by-word feature vector in the text modality.

[0026] 4. Training of Multimodal and Multigranular Sentiment Analysis Models

[0027] (4-1) Building a Fully Connected Network: The specific structure of a fully connected network is as follows: Figure 7 As shown, it contains two linear layers and one ReLU activation function. The L-dimensional text-speech bimodal word-level interaction sentence-level feature vector obtained in step (2-2) and the L-dimensional text-speech bimodal sentence-level feature vector weighted by word-by-word sentiment content obtained in step (3-5) are concatenated to obtain a 2L-dimensional multimodal sentence-level feature vector containing multi-granularity information, which is used as the input to the fully connected network. Figure 7 Box 1), Figure 7 The linear layer input length of box 2 is the length of the multimodal, multi-granularity information statement-level feature vector, and the output length is the same as the input length. Figure 7 The output of the linear layer in box 2 is input into the Rectified Linear Unit (ReLU) activation function, and the ReLU output is input into... Figure 7 The linear layer in box 3 has an input length that is... Figure 7 Similar to box 2, the output length is N, which represents the number of predicted sentiments. Input the network's output sentiment prediction probability vector into the softmax function:

[0028]

[0029] The Softmax function is shown in formula (6), α i (α j ) represents the i(j)th element of the sentiment prediction probability vector, and the position of the maximum value of the function output result is the number of the predicted sentiment.

[0030] (4-2) Multi-network training: The networks participating in the training include the Transformer encoder and linear layers in the Transformer network configured in step (2-2) and the fully connected network configured in step (4-1). All data in the dataset is divided into training and test sets. The training set is used to train the networks participating in the training, and the test set is used to evaluate the network training performance. The cross-entropy loss function is used for network training, as shown in formula (7):

[0031]

[0032] N is the length of the network's output sentiment prediction probability vector, i.e., the number of predicted sentiments, β i For sentiment prediction probability vector i, p(β) i Let q(β) be the true probability of the sentiment numbered i, expressed in one-hot form. iLet H(p,q) represent the predicted probability of sentiment with index i. A smaller cross-entropy value indicates a better prediction result. Based on the calculated loss, backpropagation is performed to optimize the fully connected network and the Transformer network sequentially until the accuracy of the training set increases while the accuracy of the test set reaches its optimal value and no longer increases. Training is then stopped, resulting in a complete model network with multimodal and multigranular sentiment analysis capabilities.

[0033] Prediction phase:

[0034] The complete model network with multimodal and multigranular sentiment analysis capabilities obtained through the above steps is the final result obtained by the method proposed in this invention. A new sentence is processed through steps (1-1) and (1-2) to obtain the speech modality word-level feature matrix and the text modality word-level feature matrix. Through steps (2-1) and (2-2) to obtain the sentence-level feature vector of text-speech bimodal word-level interaction, through steps (3-3), (3-4) and (3-5) to obtain the text-speech bimodal sentence-level feature vector with word-by-word sentiment content weighting, and through step (4-1) to finally obtain the predicted sentiment label.

[0035] Beneficial effects:

[0036] The multimodal sentiment analysis method proposed in this invention, based on word-level interaction and word-by-word sentiment weighting, aims to perform sentiment recognition at two different granular levels: word-level features and sentence-level features. This invention achieves the acquisition of two different sentence-level feature vectors. The sentence-level feature vector extracted from the multimodal time-aligned and word-level granular multimodal fusion part contains the contextual information of each word within a single sentence, and is a multimodal sentence-level feature vector containing word-level granular information. The sentence-level feature vector extracted from the word-by-word sentiment weighting and sentence-level granular multimodal fusion part is a text-speech bimodal sentence-level feature vector weighted by word-by-word sentiment content. The sentence-level feature vector is calculated using the word-level feature vectors by weighting and summing, resulting in a multimodal sentence-level granular information sentence-level feature vector. Concatenating sentence-level feature vectors of multiple granularities avoids the loss of sentiment information contained in the sentence, obtains the relationship between the sentiment information contained at different granularities, and provides a more comprehensive analysis of the sentiment expressed by the sentence.

[0037] In this invention, the word-by-word sentiment content weighting and sentence-level multimodal fusion section utilizes the variance of the sentiment probability of the model's inference results from word feature vectors as a standard for evaluating the sentiment content of words. This effectively improves the rationality of the calculation method from word-level feature vectors to sentence-level feature vectors. The weighting method in this section reduces the sentiment guidance of words with low sentiment content on complete sentences and increases the sentiment guidance of words with high sentiment content. Words with low sentiment content can be considered noise signals during training. The method proposed in this invention can reduce the weight of noise signals in sentence-level feature vectors, thereby reducing the impact of noise on model training and inference while increasing the weight of words with high sentiment content in sentence-level feature vectors. This section can improve the complementarity between modalities during multimodal fusion and reduce the negative impact of traditional noise data removal on modal complementarity.

[0038] Compared with existing methods, the present invention has the following advantages:

[0039] Traditional sentiment analysis methods use single-granularity features as the standard for sentiment analysis. However, different granularities contain different sentiment information, and using a single granularity can lead to the loss of sentiment information. The expression of sentiment information is complex; the sentiment expressed by each word differs from the sentiment expressed by the complete sentence, and they also have complementary capabilities. The multimodal sentence-level feature vector proposed in this invention contains both word-level and sentence-level sentiment information, reducing the loss of sentiment information caused by granularity differences. The word-level sentiment content weighting and sentence-level granularity multimodal fusion method proposed in this invention can reduce the weight of words with low sentiment content in the sentence-level feature vector, reducing noise from useless data. In traditional methods, calculating the sentence-level feature vector requires averaging the word-level feature vectors to represent the sentence. During the averaging process, the distinguishability between feature vectors of words with high and low sentiment content is reduced, thus amplifying the influence of noisy words on the overall sentence features and reducing the representational ability of important words with high sentiment content for the complete sentence features. The sentence-level feature vectors processed by the method of this invention, weighted by word-by-word sentiment content, solve the above-mentioned problems. Noisy words have lower weights, while important words have higher weights, thus improving the accuracy of the sentence-level feature vectors. This method can improve the ability of traditional methods to reduce inter-modal complementarity when removing noisy data. Because different modalities express sentiment differently, noisy words in one modality cannot be identified as noisy words in another modality. Therefore, weighting multiple modalities separately and then concatenating them can avoid the decrease in inter-modal complementarity caused by directly removing noisy words from a single modality. Attached Figure Description

[0040] Figure 1 This describes the complete process of the present invention and the operations performed in each part.

[0041] Figure 2 This document describes the specific operation process and input / output data for the multimodal time alignment and word-level granular multimodal fusion part of this invention.

[0042] Figure 3 This describes the specific structure of the Transformer network configured in this invention, as well as the network's input and output data.

[0043] Figure 4A This is the complete process of word-by-word sentiment content weighting and sentence-level granular multimodal fusion in this invention. Figure 4B This refers to the input and output data for this section.

[0044] Figure 5 This invention describes the specific network structure and input / output data in the Bi-LSTM network configuration step.

[0045] Figure 6 This describes the specific shape of the output data from the two-modal Bi-LSTM networks when calculating the shape of the probability matrix in this invention.

[0046] Figure 7 This invention describes the specific structural configuration of the network and the specific input and output data in the step of configuring a fully connected network. Detailed Implementation

[0047] This example uses a dataset containing both speech and text bimodal data as a case study. The dataset contains 10 sentences, with 4 sentences to be predicted in terms of sentiment. Sentence K is a sentence in the dataset containing two words, demonstrating the complete process from dataset preparation to sentiment prediction. Figure 1 For a complete process, it includes Figure 1 This includes all the flowcharts and specific examples of all operations within the framework.

[0048] Preparation phase:

[0049] 1. Text-Speech Dual-Modal Word-Level Feature Extraction

[0050] (1-1) Extracting word-by-word feature vectors of speech modalities: Segment the speech modal sentences in the dataset and mark the start and end frames of each word. For example, sentence K contains two words. The start time ST of the first word is second 0, and the end time ET is second 2. The audio sampling rate fs is 16000Hz, and the sampling points N in one audio frame are 1152. The audio frame length t is calculated to be 0.072 seconds by formula (1). The start frame of this word is calculated to be frame 0 and the end frame is calculated to be frame 28 by formulas (2) and (3), respectively. The frame range is from frame 0 to frame 28. Repeat the operations in formulas (1), (2), and (3) for all words in this sentence to obtain the frame range of all words in the sentence. By calculating the frame range of all 10 speech modal data in the dataset, the word-level audio data of the complete dataset can be obtained.

[0051] The data within the frame range of all words is input into the Wav2vec 2.0 pre-trained model to extract word-by-word feature vectors of the speech modality. Taking sentence K as an example, the processing for other sentences is exactly the same. Sentence K contains two words, with the first word's frame range being 0 to 28. The audio data of frames 0 to 28 of this sentence are input into the Wav2vec 2.0 pre-trained model, and the model outputs a 512-dimensional feature vector. The data within the frame range of all words of this sentence are then sequentially input into the Wav2vec 2.0 pre-trained model, ultimately obtaining a 512×2 speech modality feature matrix. Each column of the matrix is ​​a 512-dimensional word-by-word feature vector of the speech modality. Since the sentence has two words, each word corresponds to a column of 512-dimensional word-level speech modality feature vectors. The same feature extraction operation is performed on the audio of all 10 sentences in the dataset, ultimately obtaining all 10 feature vectors of shape 512×M. i The characteristic matrix, M i Let be the number of words in each sentence, and i be the sentence number. Each speech modality word-level feature matrix consists of speech modality word-level feature vectors, i.e. Figure 2 Box 1 and Box 2 in Figure 4.

[0052] (1-2) Extracting Text Modality Word-by-Word Feature Vectors: Text modality feature extraction employs word-level feature extraction, implemented using BERT pre-training. Taking sentence K as an example, sentence K contains two words. Inputting sentence K into the BERT pre-trained model and using BERT's word-level feature vectors as output yields a text modality feature matrix of shape 768×2, where each column of the matrix is ​​a 768-dimensional text modality word-by-word feature vector. Sentence K contains two words, and each word corresponds to a 768-dimensional text modality word-by-word feature vector. Text modality word-level feature extraction is performed on all text modality data in the dataset, resulting in a total of 768×M vectors for all 10 text modality sentences. i The text modal feature matrix, Mi Let be the number of words in each sentence, and i be the sentence number. Each text modality feature matrix consists of word-by-word feature vectors of the text modality, i.e. Figure 2 Box 2 and Box 1 in Figure 4.

[0053] 2. Multimodal temporal alignment and word-level granular multimodal fusion

[0054] (2-1) Multimodal temporal alignment: Concatenate the word-level feature vectors of the same words obtained in steps (1-1) and (1-2) to obtain a text-speech bimodal word-level feature vector, and then obtain the text-speech bimodal word-level feature matrix for each sentence. Taking sentence K as an example, sentence K contains two words. In step (1-1), a speech modal feature matrix with a shape of 512×2 is obtained, and in step (1-2), a text modal feature matrix with a shape of 768×2 is obtained. Each column of the matrix is ​​a word-level feature matrix of one modality. Concatenate the two matrices, that is, concatenate the word-level feature vectors of each column of the two matrices, to obtain a time-aligned text-speech bimodal word-level feature matrix with a shape of 1280×2. Each column of the matrix is ​​a 1280-dimensional time-aligned text-speech bimodal word-level feature matrix, containing a total of 2 words. Extract features from all sentences in the dataset using the same method as sentence K to obtain the time-aligned text-speech bimodal word-level feature matrix for each sentence. Figure 2 (Box 3)

[0055] (2-2) Configure the Transformer network: The Transformer network structure is as follows: Figure 3 As shown, the network input is the time-aligned text-speech bimodal word-level feature matrix obtained in step (2-1). Taking sentence K as an example, the Transformer network input is the time-aligned text-speech bimodal word-level feature matrix with a shape of 1280×2 obtained in step (2-1). The number of Transformer encoder layers is set to S=2, and the input and output of each encoder layer are feature matrices with a shape of 1280×2. The linear layer input is a 1280×2 feature matrix, and the output is a 1280-dimensional feature vector, which serves as the output of the Transformer network. The network output is the sentence-level feature vector of text-speech bimodal word-level interaction (…). Figure 2 (Box 5)

[0056] 3. Word-by-word sentiment weighting and sentence-level granular multimodal fusion

[0057] (3-1) Configure Bi-LSTM networks: Define two identical Bi-LSTM networks for the audio and text modalities respectively, such as... Figure 5 As shown. The input to the speech modal network is L1×M. i The speech modal feature matrix, which is an L1-row M matrix.i Column, where M i Let be the number of words in a sentence, i be the sentence number, and L1 be the length of the word-by-word feature vector of the speech modality. The input to the text modality network is L2×M. i The text modal feature matrix, which is an L2 row M matrix. i Column, where M i Let represent the number of words in a sentence, ...

[0058] (3-2) Split all sentences in the dataset into training and test sets in a 4:1 ratio. Input the speech modality feature matrix and text modality feature matrix in the training set into the corresponding modality Bi-LSTM network according to the method of sentence K in step (3-1), and use the cross-entropy loss function as the model's loss calculation function. During training, backpropagate the model based on the loss and calculate the accuracy of the training set and test set respectively. Stop training when the accuracy of the training set increases and the accuracy of the test set reaches the optimal value and no longer increases. The speech modality data is used to train the corresponding Bi-LSTM model for the speech modality, and the text modality data is used to train the corresponding Bi-LSTM model for the text modality.

[0059] (3-3) Input the speech modality feature matrix and the text modality feature matrix into the corresponding modality Bi-LSTM model trained in step (3-2) for inference, to obtain the word-by-word sentiment prediction probability matrices for the two modalities. Input the 512×2 feature matrix of the speech modality in sentence K into the corresponding Bi-LSTM network for inference, to obtain a 4×2 word-by-word sentiment prediction probability matrix of the speech modality, with a matrix of 4 rows and 2 columns. Input the 768×2 feature matrix of the text modality in sentence K into the corresponding Bi-LSTM network for inference, to obtain a 4×2 word-by-word sentiment prediction probability matrix of the text modality, with a matrix of 4 rows and 2 columns. Each row in the word-by-word sentiment prediction probability matrix of each modality is the sentiment prediction probability vector of the modality corresponding to a word in sentence K, and the number of columns is the number of words in sentence K. Perform the same operation as for sentence K on all sentences in the dataset, to obtain a total of 10 word-by-word sentiment prediction probability matrices for each modality, each matrix having a shape of 4×M. i M i denoted by , where i represents the number of words in the statement and is the statement number.

[0060] (3-4) Taking statement K as an example, the sentiment prediction probability vector of the first word of statement K obtained in step (3-3) is (0.1, 0.1, 0.1, 0.7). T According to formula (4), the average value of the speech modality sentiment prediction probability for this word is 0.25, and according to formula (5), the variance of the speech modality sentiment prediction is 0.0675. The sentiment prediction probability vector for the second word obtained in step (3-3) is (0.2, 0.2, 0.2, 0.4). T Therefore, according to formula (4), the average value of the sentiment prediction probability of this word is 0.25, and according to formula (5), the variance of the sentiment prediction probability is 0.0075. Thus, the vector of the variance of the sentiment prediction probability of statement K is (0.0675, 0.0075). T After normalization, the speech modality sentiment content weight vector is obtained as (0.9, 0.1). T The first word has a weight of 0.9, the second word has a weight of 0.1, and the sum of all word weights equals 1. For all sentences in the dataset, both text and speech modalities are calculated using the same method as for sentence K, resulting in 10 sentiment weight vectors for each modality, with a vector shape of M. i ×1, M i The number of words contained in the statement, where i is the statement number.

[0061] (3-5) Multiply the sentiment content weight vector obtained in step (3-4) with the feature matrix of the corresponding sentence in steps (1-1) and (1-2) to obtain the sentence-level feature vector of word-by-word sentiment content weighted for each sentence in each modality. Taking sentence K as an example, the speech modality word-level feature matrix with a shape of 512×2 obtained in step (1-1) is denoted as B. K In step (3-4), the emotional content weight vector of the speech modality is obtained as (0.9, 0.1). T Then, the sentence-level sentiment feature vector of sentence K, weighted by sentiment content in the speech modality, is B. K ×(0.9,0.1) T This yields a vector of shape 512×1. Similarly, the text modality word-by-word sentiment weighted sentence-level feature vector of sentence K is obtained by multiplying the text modality word-by-word sentiment weight vector by the text modality feature matrix of shape 768×2 from step (1-2), resulting in a vector of shape 768×1 columns. Weighting is performed on all data in the dataset to obtain the text modality 768-dimensional word-by-word sentiment weighted sentence-level feature vectors and the speech modality 512-dimensional word-by-word sentiment weighted sentence-level feature vectors for all 10 sentences. The speech modality word-by-word sentiment weighted sentence-level feature vector and the text modality word-by-word sentiment weighted sentence-level feature vector of the same sentence are concatenated with the speech modality first and the text modality second, finally yielding a 1280-dimensional word-by-word sentiment weighted text-speech bimodal sentence-level feature vector. For example, the sentence-level feature vector of speech modality weighted by word sentiment content obtained from sentence K is a 512-dimensional vector, and the sentence-level feature vector of text modality weighted by word sentiment content is a 768-dimensional vector. After concatenation, a 1280-dimensional sentence-level feature vector of text and speech modality weighted by word sentiment content is obtained.

[0062] 4. Training of Multimodal and Multigranular Sentiment Analysis Models

[0063] (4-1) Building a Fully Connected Network: The specific structure of a fully connected network is as follows: Figure 7 As shown, the network input is a multimodal sentence-level feature vector containing multi-granular information. Taking sentence K as an example, the 1280-dimensional text-speech bimodal word-level interaction sentence-level feature vector obtained in step (2-1) and the 1280-dimensional text-speech bimodal sentence-level feature vector weighted by word-by-word sentiment content obtained in step (3-5) are concatenated to obtain a 2560-dimensional multimodal sentence-level feature vector containing multi-granular information, which is then used as the input to the fully connected network. Figure 7 Box 1). Figure 7 The linear layer in box 2 takes a 2560-dimensional input and outputs a 2560-dimensional feature vector. Figure 7 The output of the linear layer in box 2 is input into the ReLU activation function, and the ReLU output is then input into... Figure 7The linear layer in box 3 has an input length of 2560-dimensional feature vectors and an output length of 4, which is the number of predicted sentiments. The sentiment prediction probability vector output by the network is input into the softmax function of formula (6), and the position of the maximum value of the function output is the number of the predicted sentiment.

[0064] (4-2) Multi-network training: The networks involved in training include the Transformer encoder and linear layers in the Transformer network configured in step (2-2) and the fully connected network configured in step (4-1). The 10 sentences in the dataset are divided into training and test sets in a 4:1 ratio, so the training set contains 8 sentences and the test set contains 2 sentences. The network uses the cross-entropy loss function of formula (7) as the training loss function. The cross-entropy loss is calculated between the predicted sentiment number and the target sentiment number obtained in step (4-1), and the network is optimized according to the backpropagation of the loss. The fully connected network configured in step (4-1) and the Transformer network configured in step (2-2) are optimized until the accuracy of the training set increases while the accuracy of the test set reaches the optimal value and no longer increases. The trained fully connected network model is then obtained.

[0065] Prediction phase:

[0066] In this example, for the new sentences not included in the dataset, there are a total of 3 words, and the number of predicted emotions is 4. Step (1-1) is performed on the speech modality data to obtain a 512×3 speech modality feature matrix. Step (1-2) is performed on the text modality data to obtain a 768×3 text modality feature matrix. The speech and text modality feature matrices are then input into the Transformer networks trained in steps (2-1) and (2-2), respectively, to obtain sentence-level feature vectors for word-level interaction between the speech and text modalities. The speech and text modality feature matrices are then input into the corresponding Bi-LSTM networks in step (3-2), respectively, to obtain a 4×3 word-by-word sentiment prediction probability matrix for the speech modality and a 4×3 word-by-word sentiment prediction probability matrix for the text modality. The matrices are 4 rows and 3 columns, with each column representing a sentiment prediction probability vector. The word-by-word sentiment prediction probability matrices of the two modalities are weighted using steps (3-4) and (3-5) respectively, resulting in a 512-dimensional sentence-level feature vector weighted by word-by-word sentiment content for the speech modality and a 768-dimensional sentence-level feature vector weighted by word-by-word sentiment content for the text modality. The sentence-level feature vectors weighted by word-by-word sentiment content for the two modalities are then concatenated using step (3-5) to obtain a 1280-dimensional sentence-level feature vector weighted by word-by-word sentiment content for both the text and speech modalities. Step (4-1) yields a 2560-dimensional multimodal sentence-level feature vector containing multi-granularity information. This vector is then input into the fully connected network trained in step (4-1) to obtain the predicted sentiment index.

Claims

1. A multimodal sentiment analysis method based on word-level interaction and word-by-word sentiment weighting, characterized by: Preparation phase: 1) Text-to-speech dual-modal word-level feature extraction (1-1) Extracting word-level feature vectors of speech modalities: Mark the start and end times of each word in a sentence. While reading the audio waveform, read the start and end times of the words contained in the audio. The calculation formulas for the audio start frame SFrm and end frame EFrm are as follows: t=N÷fs (1) SFrm=ST÷t (2) EFrm=ET÷t (3) The t obtained from formula (1) is the audio frame length, representing the duration of one frame. N is the number of sampling points in one audio frame, fs is the audio sampling rate, ST is the start time of one word, and ET is the end time of the same word. The start frame SFrm and end frame EFrm of this word are calculated using formulas (1)(2)(3), thereby obtaining the accurate frame range occupied by each word in the audio within a sentence. The word-level audio data of all sentences in the dataset are calculated by applying the above steps to all sentences in the dataset. The audio waveforms corresponding to each word of a sentence are input into the Wav2vec 2.0 pre-trained model to extract features, resulting in word-by-word feature vectors for the sentence. Since a sentence contains M words, and the feature vectors extracted by Wav2vec 2.0 are L1-dimensional, the final result is a feature matrix of shape L1×M. Each column of the matrix represents a word-level feature vector of the speech modality, M is the number of words in the sentence, and L1 is the length of the feature vector. This process is repeated for all sentences in the dataset to obtain the complete dataset's word-level feature vectors of the speech modality and the speech modality feature matrix for each sentence. (1-2) Extracting text modality word-level feature vectors: Text feature extraction adopts word-level feature extraction; the sentences to be trained are input into the BERT pre-trained model, and the word-level feature vectors output by BERT are used as word-by-word feature vectors; a sentence contains M words, and BERT outputs a feature matrix of shape L2×M, where each column of the matrix is ​​a text modality word-level feature vector, the number of columns M is the number of words contained in the sentence, and the number of rows L2 is the length of the feature vector; the above operation is performed on all sentences in the dataset to obtain the text modality word-level feature vectors of the complete dataset and the text modality feature matrix of each sentence; 2) Multimodal temporal alignment and word-level granular multimodal fusion (2-1) Multimodal temporal alignment: The speech modal and text modal word-level feature vectors obtained in steps (1-1) and (1-2) are concatenated to obtain a text-speech bimodal word-level feature vector, which in turn yields the text-speech bimodal word-level feature matrix for each sentence; A sentence containing M words is given a speech modal word-level feature matrix of shape L1×M in step (1-1), with each column representing the speech modal feature vector of a word, and a text modal word-level feature matrix of shape L2×M in step (1-2), with each column representing the text modal feature vector of a word. The two feature matrices are concatenated to obtain a time-aligned speech-text bimodal word-level feature matrix of shape (L1+L2)×M, with each column representing a time-aligned text-speech bimodal word-level feature vector; The text-speech bimodal concatenation operation is performed on the feature matrices of all sentences in the dataset to obtain the time-aligned text-speech bimodal word-level feature matrix for each sentence; (2-2) Configure the Transformer network: The network input is the time-aligned text-speech bimodal word-level feature matrix of shape L×M obtained in step (2-1), where L=L1+L2; the model network consists of S Transformer encoders, each of which has an input and output of L×M. The last encoder layer is connected to the linear layer; the linear layer takes an L×M feature matrix as input and outputs an L-dimensional feature vector, which is the sentence-level feature vector of the text-speech bimodal word-level interaction. 3) Word-by-word sentiment weighting and sentence-level granular multimodal fusion (3-1) Configure the Bi-LSTM network: Define two Bi-LSTM networks with the same structure for the audio and text modalities respectively. Input the L1×M speech modal feature matrix extracted in step (1-1) into the speech modal Bi-LSTM network, and input the L2×M text modal feature matrix extracted in step (1-2) into the text modal Bi-LSTM network; the feature matrices are respectively passed into the encoders of the corresponding modal networks, and the number of RNN layers in the network is a set value; the encoder output is dropped out to approximate the output. Neurons are randomly pruned at a rate p, and this process is repeated more than 10 times during subsequent network training. The probability p that maximizes the accuracy on the test set is chosen as the Dropout probability. The pruned data is then input into a linear layer, and the output of the linear layer is the probability of each sentiment in the predicted feature vector. This probability is used as the output of the Bi-LSTM network. If a sentence contains M words and N sentiments are predicted, the network output is an N×M sentiment probability matrix. Each column of the matrix represents the N predicted sentiment probability vectors, M is the number of words in the sentence, and N is the number of predicted sentiments. (3-2) Input the speech modality feature matrix and the text modality feature matrix into the corresponding Bi-LSTM network of step (3-1) for training, and use the cross-entropy loss function as the model loss calculation function. During the training process, backpropagate the model according to the loss until the accuracy of the training set increases and the accuracy of the test set reaches the optimal value and no longer increases. By training the two modalities separately, the text modality Bi-LSTM model and the speech modality Bi-LSTM model are obtained. (3-3) Input the speech modality feature matrix into the speech modality Bi-LSTM model for inference to obtain the speech modality word-by-word sentiment prediction probability matrix; input the text modality feature matrix into the text modality Bi-LSTM model for inference to obtain the text modality word-by-word sentiment prediction probability matrix; the number of columns M of the probability matrix obtained from the two modalities is the number of words in the predicted sentence, the number of rows N of the matrix is ​​the number of predicted sentiments, and the columns of the matrix are composed of word-by-word sentiment prediction probability vectors. Each element in each column of the matrix corresponds to the probability of a predicted sentiment. The complete matrix represents that there are M words in this sentence, and each word contains N kinds of sentiments with different probabilities; the two modalities of each sentence in the dataset are passed through the corresponding modality Bi-LSTM model respectively, and a word-by-word sentiment prediction probability matrix of the corresponding modality is obtained for each sentence; (3-4) The variance of the word-by-word sentiment prediction probability vector in each row of the word-by-word sentiment prediction probability matrix obtained in step (3-3) is used as a measure to evaluate the word-by-word sentiment content. The specific formula for calculating the variance is as follows: Formula (4) is used to calculate the average modal sentiment prediction probability of a word. Each x in the molecule i Formula (5) is used to predict the probability of each emotion in the same modality for a word, with a total of N emotions predicted; Formula (5) is used to calculate the variance of the predicted emotion probability for a word in a modality. S is the average value of the same word and the same modality calculated in formula (4). 2 The variance of the sentiment prediction probability for this word in this modality; The set of all sentiment prediction probability variances for the same modality in a sentence is formed. All variances in the set are normalized so that the sum of the variances of all words equals 1. This yields the weight of the sentiment content contained in each word within the corresponding modality of the sentence. The weights obtained for each modality are sorted according to the order of the corresponding words and formed into a vector of length M for the sentiment content weights, denoted as vector A. The sentiment content weight vectors are calculated for the text and speech modalities of all sentences in the dataset. (3-5) Multiply the sentiment content weight vector A obtained in step (3-4) with the feature matrix of the corresponding sentence in the same modality from steps (1-1) and (1-2) to obtain the sentence-level feature vector of the sentence with word-by-word sentiment content weighted within a modality; in the speech modality, a sentence has M words, the speech modality feature matrix is ​​B, the shape of matrix B is L1×M, matrix B has L1 rows and M columns, the sentiment content weight vector is A, the shape of vector A is M×1, vector A has M rows and 1 columns; this sentence is weighted by sentiment content... The eigenvector C is B×A, and the shape of vector C is L1×1, with L1 rows and 1 column. For the text modality and speech modality of all sentences in the dataset, the sentence-level feature vectors weighted by word sentiment content are calculated separately. The sentence-level feature vectors weighted by word sentiment content of the speech modality and the text modality of the same sentence are concatenated to finally obtain the L-dimensional sentence-level feature vector of text and speech modality weighted by word sentiment content, L=L1+L2, where L1 is the length of the word-level feature vector of the speech modality and L2 is the length of the word-level feature vector of the text modality. 4) Training of multimodal, multigranular sentiment analysis models (4-1) Construction of a fully connected network: The fully connected network consists of two linear layers and one ReLU activation function. The L-dimensional text-speech bimodal word-level interaction sentence-level feature vector obtained in step (2-2) and the L-dimensional text-speech bimodal sentence-level feature vector weighted by word-by-word sentiment content obtained in step (3-5) are concatenated to obtain a 2L-dimensional multimodal sentence-level feature vector with multi-granularity information, which is used as the input of the fully connected network. The input length of the linear layer is the length of the multimodal sentence-level feature vector with multi-granularity information, and the output length is the same as the input length. The output of the linear layer is input into the linear correction unit activation function, and the ReLU output is input into the linear layer. The output length is the number of predicted sentiments N. The network output sentiment prediction probability vector is input into the softmax function. The Softmax function is shown in formula (6), a i or α j The position of the maximum value of the function output is the number of the predicted sentiment, which is the i-th or j-th element of the sentiment prediction probability vector. (4-2) Multi-network training: The networks involved in training include the Transformer encoder and linear layers in the Transformer network configured in step (2-2) and the fully connected network configured in step (4-1); all data in the dataset is divided into training set and test set, the training set is used to train the networks involved in training, and the test set is used to evaluate the network training performance; the cross-entropy loss function is used for network training loss function, as shown in formula (7): N is the length of the network's output sentiment prediction probability vector, i.e., the number of predicted sentiments, β i For sentiment prediction probability vector i, p(β) i Let q(β) be the true probability of the sentiment numbered i, expressed in one-hot form. i H(p,q) represents the predicted probability of sentiment numbered i. The smaller the cross-entropy H(p,q) value, the better the prediction result. Based on the calculated Loss, reverse transmission is performed to optimize the fully connected network and the Transformer network in turn. Training is stopped when the accuracy of the training set increases and the accuracy of the test set reaches the optimal value and no longer increases. This yields a complete model network with multimodal and multigranular sentiment analysis capabilities. Prediction phase: The complete model network with multimodal and multigranular sentiment analysis capabilities obtained through the above steps is used to obtain the speech modal word-level feature matrix and text modal word-level feature matrix for a new sentence through steps (1-1) and (1-2). The sentence-level feature vector of text-speech bimodal word-level interaction is obtained through steps (2-1) and (2-2). The sentence-level feature vector of text-speech bimodal sentence-level with word-by-word sentiment content weighted is obtained through steps (3-3), (3-4) and (3-5). Finally, the predicted sentiment label is obtained through step (4-1).

Images