A multi-modal sentiment analysis method based on modal feature enhancement

Abstract

The application relates to a multi-modal sentiment analysis method based on modal feature enhancement, which comprises the following steps: first, extracting image features and text features based on image enhancement; then, reducing the image and text feature dimensions through a tensor decomposition method, fusing the features of different modes, and then performing sentiment classification to obtain a relatively accurate sentiment classification result.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and more specifically, to a multimodal sentiment analysis method based on modal feature enhancement. Background Technology

[0002] With the rapid development of the internet and the emergence of big data, computers can use media such as text, images, audio, and video to identify the emotions expressed by people. In the era of big data, data means value. By conducting sentiment analysis on data resources on online platforms, their hidden value can be discovered and utilized, which is of great significance in many fields. Current technologies primarily analyze sentiment based on one modality, such as text, images, audio, or video. Single-modal sentiment analysis can only obtain sentiment information within that modality, and in some cases, it has many limitations. Summary of the Invention

[0003] To overcome at least one deficiency in the prior art, this application provides a multimodal sentiment analysis method based on modal feature enhancement.

[0004] Firstly, a multimodal sentiment analysis method based on modal feature enhancement is provided, including:

[0005] Acquire the image to be analyzed;

[0006] Extract image features from the image to be analyzed;

[0007] A word vector set is generated based on image features, containing word vectors corresponding to multiple words. The word vector set is then processed using Bi-LSTM to obtain a word embedding representation set that incorporates context, containing the word embedding representation for each word. This context-integrated word embedding representation set is then incorporated into word position vectors to generate a text sequence. This text sequence is input into a Transformer encoder to obtain an encoded text sequence, which includes the text embedding representation for each word. Attention coefficients for each word in the encoded text sequence are calculated, and the text embedding representations of all words are weighted and summed based on these attention coefficients to obtain the text features.

[0008] Tensor decomposition is used to fuse image features and text features to obtain fused features;

[0009] The fused features are used for sentiment classification using a multi-task learning method to obtain sentiment analysis results.

[0010] In one embodiment, extracting image features from the image to be analyzed includes:

[0011] The VGG19 model is used to process the image to be analyzed and obtain image features. The VGG19 model includes multiple convolutional pooling units, a first fully connected layer, and a second fully connected layer connected in sequence. Each convolutional pooling unit is used to perform convolution and pooling operations on the input.

[0012] In one embodiment, generating a set of word vectors based on image features includes:

[0013] The Word2Vec model is used to process the image features to obtain a set of word vectors; the set of word vectors includes the word vector corresponding to each word in the image features.

[0014] In one embodiment, a text sequence is input into a Transformer encoder to obtain an encoded text sequence, including:

[0015] Multi-head attention is performed on the text sequence to obtain the feature matrix L;

[0016] The feature matrix L and the text sequence are residually concatenated, and Layer Normalization is calculated to obtain the result.

[0017] The calculation results are subjected to two linear transformations and normalized to obtain the encoded text sequence, which includes the text embedding representation of each word.

[0018] In one embodiment, the attention coefficient for each word in the encoded text sequence is calculated using the following formula:

[0019]

[0020] Where ⊙ represents the dot product operation between vectors, a i Let E be the attention coefficient for the i-th word. I For image features, Let be the text embedding representation of the i-th word, and n be the number of words.

[0021] In one embodiment, tensor decomposition is used to fuse image features and text features to obtain fused features, using the following formula:

[0022] y=((T g ×(M T W m ))×(E I T W n ))×W o

[0023] Where y represents the fused feature, and T g Let M be the core tensor, and E be the text feature.I For image features, W m W n and W o These are the three factor matrices resulting from the three-dimensional tensor decomposition.

[0024] In one embodiment, the sentiment analysis results include sentiment classification results and emotion classification results.

[0025] Secondly, a multimodal sentiment analysis device based on modal feature enhancement is provided, comprising:

[0026] The image acquisition module is used to acquire the image to be analyzed.

[0027] The image feature extraction module is used to extract image features from the image to be analyzed.

[0028] The text feature extraction module generates a set of word vectors based on image features, which includes word vectors corresponding to multiple words. It then processes the word vector set using Bi-LSTM to obtain a set of word embeddings that incorporates context, including the word embedding representation for each word. This context-integrated word embedding representation set is then combined with word position vectors to generate a text sequence. The text sequence is input into a Transformer encoder to obtain an encoded text sequence, which includes the text embedding representation for each word. Finally, the module calculates the attention coefficient for each word in the encoded text sequence and performs a weighted sum of the text embedding representations for all words based on the attention coefficient for each word to obtain the text features.

[0029] The feature fusion module is used to fuse image features and text features using tensor decomposition to obtain fused features;

[0030] The sentiment classification module is used to classify the fused features using a multi-task learning method to obtain sentiment analysis results.

[0031] Thirdly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the aforementioned multimodal sentiment analysis method based on modal feature enhancement.

[0032] Fourthly, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements the aforementioned multimodal sentiment analysis method based on modal feature enhancement.

[0033] Compared with the prior art, this application has the following beneficial effects: The multimodal sentiment analysis method based on modal feature enhancement in this application first extracts image features and text features based on image enhancement, and then reduces the dimensionality of image and text features by tensor decomposition, integrates features of different modalities, and then performs sentiment classification to obtain a more accurate sentiment classification result. Attached Figure Description

[0034] This application can be better understood by referring to the description given below in conjunction with the accompanying drawings, which, together with the detailed description below, are incorporated in and form part of this specification. In the drawings:

[0035] Figure 1 A flowchart of a multimodal sentiment analysis method based on modal feature enhancement according to an embodiment of this application is shown. Detailed Implementation

[0036] Exemplary embodiments of the present application will be described below with reference to the accompanying drawings. For clarity and brevity, not all features of the actual embodiments are described in the specification. However, it should be understood that many embodiment-specific decisions can be made in the development of any such actual embodiment to achieve the developer’s specific objectives, and these decisions may vary as the embodiments differ.

[0037] It should also be noted that, in order to avoid obscuring this application with unnecessary details, only the device structure closely related to the solution according to this application is shown in the accompanying drawings, while other details that are not closely related to this application are omitted.

[0038] It should be understood that this application is not limited to the described embodiments by virtue of the following description with reference to the accompanying drawings. In this document, embodiments may be combined with each other, features may be substituted or borrowed between different embodiments, and one or more features may be omitted in one embodiment, where feasible.

[0039] This application provides a multimodal sentiment analysis method based on modal feature enhancement. Figure 1 A flowchart of a multimodal sentiment analysis method based on modal feature enhancement according to an embodiment of this application is shown. See also: Figure 1 The methods include:

[0040] Step S1: Obtain the image to be analyzed; here, the image to be analyzed may be, for example, an image frame in a video, including people and text.

[0041] Step S2: Extract image features from the image to be analyzed.

[0042] Specifically, the improved VGG19 model can be used to process the image to be analyzed and obtain image features. The VGG19 model includes multiple convolutional pooling units, a first fully connected layer, and a second fully connected layer connected in sequence. Each convolutional pooling unit is used to perform convolution and pooling operations on the input.

[0043] Here, the VGG19 model includes 5 convolutional pooling units, each with a 3×3×n kernel size, where n represents the depth of the feature matrix. The ReLU activation function is used to increase the model's non-linearity, and max pooling is used for feature selection. Furthermore, the structure of the VGG19 model is fine-tuned. The convolutional layer parameters are fixed, and a fully connected layer with 300 neurons is added outside the last fully connected layer to extract image feature vectors of a specified dimension. Local features of the image are extracted by performing convolution operations on the image's representation matrix using the convolution kernel. The convolution kernel is then moved with a stride until the entire image is processed, resulting in a feature map of the image based on that convolution kernel, i.e., image features. These image features are digital representations of images including people and text.

[0044] Step S3: Generate a word vector set based on image features, which includes word vectors corresponding to multiple words; process the word vector set based on Bi-LSTM to obtain a word embedding representation set that incorporates context, which includes the word embedding representation of each word; integrate the word embedding representation set with context into the word position vector to generate a text sequence; input the text sequence into a Transformer encoder to obtain an encoded text sequence, which includes the text embedding representation of each word; calculate the attention coefficient of each word in the encoded text sequence, and perform a weighted sum of the text embedding representations of all words based on the attention coefficient of each word to obtain the text features.

[0045] Specifically, the Word2Vec model is used to process image features, resulting in a word vector set. This set includes the word vector corresponding to each word in the image features. Here, the model is trained using sentence text from a corpus. The trained model can map each word in the corpus to a vector in the word vector space. The words from the image features constitute the initial text sequence T = {t1, t2, ..., t...}. n}, where t i Let t be the i-th word in the initial text sequence, and n be the number of words. This is obtained using Word2Vec. i word vector ω i ∈R n Let n be the dimension of the vector, then the set of word vectors for the entire initial text sequence can be represented as ω = {ω1, ω2, ..., ωn}. n}

[0046] Specifically, by incorporating the set of word embeddings that integrate context into word position vectors to generate a text sequence, the following formula can be used:

[0047] K = F + PE

[0048] Where K is the text sequence, F is the set of word embeddings that incorporate context, and PE is the word position vector, which represents the position of each word and is a predefined vector.

[0049] Step S4: The image features and text features are fused using tensor decomposition to obtain the fused features;

[0050] Step S5 involves performing sentiment classification on the fused features using a multi-task learning method to obtain the sentiment analysis results. Here, a PCGrad-based multi-task learning method can be employed. Multi-task learning achieves information sharing by sharing representation layer parameters, thereby simultaneously improving the performance of multiple tasks. This paper uses hard parameter sharing, sharing most of the hidden layer parameters in the model, while retaining only the hidden layer parameters of the corresponding task layers as independent parameters.

[0051] The multimodal sentiment analysis method based on modal feature enhancement in this embodiment first extracts image features and text features based on image enhancement. Then, it reduces the dimensionality of image and text features by tensor decomposition, fuses features from different modalities, and then performs sentiment classification to obtain a more accurate sentiment classification result.

[0052] In one embodiment, a text sequence is input into a Transformer encoder to obtain an encoded text sequence, including:

[0053] Multi-head attention is performed on the text sequence to obtain the feature matrix L;

[0054] The feature matrix L and the text sequence K are residually concatenated, and Layer Normalization is calculated to obtain the result M; this can be expressed by the following formula: M = norm(K + L)

[0055] Here, norm represents Layer Normalizition calculation.

[0056] The calculation results are subjected to two linear transformations and normalized to obtain the encoded text sequence. The encoded text sequence includes the text embedding representation of each word, which can be expressed by the following formula:

[0057] F(M) = max(0, MW1 + b1)W2 + b2

[0058] Where F(M) is the encoded text sequence, W1 and W2 are the weights of the fully connected layer, and b1 and b2 are the bias terms corresponding to W1 and W2. Here, the fully connected layer is used to perform linear transformation.

[0059] In this embodiment, the Transformer encoder includes multiple encoding units, for example, six. Each encoding unit includes a self-attention layer and a feedforward neural network layer for multi-head attention calculation. The text sequence is input to multiple encoding units connected in sequence for multi-head attention calculation, and the last encoding unit outputs the feature matrix L.

[0060] In one embodiment, the attention coefficient for each word in the encoded text sequence is calculated using the following formula:

[0061]

[0062] Where ⊙ represents the dot product operation between vectors, a i Let E be the attention coefficient for the i-th word. I For image features, Let be the text embedding representation of the i-th word, and n be the number of words.

[0063] Here, the similarity between the text embedding representation of a word and its image embedding representation is calculated, and words that are similar to the image are assigned greater weight.

[0064] In one embodiment, tensor decomposition is used to fuse image features and text features to obtain fused features, using the following formula:

[0065] y=((T g ×(M T W m ))×(E I T W n ))×W o

[0066] Where y represents the fused feature, and T g Let M be the core tensor, and E be the text feature. I For image features, W m W n and W o These are the three factor matrices resulting from the three-dimensional tensor decomposition.

[0067] Here, T g W m W n and W o The result is obtained by decomposing a 3D tensor T using a tensor decomposition method based on t-SVD. The 3D tensor T is compared with text features M and image features E.I The dimensions remain consistent.

[0068] In one embodiment, a multi-task learning method is used to classify the fused features to obtain sentiment analysis results. These results include both sentiment classification and emotion classification. Here, emotion classification refers to the emotion category to which the input image belongs, which can include happiness, sadness, anger, fear, disgust, surprise, etc. The sentiment classification result is used to represent the accuracy of the emotion classification.

[0069] Here, the multi-task learning method can be a PCGrad-based multi-task learning method, which is based on a classifier that includes a fully connected layer, a sigmoid layer, and a softmax layer.

[0070] In multi-task learning, the PCGrad method is introduced to resolve conflicts in gradient updates between multiple tasks, which can lead to oscillations and negative transfer in model parameter updates. For example, if gradient updates from two tasks conflict, the gradient of the other task is projected onto the normal vector of the other task to eliminate the conflict. When there is no conflict between the gradients of the two tasks, updates proceed normally.

[0071] Employing the same inventive concept as the modal feature enhancement-based multimodal sentiment analysis method, this embodiment also provides a corresponding modal feature enhancement-based multimodal sentiment analysis device, including:

[0072] The image acquisition module is used to acquire the image to be analyzed.

[0073] The image feature extraction module is used to extract image features from the image to be analyzed.

[0074] The text feature extraction module generates a set of word vectors based on image features, which includes word vectors corresponding to multiple words. It then processes the word vector set using Bi-LSTM to obtain a set of word embeddings that incorporates context, including the word embedding representation for each word. This context-integrated word embedding representation set is then combined with word position vectors to generate a text sequence. The text sequence is input into a Transformer encoder to obtain an encoded text sequence, which includes the text embedding representation for each word. Finally, the module calculates the attention coefficient for each word in the encoded text sequence and performs a weighted sum of the text embedding representations for all words based on the attention coefficient for each word to obtain the text features.

[0075] The feature fusion module is used to fuse image features and text features using tensor decomposition to obtain fused features;

[0076] The sentiment classification module is used to classify the fused features using a multi-task learning method to obtain sentiment analysis results.

[0077] The multimodal sentiment analysis device based on modal feature enhancement in this embodiment has the same inventive concept as the multimodal sentiment analysis method based on modal feature enhancement described above. Therefore, the specific implementation of this device can be found in the embodiment section of the multimodal sentiment analysis method based on modal feature enhancement described above, and its technical effects correspond to the technical effects of the above method, so it will not be repeated here.

[0078] This application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the above-described multimodal sentiment analysis method based on modal feature enhancement.

[0079] This application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the above-described multimodal sentiment analysis method based on modal feature enhancement.

[0080] The above descriptions are merely various embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multimodal sentiment analysis method based on modal feature enhancement, characterized in that, include: Acquire the image to be analyzed; Extract the image features of the image to be analyzed; A set of word vectors is generated based on the image features, and the set of word vectors includes word vectors corresponding to multiple words; The word vector set is processed using Bi-LSTM to obtain a word embedding representation set that incorporates context, wherein the word embedding representation set includes the word embedding representation of each word; the word embedding representation set that incorporates context is then integrated with the word position vector to generate a text sequence; the text sequence is then input into a Transformer encoder to obtain an encoded text sequence, wherein the encoded text sequence includes the text embedding representation of each word; Calculate the attention coefficient of each word in the encoded text sequence, and perform a weighted sum of the text embedding representations of all words based on the attention coefficient of each word to obtain the text features; The image features and text features are fused using tensor decomposition to obtain the fused features; The fused features are then used for sentiment classification using a multi-task learning method to obtain the sentiment analysis results.

2. The method as described in claim 1, characterized in that, in, Extracting image features from the image to be analyzed includes: The image to be analyzed is processed using the VGG19 model to obtain image features; the VGG19 model includes multiple convolutional pooling units, a first fully connected layer and a second fully connected layer connected in sequence; each convolutional pooling unit is used to perform convolution and pooling operations on the input.

3. The method as described in claim 1, characterized in that, in, Generate a set of word vectors based on the image features, including: The image features are processed using the Word2Vec model to obtain a set of word vectors; the set of word vectors includes the word vector corresponding to each word in the image features.

4. The method as described in claim 1, characterized in that, in, The text sequence is input into the Transformer encoder to obtain the encoded text sequence, including: Multi-head attention is performed on the text sequence to obtain the feature matrix L; The feature matrix L and the text sequence are residually concatenated, and Layer Normalization is calculated to obtain the result. The calculation results are subjected to two linear transformations and normalized to obtain an encoded text sequence, which includes the text embedding representation of each word.

5. The method as described in claim 1, characterized in that, in, The attention coefficient for each word in the encoded text sequence is calculated using the following formula: Where ⊙ represents the dot product operation between vectors, a i Let E be the attention coefficient for the i-th word. I For image features, Let be the text embedding representation of the i-th word, and n be the number of words.

6. The method as described in claim 1, characterized in that, in, The image features and text features are fused using tensor decomposition to obtain the fused features, as shown in the following formula: y=((T g ×(M T W m ))×(E I T W n ))×W o Where y represents the fused feature, and T g Let M be the core tensor, and E be the text feature. I For image features, W m W n and W o These are the three factor matrices resulting from the three-dimensional tensor decomposition.

7. The method as described in claim 1, characterized in that, The sentiment analysis results include sentiment classification results and emotion classification results.

8. A multimodal sentiment analysis device based on modal feature enhancement, characterized in that, include: The image acquisition module is used to acquire the image to be analyzed. The image feature extraction module is used to extract image features from the image to be analyzed. The text feature extraction module is used to generate a set of word vectors based on the image features, wherein the set of word vectors includes word vectors corresponding to multiple words; The word vector set is processed using Bi-LSTM to obtain a word embedding representation set that incorporates context, wherein the word embedding representation set includes the word embedding representation of each word; the word embedding representation set that incorporates context is then integrated with the word position vector to generate a text sequence; the text sequence is then input into a Transformer encoder to obtain an encoded text sequence, wherein the encoded text sequence includes the text embedding representation of each word; Calculate the attention coefficient of each word in the encoded text sequence, and perform a weighted sum of the text embedding representations of all words based on the attention coefficient of each word to obtain the text features; The feature fusion module is used to fuse the image features and the text features using tensor decomposition to obtain fused features; The sentiment classification module is used to classify the fused features using a multi-task learning method to obtain sentiment analysis results.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the multimodal sentiment analysis method based on modal feature enhancement as described in any one of claims 1-7.

10. A computer program product, characterized in that, Includes a computer program / instruction, which, when executed by a processor, implements the multimodal sentiment analysis method based on modal feature enhancement as described in any one of claims 1-7.

Images