Multimodal sentiment analysis method, apparatus and electronic device

Abstract

The application relates to a multi-modal sentiment analysis method, comprising the steps of: performing feature extraction on image data, text data and voice data in video data to be analyzed to obtain image features, text features and voice features; performing feature enhancement on the image features and the voice features according to the text features to obtain image enhanced features and voice enhanced features; performing cross-modal attention interaction on the image enhanced features, the voice enhanced features and the text features respectively to obtain image-text interaction features and voice-text interaction features; performing feature extraction based on a multi-head self-attention mechanism on the image features, the text features and the voice features respectively to obtain image self-attention features, text self-attention features and voice self-attention features; fusing the text self-attention features, the image self-attention features, the voice self-attention features, the image-text interaction features and the voice-text interaction features to obtain multi-modal video features; and performing sentiment classification prediction on the multi-modal video features to obtain a sentiment classification result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal sentiment analysis, in particular to a multi-modal sentiment analysis method, device and electronic equipment. BACKGROUND

[0002] With the development of the Internet, more and more users express their opinions on social platforms with videos. Sentiment analysis on these video information can quickly capture the emotional tendencies expressed by users. Based on the results of sentiment analysis, automatic decision support systems can be developed, such as automatic customer service systems, automatic marketing strategy generation systems, etc. The amount of information in video data is huge, including voice information (such as sound expression, sound size, tone) and image information (such as people's facial expressions, body movements) and text information. Traditional text-based single-modal sentiment analysis cannot effectively analyze the sentiment of video data, and multi-modal sentiment analysis is needed to predict the sentiment of video data. However, the current multi-modal sentiment analysis method has many noise interferences, resulting in low sentiment analysis accuracy. SUMMARY

[0003] The present application aims to overcome the shortcomings and deficiencies of the prior art, and provides a multi-modal sentiment analysis method that can improve sentiment analysis accuracy.

[0004] The present application is implemented by the following technical scheme: a multi-modal sentiment analysis method, comprising the steps of:

[0005] Obtaining image data in the video data to be analyzed, performing feature extraction on the image data to obtain image features;

[0006] Obtaining text data in the video data to be analyzed, performing feature extraction on the text data to obtain text features;

[0007] Obtaining voice data in the video data to be analyzed, performing feature extraction on the voice data to obtain voice features;

[0008] Performing feature enhancement on the image features according to the text features to obtain image enhanced features;

[0009] Performing feature enhancement on the voice features according to the text features to obtain voice enhanced features;

[0010] Performing cross-modal attention interaction on the image enhanced features and the text features to obtain image-text interaction features;

[0011] Performing cross-modal attention interaction on the voice enhanced features and the text features to obtain voice-text interaction features;

[0012] perform feature extraction on the image features based on a multi-head self-attention mechanism to obtain image self-attention features;

[0013] perform feature extraction on the text features based on a multi-head self-attention mechanism to obtain text self-attention features;

[0014] perform feature extraction on the speech features based on a multi-head self-attention mechanism to obtain speech self-attention features;

[0015] fuse the text self-attention features, the image self-attention features, the speech self-attention features, the image-text interaction features, and the audio-text interaction features to obtain multi-modal video features;

[0016] perform emotion classification prediction on the multi-modal video features to obtain an emotion classification result.

[0017] Compared with the prior art, the multi-modal emotion analysis method can identify information with a large contribution to emotion classification within each modality by further performing self-attention feature extraction on text features, image features, and speech features, thereby reducing external information dependence and noise interference. Meanwhile, the image enhancement features and the speech enhancement features are fused with the text features through cross-modal attention, which can increase the interaction and feature complementarity between the text modality and the image modality and the speech modality. Thus, the text self-attention features, the image self-attention features, the speech self-attention features, the image-text interaction features, and the audio-text interaction features are fused to obtain multi-modal video features for classification prediction, which is beneficial to improving the accuracy of emotion classification prediction.

[0018] Further, the image features are enhanced based on the text features to obtain image enhancement features, including the steps of:

[0019] performing first self-attention encoding on the text features to obtain a first text encoding vector;

[0020] performing multi-modal attention decoding on the first text encoding vector and the image features to obtain image enhancement features.

[0021] Further, the speech features are enhanced based on the text features to obtain speech enhancement features, including the steps of:

[0022] performing second self-attention encoding on the text features to obtain a second text encoding vector;

[0023] performing multi-modal attention decoding on the second text encoding vector and the speech features to obtain speech enhancement features.

[0024] Further, the text features are secondly self-attention encoded by N first encoders connected in a chain structure, the first encoders are used for multi-head attention calculation with the text features or output data of a previous first encoder as query Q, key K and value V respectively, and the calculation results are residual connected with the text features or output data of the previous first encoder and normalized to obtain first multi-head attention; the first multi-head attention is processed by a feedforward neural network, and the processing results are residual connected with the first multi-head attention and normalized to obtain output data of the first encoder;

[0025] The first text encoding vector and the image features are multimodal attention decoded by N image-text decoders connected in a chain structure, the image-text decoders are used for multi-head attention calculation with the first text encoding vector as query Q and key K, and the image features or output data of a previous image-text decoder as value V, and the calculation results are residual connected with the image features or output data of the previous image-text decoder and normalized to obtain image-text multi-head attention; the image-text multi-head attention is processed by a feedforward neural network, and the processing results are residual connected with the image-text multi-head attention and normalized to obtain output data of the image-text encoder.

[0026] Further, the text features are firstly self-attention encoded by N second encoders connected in a chain structure, the second encoders are used for multi-head attention calculation with the text features or output data of a previous second encoder as query Q, key K and value V respectively, and the calculation results are residual connected with the text features or output data of the previous second encoder and normalized to obtain second multi-head attention; the second multi-head attention is processed by a feedforward neural network, and the processing results are residual connected with the second multi-head attention and normalized to obtain output data of the second encoder;

[0027] The second text encoding vector and the speech features are multimodal attention decoded by N audio-text decoders connected in a chain structure, the audio-text decoders are used for multi-head attention calculation with the second text encoding vector as query Q and key K, and the speech features or output data of a previous audio-text decoder as value V, and the calculation results are residual connected with the speech features or output data of the previous audio-text decoder and normalized to obtain audio-text multi-head attention; the audio-text multi-head attention is processed by a feedforward neural network, and the processing results are residual connected with the audio-text multi-head attention and normalized to obtain output data of the audio-text encoder.

[0028] Further, the image enhancement feature and the text feature are subjected to cross-modal attention interaction, including the steps of: taking the image enhancement feature as a query Q, taking the text feature as a key K and a value V to perform multi-head attention calculation to obtain a picture-text interaction feature.

[0029] Further, the speech enhancement feature and the text feature are subjected to cross-modal attention interaction, including the steps of: taking the speech enhancement feature as a query Q, taking the text feature as a key K and a value V to perform multi-head attention calculation to obtain a sound-text interaction feature.

[0030] Further, the text self-attention feature, the image self-attention feature, the speech self-attention feature, the picture-text interaction feature and the sound-text interaction feature are fused, including the steps of:

[0031] After the text self-attention feature, the image self-attention feature, the speech self-attention feature, the picture-text interaction feature and the sound-text interaction feature are spliced, linear transformation is performed, and the linear transformation result is activated to obtain a multi-modal video feature.

[0032] Based on the same inventive concept, the present application also provides a multi-modal sentiment analysis device, comprising:

[0033] An image feature extraction module is configured to acquire image data in the video data to be analyzed, and extract features from the image data to obtain image features.

[0034] A text feature extraction module is configured to acquire text data in the video data to be analyzed, and extract features from the text data to obtain text features.

[0035] A speech feature extraction module is configured to acquire speech data in the video data to be analyzed, and extract features from the speech data to obtain speech features.

[0036] An image feature enhancement module is configured to enhance features of the image features based on the text features to obtain image enhancement features.

[0037] A speech feature enhancement module is configured to enhance features of the speech features based on the text features to obtain speech enhancement features.

[0038] A picture-text feature interaction module is configured to subject the image enhancement features and the text features to cross-modal attention interaction to obtain picture-text interaction features.

[0039] A sound-text feature interaction module is configured to subject the speech enhancement features and the text features to cross-modal attention interaction to obtain sound-text interaction features.

[0040] an image self-attention module configured to perform feature extraction on the image feature based on a multi-head self-attention mechanism to obtain an image self-attention feature;

[0041] a text self-attention module configured to perform feature extraction on the text feature based on a multi-head self-attention mechanism to obtain a text self-attention feature;

[0042] a speech self-attention module configured to perform feature extraction on the speech feature based on a multi-head self-attention mechanism to obtain a speech self-attention feature;

[0043] a feature fusion module configured to fuse the text self-attention feature, the image self-attention feature, the speech self-attention feature, the image-text interaction feature and the speech-text interaction feature to obtain a multi-modal video feature;

[0044] a sentiment classification module configured to perform sentiment classification prediction on the multi-modal video feature to obtain a sentiment classification result.

[0045] Based on the same inventive concept, the present application also provides an electronic device comprising:

[0046] a processor;

[0047] a memory configured to store a computer program executed by the processor;

[0048] wherein the processor implements the steps of the above method when executing the computer program.

[0049] For better understanding and implementation, the present application will be described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 a schematic diagram of an exemplary application environment of a multi-modal sentiment analysis method according to an embodiment;

[0051] Figure 2 a schematic diagram of a flow of a multi-modal sentiment analysis method according to an embodiment;

[0052] Figure 3 a schematic diagram of a multi-modal sentiment analysis device for executing the multi-modal sentiment analysis method shown. Figure 2 DETAILED DESCRIPTION

[0053] To make the purposes, technical solutions and advantages of the present application clearer, the embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

[0054] Please refer to Figure 1 ​It is an exemplary application environment schematic diagram of a multimodal sentiment analysis method according to the embodiment, which comprises a camera 1, a voice collection device 2, a user terminal 3 and a server 4. The camera 1 can be a camera or a camera built in the user terminal 3. The voice collection device 2 is used for collecting sound signals, which can be a microphone, a voice recorder, etc. or a microphone built in the user terminal 3. The user terminal 3 can be any smart terminal with Internet function, such as a computer, a mobile phone, a tablet computer, a PDA (Personal Digital Assistant), an e-book reader, a multimedia player, etc. The server 4 can be a computer or a special server. The user terminal 3 can access a router through a wireless local area network, and access the server 4 on the public network through the router. The user records the video data through the camera 1 and the voice collection device 2, and transmits the recorded video data to the server 4 through the user terminal 3. The server 4 processes the video data through the multimodal sentiment analysis method of the application to obtain the sentiment analysis result, and returns the sentiment analysis result to the user terminal 3.

[0055] Please refer to Figure 2 and Figure 3 wherein, Figure 2 is a flow schematic diagram of the multimodal sentiment analysis method of the embodiment; Figure 3 is a structural schematic diagram of a multimodal sentiment analysis device for executing the multimodal sentiment analysis method of the embodiment. The multimodal sentiment analysis device comprises an image feature extraction module 10a, a text feature extraction module 10b, a voice feature extraction module 10c, an image feature enhancement module 20a, a voice feature enhancement module 20b, an image-text feature interaction module 30a, an audio-text feature interaction module 30b, an image self-attention module 40a, a text self-attention module 40b, a voice self-attention module 40c, a feature fusion module 50 and a sentiment classification module 60.

[0056] Specifically, the image feature extraction module 10a is used to execute step S1a: obtaining the image data in the video data to be analyzed, performing feature extraction on the image data to obtain the image features.

[0057] The image data is the image presented by the video data to be analyzed in the video player, which can be directly extracted from the video data to be analyzed.

[0058] For image data, feature information can be extracted by a neural network, and in this embodiment, a temporal convolutional network (TCN) is selected to extract features from image data to obtain image features. The temporal convolutional network extracts the context temporal features of the image data through causal convolution, dilated convolution and residual connection, and then processes the context temporal features of the image data through a fully connected layer to output image features, denoted as wherein, is a vector representation of the image data, is a network parameter of the temporal convolutional network for extracting image features, is a feature dimension.

[0059] The text feature extraction module 10b is configured to perform step S1b: obtaining text data in the video data to be analyzed, and extracting features from the text data to obtain text features.

[0060] The text data is the text in the voice in the video data to be analyzed, and can be obtained by transcribing the voice data in the video data to be analyzed.

[0061] For text data, feature information can be extracted by a natural language processing model, and in this embodiment, a pre-trained BERT model is selected to extract features from text data to obtain text features. The BERT model selected is a 12-layer bert-base version, and the hidden layer dimension is 768. The output of the last hidden layer of the BERT model is used as the text feature, denoted as wherein, is a vector representation of the text data, is a network parameter of the BERT model for extracting text features, is a feature dimension.

[0062] The voice feature extraction module 10c is configured to perform step S1c: obtaining voice data in the video data to be analyzed, and extracting features from the voice data to obtain voice features.

[0063] The voice data is the sound when the video data to be analyzed is played by a video player, and can be directly extracted from the video data to be analyzed.

[0064] For voice data, feature information can be extracted by a neural network, and in this embodiment, a temporal convolutional network is selected to extract features from voice data to obtain voice features. The temporal convolutional network extracts the context temporal features of the voice data through causal convolution, dilated convolution and residual connection, and then processes the context temporal features of the voice data through a fully connected layer to output voice features, denoted as wherein, is a vector representation of the speech data, is a network parameter of a time convolutional neural network for extracting the speech feature, is a feature dimension.

[0065] The image feature enhancement module 20a is configured to perform step S2a: performing feature enhancement on the image feature according to the text feature to obtain an image enhanced feature.

[0066] wherein, the feature enhancement on the image feature according to the text feature is to organically fuse the text feature and the image feature, to enrich the image feature by repeated low-level features in the text form, and to obtain the image enhanced feature. Specifically, the image feature enhancement module 20a includes a first text encoding submodule 20a1 and a text-image decoding submodule 20a2.

[0067] The first text encoding submodule 20a1 is configured to perform step S2a1: performing first self-attention encoding on the text feature to obtain a first text encoding vector.

[0068] wherein, the text feature is input into a first one of N first encoders connected in a chain structure, and the input data of the second to the last first encoders is the output data of the previous first encoder, and the output of the last first encoder is the first text encoding vector.

[0069] The processing steps of each first encoder on its input data include: performing multi-head attention calculation on the text feature or the output data of the previous first encoder as query Q (Query), key K (Key) and value V (Value) respectively, and performing residual connection and normalization on the calculation result and the input data to obtain first multi-head attention; performing feedforward neural network processing on the first multi-head attention, and performing residual connection and normalization on the processing result and the first multi-head attention to obtain the output data of the first encoder.

[0070] The text-image decoding submodule 20a2 is configured to perform step S2a2: performing multi-modal attention decoding on the first text encoding vector and the image feature to obtain the image enhanced feature.

[0071] wherein, the first text encoding vector and the image feature are input into each text-image decoder by N text-image decoders connected in a chain structure, and the image feature is input into a first one of the text-image decoders, and the input data of the second to the last text-image decoders is the output data of the previous text-image decoder, and the output data of the last text-image decoder is the image enhanced feature.

[0072] The processing of each image-text decoder on its input data includes the steps of: performing multi-head attention calculation on the first text encoding vector as query Q and key K respectively, and on the image feature or the output data of the previous image-text decoder as value V, and performing residual connection and normalization on the calculation result and the image feature or the output data of the previous image-text decoder to obtain image-text multi-head attention; performing feedforward neural network processing on the image-text multi-head attention, and performing residual connection and normalization on the processing result and the image-text multi-head attention to obtain the output data of the image-text encoder.

[0073] The speech feature enhancement module 20b is configured to perform step S2b: performing feature enhancement on the speech feature according to the text feature to obtain speech enhanced feature.

[0074] The speech feature enhancement module 20b is configured to perform step S2b: performing feature enhancement on the speech feature according to the text feature to obtain speech enhanced feature.

[0075] The second text encoding submodule 20b1 is configured to perform step S2b1: performing second self-attention encoding on the text feature to obtain a second text encoding vector.

[0076] The second text encoding submodule 20b1 is configured to perform step S2b1: performing second self-attention encoding on the text feature to obtain a second text encoding vector.

[0077] The processing of each second encoder on its input data includes the steps of: performing multi-head attention calculation on the text feature or the output data of the previous second encoder as query Q, key K and value V respectively, and performing residual connection and normalization on the calculation result and the input data to obtain second multi-head attention; performing feedforward neural network processing on the second multi-head attention, and performing residual connection and normalization on the processing result and the second multi-head attention to obtain the output data of the second encoder.

[0078] The audio-text decoding submodule 20b2 is configured to perform step S2b2: performing multi-modal attention decoding on the second text encoding vector and the speech feature to obtain speech enhanced feature.

[0079] The second text encoding vector is input into each audio-text decoder, the speech feature is input into the first audio-text decoder, the output data of the first to the second last audio-text decoder is input into the subsequent audio-text decoder, and the output data of the last audio-text decoder is the speech enhancement feature.

[0080] The processing of each audio-text decoder on the input data includes the steps of: taking the second text encoding vector as a query Q and a key K respectively, taking the speech feature or the output data of the previous audio-text decoder as a value V to perform multi-head attention calculation, and performing residual connection and normalization on the calculation result and the speech feature or the output data of the previous audio-text decoder to obtain audio-text multi-head attention; performing feedforward neural network processing on the audio-text multi-head attention, and performing residual connection and normalization on the processing result and the audio-text multi-head attention to obtain the output data of the audio-text encoder.

[0081] The image-text feature interaction module 30a is configured to perform step S3a: performing cross-modal attention interaction on the image enhancement feature and the text feature to obtain image-text interaction features.

[0082] The cross-modal attention interaction on the image enhancement feature and the text feature is performed by taking the image enhancement feature as a query Q, taking the text feature as a key K and a value V to perform multi-head attention calculation, and obtaining image-text interaction features The expression of the image-text interaction features is as follows:

[0083]

[0084] wherein, is a linear transformation matrix; is an attention result calculated by m attention heads, and the attention calculation expression of the hth attention head is as follows:

[0085]

[0086] wherein, is the image enhancement feature, is a linear transformation matrix of the query, is a linear transformation matrix of the key, is a linear transformation matrix of the value, is a vector dimension.

[0087] The audio-text feature interaction module 30b is configured to perform step S3b: performing cross-modal attention interaction on the speech enhancement feature and the text feature to obtain audio-text interaction features.

[0088] Wherein, when the speech enhancement feature and the text feature are cross-modal attention interaction, the speech enhancement feature is taken as a query Q, and the text feature is taken as a key K and a value V to perform multi-head attention calculation to obtain an audio-text interaction feature , the expression of the audio-text interaction feature is as follows:

[0089]

[0090] Wherein, is a linear transformation matrix; is an attention result calculated by m attention heads, wherein the attention calculation expression of the hth attention head is as follows:

[0091]

[0092] Wherein, is a linear transformation matrix of the query, is a linear transformation matrix of the key, is a linear transformation matrix of the value, is a vector dimension.

[0093] The image self-attention module 40a is configured to perform step S4a: performing feature extraction on the image feature based on a multi-head self-attention mechanism to obtain an image self-attention feature.

[0094] Wherein, when the image feature is extracted based on the multi-head self-attention mechanism, the image feature is taken as a query Q, a key K and a value V to perform multi-head attention calculation to obtain an image self-attention feature , the expression of the image self-attention feature is as follows:

[0095]

[0096] Wherein, is a linear transformation matrix; is an attention result calculated by m attention heads, wherein the attention calculation expression of the hth attention head is as follows:

[0097]

[0098] Wherein, is a linear transformation matrix of the query, is a linear transformation matrix of the key, is a linear transformation matrix of the value, is a vector dimension.

[0099] The text self-attention module 40b is configured to perform step S4b: performing feature extraction on the text feature based on a multi-head self-attention mechanism to obtain a text self-attention feature.

[0100] wherein, when the text feature is subjected to the feature extraction based on the multi-head self-attention mechanism, the text feature is taken as the query Q, the key K and the value V to perform the multi-head attention calculation, to obtain a text self-attention feature , and an expression of the text self-attention feature is as follows:

[0101]

[0102] wherein, is a linear transformation matrix; is an attention result calculated by m attention heads, wherein an attention calculation expression of the hth attention head is as follows:

[0103]

[0104] wherein, is a linear transformation matrix of the query, is a linear transformation matrix of the key, is a linear transformation matrix of the value, is a vector dimension.

[0105] The speech self-attention module 40c is configured to perform step S4c: performing feature extraction based on a multi-head self-attention mechanism on the speech feature to obtain a speech self-attention feature.

[0106] wherein, when the text feature is subjected to the feature extraction based on the multi-head self-attention mechanism, the text feature is taken as the query Q, the key K and the value V to perform the multi-head attention calculation, to obtain a text self-attention feature , and an expression of the text self-attention feature is as follows:

[0107]

[0108] wherein, is a linear transformation matrix; is an attention result calculated by m attention heads, wherein an attention calculation expression of the hth attention head is as follows:

[0109]

[0110] wherein, is a linear transformation matrix of the query, is a linear transformation matrix of the key, is a linear transformation matrix of the value, is a vector dimension.

[0111] The feature fusion module 50 is configured to perform step S5 of fusing the text self-attention feature, the image self-attention feature, the speech self-attention feature, the image-text interaction feature, and the audio-text interaction feature to obtain a multi-modal video feature.

[0112] In the fusion of the text self-attention feature, the image self-attention feature, the speech self-attention feature, the image-text interaction feature, and the audio-text interaction feature, the text self-attention feature, the image self-attention feature, the speech self-attention feature, the image-text interaction feature, and the audio-text interaction feature are spliced, then linearly transformed by a linear layer, and the linear transformation result is activated by a RELU activation function to obtain the multi-modal video feature. The expression of the multi-modal video feature is as follows:

[0113]

[0114]

[0115] wherein, is a weight matrix of the linear transformation, is a bias coefficient.

[0116] In the linear transformation by the linear layer, dropout can be used to prevent overfitting.

[0117] The sentiment classification module 60 is configured to perform step S6 of performing sentiment classification prediction on the multi-modal video feature to obtain a sentiment classification result.

[0118] In the sentiment classification prediction on the multi-modal video feature, the sentiment tendency represented by the video data to be analyzed is predicted, and the sentiment classification result usually has two categories of negative and positive.

[0119] In this embodiment, the sentiment classification prediction on the multi-modal video feature is performed by a full connection layer of a softmax function to obtain the sentiment classification result , and the expression thereof is as follows:

[0120]

[0121] wherein, , is a weight coefficient of the full connection layer, , is a bias coefficient of the full connection layer.

[0122] Compared with the prior art, the multi-modal sentiment analysis method of the present application can identify information with high contribution to sentiment classification within each modality by further performing self-attention feature extraction on text features, image features and speech features, thereby reducing external information dependence and noise interference. Meanwhile, the image enhancement features and the speech enhancement features are respectively fused with the text features through cross-modal attention, which can increase the interaction and feature complementarity between the text modality and the image modality and the speech modality. Thus, the text self-attention features, the image self-attention features, the speech self-attention features, the image-text interaction features and the audio-text interaction features are fused to obtain multi-modal video features for classification prediction, which is conducive to improving the accuracy of sentiment classification prediction.

[0123] In addition, in the feature enhancement of the image features and the speech features, the self-attention mechanism is used to strengthen the sentiment information of the text data itself which has greater contribution to sentiment classification, and then the strengthened text features are used to further strengthen the image features and the speech features through the cross-modal multi-head attention mechanism, which can further improve the accuracy of sentiment classification prediction.

[0124] Based on the same inventive concept, the present application also provides an electronic device, which can be a server, a desktop computing device or a mobile computing device (such as a laptop computer, a handheld computing device, a tablet computer, a netbook, etc.) or a terminal device. The device includes one or more processors and a memory, wherein the processor is configured to execute a program to implement the multi-modal sentiment analysis method of the method embodiment; and the memory is configured to store a computer program executable by the processor.

[0125] Based on the same inventive concept, the present application also provides a computer readable storage medium, which corresponds to the above-mentioned multi-modal sentiment analysis method. The computer readable storage medium has a computer program stored thereon, and the program is executed by a processor to implement the steps of the multi-modal sentiment analysis method described in any of the above embodiments.

[0126] The application can take the form of a computer program product accessible from a storage medium (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code for use by or in connection with a computer. Computer-usable storage media include permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology for information storage. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0127] The above-described embodiments only express several embodiments of the present application, which are described in detail and specifically, but cannot be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, and the present application also intends to include these modifications and improvements.

Claims

1. A multi-modal sentiment analysis method, characterized in that, The method comprises the steps of: obtaining image data in the video data to be analyzed, performing feature extraction on the image data to obtain image features; obtaining text data in the video data to be analyzed, performing feature extraction on the text data to obtain text features; obtaining speech data in the video data to be analyzed, performing feature extraction on the speech data to obtain speech features; performing feature enhancement on the image features according to the text features to obtain image enhanced features; performing feature enhancement on the speech features according to the text features to obtain speech enhanced features; performing cross-modal attention interaction on the image enhanced features and the text features to obtain image-text interaction features; performing cross-modal attention interaction on the speech enhanced features and the text features to obtain audio-text interaction features; performing feature extraction on the image features based on a multi-head self-attention mechanism to obtain image self-attention features; performing feature extraction on the text features based on a multi-head self-attention mechanism to obtain text self-attention features; performing feature extraction on the speech features based on a multi-head self-attention mechanism to obtain speech self-attention features; fusing the text self-attention features, the image self-attention features, the speech self-attention features, the image-text interaction features, and the audio-text interaction features to obtain multi-modal video features; performing emotion classification prediction on the multi-modal video features to obtain emotion classification results; performing feature enhancement on the image features according to the text features to obtain image enhanced features, comprising the steps of: performing first self-attention encoding on the text features to obtain a first text encoding vector; performing multi-modal attention decoding on the first text encoding vector and the image features to obtain image enhanced features; performing first self-attention encoding on the text features through N first encoders connected in a chain structure, wherein the first encoders are used to perform multi-head attention calculation on the text features or output data of a previous first encoder as query Q, key K and value V respectively, and perform residual connection and normalization on the calculation results and the text features or output data of the previous first encoder to obtain first multi-head attention; and performing feedforward neural network processing on the first multi-head attention, and performing residual connection and normalization on the processing results and the first multi-head attention to obtain output data of the first encoder; performing multi-modal attention decoding on the first text encoding vector and the image features through N image-text decoders connected in a chain structure, wherein the image-text decoders are used to perform multi-head attention calculation on the first text encoding vector as query Q and key K, and the image features or output data of a previous image-text decoder as value V, and perform residual connection and normalization on the calculation results and the image features or output data of the previous image-text decoder to obtain image-text multi-head attention; performing feedforward neural network processing on the image-text multi-head attention, and performing residual connection and normalization on the processing results and the image-text multi-head attention to obtain output data of the image-text decoder.

2. The method of claim 1, wherein, According to the text feature, the speech feature is enhanced, and a speech enhanced feature is obtained, including the steps of: The second text encoding vector is obtained by performing second self-attention coding on the text feature. The second text encoding vector and the speech feature are multi-modal attention decoded to obtain a speech enhanced feature.

3. The method of claim 2, wherein: The text feature is coded by N second encoders connected in a chain structure, and the second encoder is used for multi-head attention calculation of the text feature or the output data of the previous second encoder as query Q, key K and value V respectively, and the calculation result is connected with the text feature or the output data of the previous second encoder in residual connection and normalized to obtain second multi-head attention. The output data of the second encoder is obtained by performing feedforward neural network processing on the second multi-head attention and connecting the processing result with the second multi-head attention in residual connection and normalization. The second text encoding vector and the speech feature are multi-modal attention decoded by N audio-text decoders connected in a chain structure, and the audio-text decoder is used for multi-head attention calculation of the second text encoding vector as query Q and key K, and the speech feature or the output data of the previous audio-text decoder as value V, and the calculation result is connected with the speech feature or the output data of the previous audio-text decoder in residual connection and normalized to obtain audio-text multi-head attention; the output data of the audio-text decoder is obtained by performing feedforward neural network processing on the audio-text multi-head attention and connecting the processing result with the audio-text multi-head attention in residual connection and normalization.

4. The method according to any one of claims 1 to 3, characterized in that, The image enhanced feature and the text feature are cross-modal attention interacted, including the steps of: the image enhanced feature is taken as query Q, and the text feature is taken as key K and value V to perform multi-head attention calculation to obtain image-text interaction feature.

5. The method of claim 4, wherein, The speech enhanced feature and the text feature are cross-modal attention interacted, including the steps of: the speech enhanced feature is taken as query Q, and the text feature is taken as key K and value V to perform multi-head attention calculation to obtain audio-text interaction feature.

6. The method of claim 5, wherein, The text self-attention feature, the image self-attention feature, the speech self-attention feature, the image-text interaction feature and the audio-text interaction feature are fused, including the steps of: After the text self-attention feature, the image self-attention feature, the speech self-attention feature, the image-text interaction feature and the audio-text interaction feature are spliced, linear transformation is performed, and the linear transformation result is activated to obtain a multi-modal video feature.

7. A multi-modal sentiment analysis device for performing the multi-modal sentiment analysis method of claim 1, wherein It includes: An image feature extraction module is configured to obtain image data in the video data to be analyzed, extract features from the image data, and obtain image features. A text feature extraction module is configured to obtain text data in the video data to be analyzed, extract features from the text data, and obtain text features. A speech feature extraction module is configured to obtain speech data in the video data to be analyzed, extract features from the speech data, and obtain speech features. an image feature enhancement module, configured to perform feature enhancement on the image feature according to the text feature, to obtain image enhanced features; a speech feature enhancement module, configured to perform feature enhancement on the speech feature according to the text feature, to obtain speech enhanced features; an image-text feature interaction module, configured to perform cross-modal attention interaction on the image enhanced features and the text feature, to obtain image-text interaction features; a speech-text feature interaction module, configured to perform cross-modal attention interaction on the speech enhanced features and the text feature, to obtain speech-text interaction features; an image self-attention module, configured to perform feature extraction on the image feature based on a multi-head self-attention mechanism, to obtain image self-attention features; a text self-attention module, configured to perform feature extraction on the text feature based on a multi-head self-attention mechanism, to obtain text self-attention features; a speech self-attention module, configured to perform feature extraction on the speech feature based on a multi-head self-attention mechanism, to obtain speech self-attention features; a feature fusion module, configured to fuse the text self-attention features, the image self-attention features, the speech self-attention features, the image-text interaction features and the speech-text interaction features, to obtain multi-modal video features; a sentiment classification module, configured to perform sentiment classification prediction on the multi-modal video features, to obtain a sentiment classification result.

8. An electronic device, comprising: comprise: a processor; a memory, configured to store a computer program executed by the processor; wherein the processor implements the steps of the method in any one of claims 1-6 when executing the computer program.

Images