Multimodal sentiment analysis method, device, equipment, storage medium and program product
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA MOBILE COMM LTD RES INST
- Filing Date
- 2025-02-06
- Publication Date
- 2026-08-07
AI Technical Summary
[0003]本申请实施例提供一种多模态情感分析方法、装置、设备、存储介质及程序产品，解决了目前由于捕捉局部特征的能力较弱导致情感分析的准确度不高的问题
[0046]本申请的实施例中，首先，获取视频数据的音频特征序列、图像特征序列和文本特征序列；其次，分别将所述音频特征序列和所述图像特征序列输入至conformer模块组，获得与所述音频特征序列对应的多个第一特征序列，以及与所述图像特征序列对应的多个第二特征序列，其中，所述第一特征序列包括所述视频数据的语音模态的全局特征和局部特征，所述第二特征序列包括所述视频数据的图像模态的全局特征和局部特征；如此，实现了对语音模态和图像模态的全局特征和局部特征的捕捉，减少了情感相关关键信息的丢失；再次，对所述第一特征序列、所述第二特征序列和所述文本特征序列，进行特征融合，获得所述视频数据的融合特征序列；如此，实现了多尺度多模态的特征融合；最后，利用所述融合特征序列，预测所述视频数据的情感类别。本申请实施例获得融合特征序列与视频数据的局部特征和全局特征等特征相关，使得基于融合特征序列进行情感类别预测的准确度更高。
Smart Images
Figure CN122528008A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a multimodal sentiment analysis method, apparatus, device, storage medium, and program product. Background Technology
[0002] In current multimodal sentiment analysis research, the processing of speech and images is mainly divided into two stages: primary feature extraction and feature encoding. In the primary feature extraction stage, various methods and feature types are available, each with its own advantages. In the feature encoding stage, most researchers use Long Short-Term Memory (LSTM) networks and Transformer networks to extract features specific to the modality and contextual information. LSTM is a special type of recurrent neural network capable of capturing long-term dependencies when processing sequential data. However, LSTM primarily focuses on long-term dependencies and is relatively weak in capturing local features. For data like speech and images with obvious local features, LSTM may not perform well. Transformer is a model architecture widely used in natural language processing and other fields in recent years, possessing excellent parallel computing capabilities and the ability to capture global features. However, it mainly relies on self-attention mechanisms to capture global features and is also insufficient in extracting local features, thus leading to lower accuracy in sentiment analysis. Summary of the Invention
[0003] This application provides a multimodal sentiment analysis method, apparatus, device, storage medium, and program product, which solves the problem that the accuracy of sentiment analysis is not high due to the weak ability to capture local features.
[0004] Firstly, to achieve the above objectives, embodiments of this application provide a multimodal sentiment analysis method, comprising:
[0005] Obtain audio feature sequences, image feature sequences, and text feature sequences from video data;
[0006] The audio feature sequence and the image feature sequence are respectively input into the conformer module group to obtain multiple first feature sequences corresponding to the audio feature sequence and multiple second feature sequences corresponding to the image feature sequence. The first feature sequence includes global and local features of the speech modality of the video data, and the second feature sequence includes global and local features of the image modality of the video data.
[0007] The first feature sequence, the second feature sequence, and the text feature sequence are fused to obtain the fused feature sequence of the video data;
[0008] The fused feature sequence is used to predict the sentiment category of the video data.
[0009] Specifically, the audio feature sequence and the image feature sequence are input into the conformer module group to obtain multiple first feature sequences corresponding to the audio feature sequence and multiple second feature sequences corresponding to the image feature sequence, including:
[0010] The audio feature sequence is input into the first conformer module group to obtain the first feature sequence output by each conformer block of the first conformer module group;
[0011] The image feature sequence is input into the second conformer module group to obtain the second feature sequence output by each conformer block of the second conformer module group;
[0012] The conformer module group includes the first conformer module group and the second conformer module group.
[0013] The process of fusing features from the first feature sequence, the second feature sequence, and the text feature sequence to obtain a fused feature sequence of the video data includes:
[0014] Multi-scale feature fusion is performed on the first feature sequence to obtain a third feature sequence, and multi-scale feature fusion is performed on the second feature sequence to obtain a fourth feature sequence; wherein the sequence lengths of the third feature sequence and the fourth feature sequence are the same as the sequence length of the text feature sequence;
[0015] The third feature sequence, the fourth feature sequence, and the text feature sequence are concatenated along the feature dimension to obtain the fifth feature sequence;
[0016] Attention statistical pooling and linear regularization are applied to the fifth feature sequence to obtain the fused feature sequence.
[0017] The process involves multimodal feature fusion of the first feature sequence to obtain a third feature sequence, including:
[0018] The first feature sequences are concatenated along the feature dimension to obtain the sixth feature sequence;
[0019] The sixth feature sequence is subjected to multi-head attention and average pooling to obtain the third feature sequence.
[0020] The second feature sequence is subjected to multimodal feature fusion to obtain a fourth feature sequence, including:
[0021] Multiple second feature sequences are concatenated along the feature dimension to obtain the seventh feature sequence;
[0022] The seventh feature sequence is subjected to multi-head attention and average pooling to obtain the fourth feature sequence.
[0023] The fifth feature sequence is subjected to attention statistical pooling and linear regularization to obtain the fused feature sequence, including:
[0024] The fifth feature sequence is subjected to attention statistical pooling to obtain the eighth feature sequence;
[0025] Delete the data in the target dimension from the eighth feature sequence to obtain the ninth feature sequence; wherein, the data in the target dimension is the first value;
[0026] The ninth feature sequence is subjected to linear regularization to obtain the fused feature sequence.
[0027] The acquisition of audio feature sequences from video data includes:
[0028] In the speech modality of the video data, low-level descriptor features and high-level descriptor features are collected according to a preset sampling rate;
[0029] The low-level descriptor features and the high-level descriptor features are subjected to convolutional downsampling and linear regularization to obtain the audio feature sequence.
[0030] The acquisition of image feature sequences from video data includes:
[0031] In the target image of the video data, a target object and the location of the target object are detected, wherein the target image is an image extracted from the video data at a preset image extraction speed, and the target object is the object that generates the speech in the video data;
[0032] Based on the location, extract the image features of the target object;
[0033] The extracted image features are subjected to convolutional downsampling and linear regularization to obtain the image feature sequence.
[0034] The method of predicting the sentiment category of the video data using the fused feature sequence includes:
[0035] The fused feature sequence is input into an activation function to obtain the processing result output by the activation function;
[0036] The processing result is subjected to linear regularization to obtain the emotion category.
[0037] Secondly, to achieve the above objectives, embodiments of this application provide a multimodal emotion analysis device, comprising:
[0038] The first acquisition module is used to acquire the audio feature sequence, image feature sequence, and text feature sequence of the video data;
[0039] The processing module is used to input the audio feature sequence and the image feature sequence into the conformer module group respectively to obtain a plurality of first feature sequences corresponding to the audio feature sequence and a plurality of second feature sequences corresponding to the image feature sequence, wherein the first feature sequence includes global and local features of the speech modality of the video data, and the second feature sequence includes global and local features of the image modality of the video data;
[0040] The fusion module is used to perform feature fusion on the first feature sequence, the second feature sequence, and the text feature sequence to obtain the fused feature sequence of the video data;
[0041] A prediction module is used to predict the sentiment category of the video data using the fused feature sequence.
[0042] Thirdly, to achieve the above objectives, embodiments of this application provide a multimodal sentiment analysis device, including a transceiver, a processor, a memory, and a program stored in the memory and executable on the processor; when the processor executes the program, it implements the multimodal sentiment analysis method as described in the first aspect.
[0043] Fourthly, to achieve the above objectives, embodiments of this application provide a readable storage medium having a program or instructions stored thereon, which, when executed by a processor, implement the multimodal sentiment analysis method as described in the first aspect.
[0044] Fifthly, to achieve the above objectives, embodiments of this application provide a computer program product including computer instructions that, when executed by a processor, implement the multimodal sentiment analysis method as described in the first aspect.
[0045] The beneficial effects of the above technical solution in this application are as follows:
[0046] In the embodiments of this application, firstly, audio feature sequences, image feature sequences, and text feature sequences of video data are acquired; secondly, the audio feature sequences and image feature sequences are respectively input into a conformer module group to obtain multiple first feature sequences corresponding to the audio feature sequences and multiple second feature sequences corresponding to the image feature sequences. The first feature sequences include global and local features of the speech modality of the video data, and the second feature sequences include global and local features of the image modality of the video data. This achieves the capture of global and local features of both the speech and image modalities, reducing the loss of key emotion-related information. Thirdly, feature fusion is performed on the first feature sequences, second feature sequences, and text feature sequences to obtain a fused feature sequence of the video data. This achieves multi-scale, multi-modal feature fusion. Finally, the emotion category of the video data is predicted using the fused feature sequence. The embodiments of this application obtain fused feature sequences that are correlated with local and global features of the video data, resulting in higher accuracy in emotion category prediction based on the fused feature sequences. Attached Figure Description
[0047] Figure 1 This is one of the flowcharts illustrating the multimodal sentiment analysis method according to an embodiment of this application;
[0048] Figure 2 This is a second schematic flowchart of the multimodal sentiment analysis method according to an embodiment of this application;
[0049] Figure 3 This is a schematic diagram of the structure of the multimodal emotion analysis device according to an embodiment of this application;
[0050] Figure 4 This is a schematic diagram of the structure of a multimodal emotion analysis device according to an embodiment of this application. Detailed Implementation
[0051] To make the technical problems, technical solutions and advantages of this application clearer, a detailed description will be provided below in conjunction with the accompanying drawings and specific embodiments.
[0052] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.
[0053] In the various embodiments of this application, it should be understood that the sequence number of each process described below does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0054] In addition, the terms "system" and "network" are often used interchangeably in this article.
[0055] In the embodiments provided in this application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.
[0056] Before describing the embodiments of this application, the relevant technical points will be explained first:
[0057] Convolutional Neural Networks (CNNs): CNNs are deep learning models particularly well-suited for image processing tasks. A CNN typically consists of a series of convolutional layers, pooling layers, and fully connected layers, and can automatically learn hierarchical features in an image. In sentiment analysis, CNNs can be used to extract local features from image modalities.
[0058] Transformer Architecture: The Transformer is a model architecture for natural language processing that abandons the traditional Recursive Neural Network (RNN) and instead relies on a self-attention mechanism to process sequential data. By processing each element in the input sequence in parallel, the Transformer significantly accelerates training and can better capture long-range dependencies.
[0059] The conformer block: The conformer is an advanced variant of the self-attention mechanism that combines the advantages of CNN and Transformer architectures, maintaining good performance when processing long sequences of data. The conformer block is commonly used in speech recognition and natural language processing tasks. Its core components include a multi-head self-attention (MHSA) module, a convolutional module, a feedforward neural network (FNN) module, residual connections, and layer normalization.
[0060] Bidirectional Encoder Representations from Transformers (BERT) Model: BERT is a pre-trained model based on the Transformer architecture, widely used in natural language processing, capable of capturing contextual information from text. The BERT model consists of 12 stacked Transformer layers, each containing a multi-head self-attention mechanism and a feedforward neural network. Through pre-training and fine-tuning, the BERT model can effectively extract text features.
[0061] TalkNelt Network: The TalkNelt network is primarily used for speaker detection, accurately identifying speakers in multi-person dialogue scenarios. This network utilizes a deep learning model to determine the speaker's identity for each face in a video frame.
[0062] Multi-task Cascaded Convolutional Networks (MTCNN) Algorithm: MTCNN is a powerful algorithm for face detection, commonly used in real-time video processing. MTCNN uses three cascaded convolutional networks to handle coarse localization, precise localization, and keypoint detection, respectively.
[0063] OpenFace Toolkit: OpenFace is an open-source facial behavior coding system used to extract facial features and support applications such as expression analysis. OpenFace can extract various facial features, including 68 facial landmarks and 17 facial action units (AUs).
[0064] OpenSMILE is an open-source audio feature extraction tool commonly used for speech sentiment analysis. It can extract various audio features, such as Mel-frequency cepstral coefficients (MFCCs), fundamental frequency (F0), and pitch.
[0065] Based on the above, embodiments of this application provide a multimodal sentiment analysis method, such as... Figure 1 As shown, the method includes:
[0066] Step 101: Obtain the audio feature sequence, image feature sequence, and text feature sequence of the video data.
[0067] Step 102: Input the audio feature sequence and the image feature sequence into the conformer module group respectively to obtain multiple first feature sequences corresponding to the audio feature sequence and multiple second feature sequences corresponding to the image feature sequence. The first feature sequence includes global and local features of the speech modality of the video data, and the second feature sequence includes global and local features of the image modality of the video data.
[0068] It's important to note that the Conformer combines convolutional layers and a self-attention mechanism in its structure. Convolutional layers excel at extracting local features, while the self-attention mechanism captures long-term dependencies, allowing the Conformer to simultaneously focus on both local and global information when processing speech and image data. Therefore, in step 102 above, by using the Conformer module group to obtain multiple first feature sequences corresponding to the audio feature sequence and multiple second feature sequences corresponding to the video feature sequence, we can obtain the local and global features of the speech modality in the video data, as well as the local and global features of the image modality in the video data, reducing the probability of losing key emotion-related information. Furthermore, although the Conformer still has a relatively high memory footprint, compared to the standard Transformer, combining it with convolutional layers can alleviate the high memory consumption problem to some extent, especially when processing long sequence data.
[0069] Step 103: Perform feature fusion on the first feature sequence, the second feature sequence, and the text feature sequence to obtain the fused feature sequence of the video data.
[0070] Step 103 above fuses the first feature sequence containing global and local features of the speech modality, the second feature sequence containing global and local features of the image modality, and the text feature sequence, realizing multi-scale feature fusion, improving the ability to capture information at different scales of data, and providing a foundation for improving the accuracy of subsequent sentiment category analysis / prediction.
[0071] Step 104: Using the fused feature sequence, predict the sentiment category of the video data.
[0072] In the multimodal sentiment analysis method of this application embodiment, firstly, audio feature sequences, image feature sequences, and text feature sequences of video data are acquired; secondly, the audio feature sequences and image feature sequences are respectively input into a conformer module group to obtain multiple first feature sequences corresponding to the audio feature sequences and multiple second feature sequences corresponding to the image feature sequences. The first feature sequences include global and local features of the speech modality of the video data, and the second feature sequences include global and local features of the image modality of the video data. This achieves the capture of global and local features of both the speech and image modalities. Thirdly, feature fusion is performed on the first feature sequences, second feature sequences, and text feature sequences to obtain a fused feature sequence of the video data. This achieves multi-scale, multimodal feature fusion of video data, reducing the probability of losing key sentiment-related information. Finally, the sentiment category of the video data is predicted using the fused feature sequence related to the multimodal local and global features of the video data. This improves the accuracy of the prediction results.
[0073] As an optional implementation, step 102 includes:
[0074] The audio feature sequence is input into the first conformer module group to obtain the first feature sequence output by each conformer block of the first conformer module group.
[0075] The first conformer module group in the above steps includes multiple stacked conformer blocks, such as... Figure 2 As shown, stacking multiple conformer blocks means that multiple conformers are connected in series. That is, the output of the first conformer block is connected to the input of the second conformer block. In addition, the outputs of each conformer block also form the output of the conformer module group. That is, the output of the conformer module group includes the outputs of each conformer block.
[0076] The image feature sequence is input into the second conformer module group to obtain the second feature sequence output by each conformer block of the second conformer module group;
[0077] Here, the structure of the second conformer module group is similar to that of the first conformer module group, and will not be described again here.
[0078] The conformer module group includes the first conformer module group and the second conformer module group.
[0079] The implementation process of the above optional method can be described as follows: Audio feature sequences / image feature sequences are input into multiple stacked conformer blocks. Global and local features of the speech and image modalities are modeled through internal multi-head self-attention blocks and convolutional blocks, thereby providing a more effective modal feature representation capability. Specifically, the output dimension in the conformer block is set to 64. Therefore, after multiple conformer blocks, the data dimension becomes L*[32, 230, 64], where L represents the number of conformer blocks.
[0080] Here, it's important to note that each conformer block consists of four stacked modules: a feedforward module, a self-attention module, a convolutional module, and a second feedforward module at the end. The key components of the conformer block are the multi-head self-attention (MHSA) and convolutional modules. The MHSA in the conformer block uses relative position encoding. The convolutional module following the MHSA consists of pointwise convolutions, 1D depthwise convolutions, and batch normalization after the convolutional layers, which helps to train deep models more easily. Each conformer block includes two macaron-like feedforward modules (feedforward neural networks, FNNs), where a 1 / 2 residual connection is sandwiched between the MHSA and the convolutional module (Conv). Mathematically, for the input x of conformer block i... i-1 The output x of this block i for:
[0081]
[0082] Where, x i-1 ∈R d×T x i ∈R d×T , where d represents the dimension of the conformer block (conformer encoder) and T represents the frame length.
[0083] It's also worth noting that extracting features from a single conformer block might lose some crucial sentiment-related information, while stacking multiple conformer blocks and concatenating their multi-scale outputs reduces this information loss. Each conformer block, extracting features at different levels, captures information at different scales of the input data: the earlier layers typically extract low-level features such as edges and textures, containing a wealth of detailed information that might be overlooked in higher-level features; these details are preserved through concatenation. Later layers gradually extract higher-level features, such as semantic information and global dependencies, providing more contextual information and a global perspective, enhancing the model's ability to understand complex patterns.
[0084] As an optional implementation, step 103 includes:
[0085] Multi-scale feature fusion is performed on the first feature sequence to obtain a third feature sequence, and multi-scale feature fusion is performed on the second feature sequence to obtain a fourth feature sequence; wherein the sequence lengths of the third feature sequence and the fourth feature sequence are the same as the sequence length of the text feature sequence.
[0086] In the above steps, the sequence lengths of the third feature sequence, the fourth feature sequence, and the text feature sequence are set to the same length to facilitate the subsequent concatenation of the three in terms of feature dimensions.
[0087] The third feature sequence, the fourth feature sequence, and the text feature sequence are concatenated along the feature dimension to obtain the fifth feature sequence.
[0088] Attention statistical pooling and linear regularization are applied to the fifth feature sequence to obtain the fused feature sequence.
[0089] As a specific implementation, multimodal feature fusion is performed on the first feature sequence to obtain a third feature sequence, including:
[0090] Multiple first feature sequences are concatenated along the feature dimension to obtain a sixth feature sequence. In this step, concatenating the first feature sequences output by each conformer block can significantly improve performance.
[0091] The sixth feature sequence is subjected to multi-head attention and average pooling to obtain the third feature sequence.
[0092] As another specific implementation, multimodal feature fusion is performed on the second feature sequence to obtain a fourth feature sequence, including:
[0093] Multiple second feature sequences are concatenated along the feature dimension to obtain the seventh feature sequence;
[0094] The seventh feature sequence is subjected to multi-head attention and average pooling to obtain the fourth feature sequence.
[0095] In the specific implementation described above, the process of concatenating the output features of each conformer block for each modality (the first feature sequence of the speech modality and the second feature sequence of the image modality) is represented as follows:
[0096] X′=Concat(x1,x2,…,x L )
[0097] Where X'∈R D×T X = [X1, X2, ..., X] T ]∈R D×T D = d × L, where L represents the number of conformer blocks.
[0098] In the specific implementation described above, the concatenated feature sequences (the sixth feature obtained by concatenating the first feature sequences and the seventh feature sequence obtained by concatenating the second feature sequences) are input into a multi-head self-attention layer for multi-head attention processing (a weighted summation operation is performed on the input feature sequences). This allows for focusing on useful information within the input sequences. The attention mechanism formula is expressed as follows:
[0099]
[0100] Where Q, K, and V are the query, key, and value matrices, respectively. The Transformer computes multiple parallel attentions, and the output of each attention is called a head. The formula for calculating the i-th head is as follows:
[0101] head i =Attention(QW i q ,KW i k VW i v )
[0102] Among them, W i q / k / v ∈R d×d It is a specific header parameter used to linearly project a matrix onto a local space, where d represents the linear dimension.
[0103] In the specific implementation described above, average pooling is applied to the hidden representations of audio and visual modes to achieve alignment with the frame length of the text.
[0104] Following the previous example, the two specific implementations are as follows: The feature dimensions output by multiple conformers are concatenated to obtain the data dimensions [32, 230, L*64]; then, the data is fed into a multi-head self-attention layer, and the sequence length is aligned with the text sequence length. The text sequence length is fixed at 50, so the speech data dimension becomes [32, 50, L*64] after average pooling. Similarly, the video data dimension is transformed to [32, 50, L*64].
[0105] As another specific implementation, attention statistical pooling and linear regularization are applied to the fifth feature sequence to obtain the fused feature sequence, including:
[0106] The fifth feature sequence is subjected to attention statistical pooling to obtain the eighth feature sequence;
[0107] The attention-based statistical pooling step described above uses attention to weight importance, generating the mean and standard deviation. For the frame-level feature X at time step t... t First, the scalar score e needs to be calculated. t and normalized score a t as follows:
[0108] e t =v T f(WX t +b)+k
[0109]
[0110] Where W∈R D×D b∈R D×1 ,v∈R D×1 , where k is a trainable parameter used for attention. f(·) denotes the Tanh activation function. After this, the normalized score α is... t Used as weights to calculate the weighted mean vector and weighted standard deviation The calculation formula is as follows:
[0111]
[0112] in, ⊙ represents the Hadamard product.
[0113] Delete the data in the target dimension of the eighth feature sequence to obtain the ninth feature sequence; wherein the data in the target dimension is a first value; for example, the first value is 1.
[0114] The ninth feature sequence is subjected to linear regularization to obtain the fused feature sequence.
[0115] Following the previous example, in the specific implementation described above, the fifth feature sequence is a feature sequence obtained by concatenating the feature sequences of text, speech, and image in the feature dimension. The dimension of the fifth feature sequence is [32, 50, L*64*2+768]. Based on this, this specific implementation performs attention pooling on the fusion vector and removes dimensions that are equal to 1, thus transforming the data dimension of the fusion feature sequence to [32, (L*64*2+768)].
[0116] As an optional implementation, step 101 involves obtaining the audio feature sequence of the video data, including:
[0117] In the speech modality of the video data, low-level descriptor features and high-level descriptor features are collected according to a preset sampling rate. Here, firstly, the preset sampling rate is, for example, 16kHz; secondly, this step can use the aforementioned OpenSMILE tool to collect the low-level and high-level descriptor features; thirdly, the low-level and high-level descriptor features are, for example, features from the eGeMAPS feature set; fourthly, the low-level and high-level descriptor features include 25-dimensional speech features such as MFCC1-4, F0, and Pitch.
[0118] The low-level descriptor features and the high-level descriptor features are subjected to convolutional downsampling and linear regularization to obtain the audio feature sequence. The sequence length of the audio feature sequence is, for example, 925.
[0119] In the above steps, the convolutional downsampling process can specifically involve performing two-dimensional convolutional downsampling on the input features, achieving a reduction in frame rate and feature dimension to one-quarter of the original using a four-fold convolutional downsampling. Specifically, the convolutional kernel size is 4, and the stride is 2. To conform to the input of the convolutional block, an additional dimension is added to the second dimension, making the input [B, 1, T, F], where T represents the sequence length and F represents the feature dimension. Taking speech data as an example, after passing through the convolutional downsampling layer, the data shape becomes [32, 230, 64].
[0120] As another optional implementation, in step 101, the image feature sequence of the video data is obtained, including:
[0121] In the target image of the video data, the target object and the location of the target object are detected, wherein the target image is an image extracted from the video data at a preset image extraction speed, and the target object is the object that generates the speech in the video data.
[0122] In the above steps, firstly, FFmpeg can be used to extract the target image at a rate of 25 frames per second. Secondly, the aforementioned TalkNelt network can be used to detect the speaker's face among all faces in a single image. During the process of detecting the speaker's face, if detection fails (no speaker's face is detected), the image is discarded. If the proportion of discarded images in the extracted target image reaches 25%, then that instance (the instance for multimodal sentiment analysis) is discarded. Thirdly, the aforementioned MTCNN face detection algorithm can be used to detect the speaker's face location.
[0123] Based on the location, image features of the target object are extracted. Specifically, this step may involve using the aforementioned OpenFace toolkit to extract facial features (image features) of the target object. The extracted facial features include 68 facial landmarks, 17 facial motion units, head pose, head orientation, and eye gaze direction, etc.
[0124] The extracted image features are subjected to convolutional downsampling and linear regularization to obtain the image feature sequence. For example, this step involves padding or truncating the 177-dimensional image features into an image feature sequence with a length of 232.
[0125] The convolutional downsampling and linear regularization processes in the above optional implementations are similar to the processing of the feature sequence of the previous speech modality, and will not be described in detail here.
[0126] In the two optional implementation methods mentioned above, by performing convolutional downsampling and linear regularization on the input features, the length of the input sequence can be shortened and the feature dimension reduced, thereby reducing the computational pressure on subsequent models and improving processing efficiency.
[0127] Alternatively, as an optional implementation, in step 101, obtaining the text feature sequence of the video data can specifically be as follows: Figure 2 As shown, contextual word embeddings can be learned using a pre-trained BERT model (a BERT model based on Chinese, Bert-base-chinese), which consists of 12 stacked Transformer layers. The sequence length is fixed to 50 using padding or truncation methods, and the final 768-dimensional hidden layer state vector is selected as the final text feature.
[0128] As an optional implementation, step 104 includes:
[0129] The fused feature sequence is input into an activation function to obtain the processing result output by the activation function; in this step, the activation function is, for example, a Gaussian Error LinearUnits (GELU) activation function.
[0130] The processing result is subjected to linear regularization to obtain the emotion category.
[0131] In simple terms, the above-mentioned optional implementation method is to use a fully connected layer with Gelu activation function to extract the sentiment embedding from the high-dimensional representation to the low-dimensional representation in order to obtain the final sentiment classification or prediction result.
[0132] This application innovatively proposes a multimodal sentiment analysis method, aiming to deeply mine sentiment information from speech and image modalities. In this embodiment, a conformer module is used to extract global and local features from speech and images, and a multi-scale feature fusion module integrates sentiment information at different levels. Text features are obtained through a pre-trained BERT model and integrated with the multimodal fusion representation. A specific example of the implementation process of the above embodiment is as follows:
[0133] First, primary features of the speech and image modalities, as well as word embeddings of the text, are separated and extracted from the video. Figure 2 (Text, audio, and images on the left). Specifically, when reading each frame of the video, the TalkNelt network is used for speaker detection, and when multiple faces appear in the video, it is used to determine the speaker's face. Then, the MTCNN face detection algorithm is used to detect the face location, and the OpenFace toolkit is used to extract facial features. For the initial feature extraction of speech, the FFmpeg toolkit is first used to separate the fixed-format audio data from the video, and then the OpenSMILE toolkit is used to extract various low-level and high-level descriptor features (including MFCC1-4, F0, Pitch, and other multi-dimensional speech features) at a sampling rate of 16000Hz. For the initial feature extraction of text, a pre-trained BERT model is used to learn contextual word embeddings from the processed text content.
[0134] Secondly, the longer the input sequence, the larger the feature dimension, and the greater the computational cost to the network. Since multiple consecutive frames of a speech signal and an image correspond to the same pronunciation and the same facial landmark, a two-dimensional convolutional downsampling layer (corresponding to...) is first used to reduce the computational burden on the model. Figure 2 The input is processed by a convolutional downsampling layer (in the code), which uses a 4x convolutional downsampling to reduce the frame rate and feature dimension to one-quarter of the original. The convolutional kernel size is 4, and the stride is 2. To conform to the input of the convolutional block, a second dimension is added, making the input [B, 1, T, F], where T represents the sequence length and F represents the feature dimension. Taking speech data as an example, after passing through the convolutional downsampling layer, the data shape becomes [32, 230, 64], and then it is input into multiple stacked conformer blocks (corresponding to...). Figure 2In the first and second conformer module groups, the global and local features of speech and image modalities are modeled through internal multi-head self-attention blocks and convolutional blocks, thereby providing more effective modal feature representation capabilities. Specifically, the output calculation of the conformer block is as described above. The output dimension in the conformer block is set to 64, so after multiple conformer blocks, the data dimension becomes L*[32, 230, 64], where L represents the number of conformer blocks. Then, the multiple conformer outputs are concatenated along the feature dimension (corresponding to...). Figure 2 The concatenation (concat) in the middle yields the data dimension [32, 230, L*64], which is then fed into the multi-head self-attention layer (corresponding to Figure 2 (Multi-head attention in the process) and aligning the sequence length with the text sequence length, with the text sequence length fixed at 50, so the speech data dimension is after average pooling (corresponding to...) Figure 2 After average pooling, the data becomes [32, 50, L*64]. Similarly, the dimensionality of the video data is transformed to [32, 50, L*64]. At this point, we have obtained the feature representations of text, speech, and images. We then concatenate these sequences along the feature dimensions (corresponding to...). Figure 2 The concatenation (concat) in the model yields a fused representation with dimensions [32, 50, L*64*2+768]. Then, attention-based statistical pooling (corresponding to...) is performed on the fused vector. Figure 2 The data is transformed into [32, (L*64*2+768)] by using Attentive Statistics Pooling (APS) and removing dimensions with a value of 1. The specific calculation formula for APS is as described above. Finally, the data is processed through a fully connected layer and a GELU activation layer to obtain the final sentiment classification or prediction.
[0135] In the multimodal sentiment analysis method described in this application embodiment, firstly, by stacking multiple conformer blocks and concatenating their multi-scale outputs, the model's ability to capture information at different scales of the input data is improved. Specifically, it captures both local features and takes into account global dependencies, reducing the loss of key sentiment-related information and thus improving the accuracy of sentiment analysis. Secondly, a two-dimensional convolutional downsampling layer is used to process the input data, reducing the computational burden on the model and improving processing efficiency. Thirdly, features from text, speech, and image modalities are extracted using the BERT model, OpenSMILE tool, and OpenFace toolkit, and then fused through the conformer module, achieving multimodal feature fusion and enhancing the modal feature representation capability. Fourthly, an attention mechanism is used to weight the importance of multimodal features, further improving the model's robustness and generalization ability.
[0136] Embodiments of this application also provide a multimodal emotion analysis device, such as... Figure 3 As shown, it includes:
[0137] The first acquisition module 301 is used to acquire the audio feature sequence, image feature sequence and text feature sequence of video data;
[0138] Processing module 302 is used to input the audio feature sequence and the image feature sequence into the conformer module group respectively to obtain a plurality of first feature sequences corresponding to the audio feature sequence and a plurality of second feature sequences corresponding to the image feature sequence, wherein the first feature sequence includes global and local features of the speech modality of the video data, and the second feature sequence includes global and local features of the image modality of the video data;
[0139] The fusion module 303 is used to perform feature fusion on the first feature sequence, the second feature sequence and the text feature sequence to obtain the fused feature sequence of the video data;
[0140] The prediction module 304 is used to predict the sentiment category of the video data using the fused feature sequence.
[0141] The processing module 302 includes:
[0142] The first processing submodule is used to input the audio feature sequence into the first conformer module group to obtain the first feature sequence output by each conformer block of the first conformer module group;
[0143] The second processing submodule is used to input the image feature sequence into the second conformer module group to obtain the second feature sequence output by each conformer block of the second conformer module group;
[0144] The conformer module group includes the first conformer module group and the second conformer module group.
[0145] The fusion module 303 includes:
[0146] The fusion submodule is used to perform multi-scale feature fusion on the first feature sequence to obtain a third feature sequence, and to perform multi-scale feature fusion on the second feature sequence to obtain a fourth feature sequence; wherein the sequence lengths of the third feature sequence and the fourth feature sequence are the same as the sequence length of the text feature sequence;
[0147] The first splicing submodule is used to splice the third feature sequence, the fourth feature sequence and the text feature sequence along the feature dimension to obtain the fifth feature sequence;
[0148] The third processing submodule is used to perform attention statistical pooling and linear regularization on the fifth feature sequence to obtain the fused feature sequence.
[0149] The first fusion submodule includes:
[0150] The first splicing unit is used to splice multiple first feature sequences along the feature dimension to obtain a sixth feature sequence;
[0151] The first processing unit is used to perform multi-head attention and average pooling processing on the sixth feature sequence to obtain the third feature sequence.
[0152] The second fusion submodule includes:
[0153] The second splicing unit is used to splice multiple second feature sequences along the feature dimension to obtain the seventh feature sequence;
[0154] The second processing unit is used to perform multi-head attention and average pooling processing on the seventh feature sequence to obtain the fourth feature sequence.
[0155] The third processing submodule includes:
[0156] The third processing unit is used to perform attention statistical pooling on the fifth feature sequence to obtain the eighth feature sequence;
[0157] The deletion unit is used to delete data in the target dimension of the eighth feature sequence to obtain the ninth feature sequence; wherein, the data in the target dimension is the first value;
[0158] The fourth processing unit is used to perform linear regularization processing on the ninth feature sequence to obtain the fused feature sequence.
[0159] The first acquisition module 301 includes:
[0160] The sampling submodule is used to collect low-level descriptor features and high-level descriptor features in the speech modality of the video data according to a preset sampling rate.
[0161] The fourth processing submodule is used to perform convolution downsampling and linear regularization processing on the low-level descriptor features and the high-level descriptor features to obtain the audio feature sequence.
[0162] The first acquisition module 301 includes:
[0163] The detection submodule is used to detect a target object and the location of the target object in the target image of the video data, wherein the target image is an image extracted from the video data at a preset image extraction speed, and the target object is the object that generates the speech in the video data;
[0164] An extraction submodule is used to extract image features of the target object based on the location;
[0165] The fifth processing submodule is used to perform convolution downsampling and linear regularization processing on the extracted image features to obtain the image feature sequence.
[0166] The prediction module 304 includes:
[0167] The sixth processing submodule is used to input the fused feature sequence into the activation function and obtain the processing result output by the activation function;
[0168] The seventh processing submodule is used to perform linear regularization on the processing result to obtain the emotion category.
[0169] It should be noted that the multimodal sentiment analysis device provided in this application embodiment can implement all the method steps implemented in the multimodal sentiment analysis method embodiment and can achieve the same technical effect. Therefore, the parts and beneficial effects that are the same as those in the method embodiment will not be described in detail here.
[0170] Embodiments of this application also provide a multimodal sentiment analysis device, including a transceiver 410, a processor 400, a memory 420, and a program stored in the memory 420 and executable on the processor 400; wherein, when the processor 400 executes the program, it implements the multimodal sentiment analysis method as described above.
[0171] The transceiver 410 is used to receive and send data under the control of the processor 400.
[0172] Among them, Figure 4In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits of one or more processors represented by processor 400 and memory represented by memory 420. The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. Transceiver 410 can be multiple elements, including transmitters and receivers, providing a unit for communicating with various other devices over a transmission medium. For different terminals, the user interface 430 can also be an interface capable of connecting external or internal devices, including but not limited to keypads, displays, speakers, microphones, joysticks, etc.
[0173] The processor 400 is responsible for managing the bus architecture and general processing, while the memory 420 can store the data used by the processor 400 when performing operations.
[0174] This application also provides a readable storage medium storing a program. When executed by a processor, this program implements the multimodal sentiment analysis method described above and achieves the same technical effect. To avoid repetition, it will not be described again here. The readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
[0175] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, disk, optical disk) and includes several instructions for executing the methods described in the various embodiments of this application.
[0176] Therefore, embodiments of this application also provide a computer program product, including computer instructions, which, when executed by a processor, implement the multimodal sentiment analysis method described above and achieve the same technical effect. To avoid repetition, further details are omitted here.
[0177] In this embodiment, the module can be implemented in software so that it can be executed by various types of processors. For example, an identified executable code module may include one or more physical or logical blocks of computer instructions, which may be constructed as objects, procedures, or functions. Nevertheless, the executable code of the identified module does not need to be physically located together, but may include different instructions stored in different bits, which, when logically combined, constitute the module and achieve the module's intended purpose.
[0178] In practice, an executable code module can be a single instruction or many instructions, and can even be distributed across multiple different code segments, different programs, and across multiple memory devices. Similarly, operational data can be identified within the module and can be implemented in any suitable form and organized within any suitable type of data structure. This operational data can be collected as a single dataset or distributed across different locations (including different storage devices), and can exist, at least in part, solely as electronic signals within the system or network.
[0179] When a module can be implemented using software, considering the current level of hardware technology, modules that can be implemented in software can be implemented using hardware circuits by those skilled in the art to achieve the corresponding functions, without considering cost. These hardware circuits include conventional very-large-scale integrated circuits (VLSI) or gate arrays, as well as existing semiconductors such as logic chips and transistors, or other discrete components. Modules can also be implemented using programmable hardware devices, such as field-programmable gate arrays, programmable array logic, and programmable logic devices.
[0180] The exemplary embodiments described above are with reference to the accompanying drawings. Many different forms and embodiments are feasible without departing from the spirit and teachings of this application. Therefore, this application should not be construed as limiting the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided to make this application complete and convey the scope of this application to those skilled in the art. In these drawings, component dimensions and relative dimensions may be exaggerated for clarity. The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. As used herein, unless clearly indicated otherwise, the singular forms “a,” “an,” and “the” are intended to include all such forms. It will be further understood that the terms “comprising” and / or “including”, when used in this specification, indicate the presence of the stated features, integers, steps, operations, components, and / or elements, but do not exclude the presence or addition of one or more other features, integers, steps, operations, components, and / or groups thereof. Unless otherwise indicated, when stated, a range of values includes the upper and lower limits of the range and any subranges in between.
[0181] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. A multimodal sentiment analysis method, characterized in that, include: Obtain audio feature sequences, image feature sequences, and text feature sequences from video data; The audio feature sequence and the image feature sequence are respectively input into the conformer module group to obtain multiple first feature sequences corresponding to the audio feature sequence and multiple second feature sequences corresponding to the image feature sequence. The first feature sequence includes global and local features of the speech modality of the video data, and the second feature sequence includes global and local features of the image modality of the video data. The first feature sequence, the second feature sequence, and the text feature sequence are fused to obtain the fused feature sequence of the video data; The fused feature sequence is used to predict the sentiment category of the video data.
2. The method according to claim 1, characterized in that, The audio feature sequence and the image feature sequence are respectively input into the conformer module group to obtain multiple first feature sequences corresponding to the audio feature sequence and multiple second feature sequences corresponding to the image feature sequence, including: The audio feature sequence is input into the first conformer module group to obtain the first feature sequence output by each conformer block of the first conformer module group; The image feature sequence is input into the second conformer module group to obtain the second feature sequence output by each conformer block of the second conformer module group; The conformer module group includes the first conformer module group and the second conformer module group.
3. The method according to claim 1, characterized in that, The first feature sequence, the second feature sequence, and the text feature sequence are fused to obtain a fused feature sequence of the video data, including: Multi-scale feature fusion is performed on the first feature sequence to obtain a third feature sequence, and multi-scale feature fusion is performed on the second feature sequence to obtain a fourth feature sequence; wherein the sequence lengths of the third feature sequence and the fourth feature sequence are the same as the sequence length of the text feature sequence; The third feature sequence, the fourth feature sequence, and the text feature sequence are concatenated along the feature dimension to obtain the fifth feature sequence; Attention statistical pooling and linear regularization are applied to the fifth feature sequence to obtain the fused feature sequence.
4. The method according to claim 3, characterized in that, Multimodal feature fusion is performed on the first feature sequence to obtain a third feature sequence, including: The first feature sequences are concatenated along the feature dimension to obtain the sixth feature sequence; The sixth feature sequence is subjected to multi-head attention and average pooling to obtain the third feature sequence.
5. The method according to claim 3, characterized in that, Multimodal feature fusion is performed on the second feature sequence to obtain a fourth feature sequence, including: Multiple second feature sequences are concatenated along the feature dimension to obtain the seventh feature sequence; The seventh feature sequence is subjected to multi-head attention and average pooling to obtain the fourth feature sequence.
6. The method according to claim 3, characterized in that, Attention statistical pooling and linear regularization are applied to the fifth feature sequence to obtain the fused feature sequence, including: The fifth feature sequence is subjected to attention statistical pooling to obtain the eighth feature sequence; Delete the data in the target dimension from the eighth feature sequence to obtain the ninth feature sequence; wherein, the data in the target dimension is the first value; The ninth feature sequence is subjected to linear regularization to obtain the fused feature sequence.
7. The method according to claim 1, characterized in that, Obtain the audio feature sequence of the video data, including: In the speech modality of the video data, low-level descriptor features and high-level descriptor features are collected according to a preset sampling rate; The low-level descriptor features and the high-level descriptor features are subjected to convolutional downsampling and linear regularization to obtain the audio feature sequence.
8. The method according to claim 1, characterized in that, Obtain the image feature sequence of the video data, including: In the target image of the video data, a target object and the location of the target object are detected, wherein the target image is an image extracted from the video data at a preset image extraction speed, and the target object is the object that generates the speech in the video data; Based on the location, extract the image features of the target object; The extracted image features are subjected to convolutional downsampling and linear regularization to obtain the image feature sequence.
9. The method according to claim 1, characterized in that, Using the fused feature sequence, predicting the sentiment category of the video data includes: The fused feature sequence is input into an activation function to obtain the processing result output by the activation function; The processing result is subjected to linear regularization to obtain the emotion category.
10. A multimodal emotion analysis device, characterized in that, include: The first acquisition module is used to acquire the audio feature sequence, image feature sequence, and text feature sequence of the video data; The processing module is used to input the audio feature sequence and the image feature sequence into the conformer module group respectively to obtain a plurality of first feature sequences corresponding to the audio feature sequence and a plurality of second feature sequences corresponding to the image feature sequence, wherein the first feature sequence includes global and local features of the speech modality of the video data, and the second feature sequence includes global and local features of the image modality of the video data; The fusion module is used to perform feature fusion on the first feature sequence, the second feature sequence, and the text feature sequence to obtain the fused feature sequence of the video data; A prediction module is used to predict the sentiment category of the video data using the fused feature sequence.
11. A multimodal sentiment analysis device, comprising a transceiver, a processor, a memory, and a program stored in the memory and executable on the processor; characterized in that, When the processor executes the program, it implements the multimodal sentiment analysis method as described in any one of claims 1 to 9.
12. A readable storage medium having a program or instructions stored thereon, characterized in that, When the program or instructions are executed by the processor, they implement the multimodal sentiment analysis method as described in any one of claims 1 to 9.
13. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the multimodal sentiment analysis method as described in any one of claims 1 to 9.