Multi-modal sentiment analysis method and device

Through the cross attention feature interaction network in the multimodal emotion analysis method, the problem of high noise in multimodal emotion analysis is solved, the analysis effect of the model is improved, and the accuracy of emotion analysis is enhanced.

CN118965263APending Publication Date: 2024-11-15GUANGDONG UNIV OF TECH

Patent Information

Application Number
CN202411005001.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The existing multimodal sentiment analysis methods have less modal feature information and more noise in video and audio, resulting in more characteristic noise after fusion, which affects the model analysis effect.

Method used

Using multimodal emotion analysis method, multimodal emotion tensors are generated through multimodal feature extraction network, cross-attention feature interaction network and multimodal feature scoring network. The interaction between sequences reduces the impact of noise and improves the model analysis effect.

Benefits of technology

Through the use of cross-attention feature interaction network, the interactive features between multimodal sequences are added, thereby reducing the impact of noise, improving the analysis effect of the model, and enhancing the accuracy of sentiment analysis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118965263A_ABST
    Figure CN118965263A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-modal sentiment analysis method and device, and the method comprises the steps: obtaining to-be-detected multi-modal sentiment data, inputting the to-be-detected sentiment data to a preset multi-modal sentiment analysis model, and enabling the preset multi-modal sentiment analysis model to comprise a multi-modal feature extraction network, a cross attention feature interaction network and a multi-modal feature scoring network; performing feature extraction on the to-be-detected multi-modal emotion data through the multi-modal feature extraction network to generate a multi-modal emotion tensor; inputting the multi-modal emotion tensor into a cross attention feature interaction network for feature interaction, and outputting multi-modal interaction features; performing feature scoring on the multi-modal interaction features and the multi-modal emotion tensor by adopting a multi-modal feature scoring network to generate a target emotion analysis score; the technical problem that the model analysis effect is poor due to the fact that an existing multi-modal sentiment analysis method causes much noise of fused features is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multimodal sentiment analysis method and device. Background Art

[0002] Multimodal sentiment analysis is a popular field in sentiment computing research. Multimodal sentiment analysis usually includes three modalities: vision (image), sound (speech), and transcribed text (spoken text transcribed from video). The reason for dividing different modalities is that the information provided by different modalities is often different, and the processing and modeling methods required for information in different modalities are also different.

[0003] In simple cases, a judgment of emotional attitude can be obtained through only a single modality, such as a review text, a conversation recording, a comment video, etc. However, if only one modality is used for analysis, comprehensive emotional information is often not obtained. Therefore, the multimodal method that combines data from multiple modalities and unifies them into a model has become the mainstream method of sentiment analysis.

[0004] Most of the existing multimodal sentiment analysis methods aggregate multimodal sentiment-related features into discourse-level representations through deep neural network models, and then complete the sentiment scoring based on the discourse-level representations. However, the video and audio modalities have less feature information and more noise, and the powerful fitting ability of deep neural network models is easily interfered by noise information, which makes the fused features also have more noise, resulting in poor model analysis effect. Summary of the invention

[0005] The present invention provides a multimodal sentiment analysis method and device, which are used to solve the technical problem that the existing multimodal sentiment analysis method causes a lot of noise in the fused features, resulting in poor model analysis effect.

[0006] A first aspect of the present invention provides a multimodal sentiment analysis method, comprising:

[0007] Acquire multimodal emotion data to be detected, and input the emotion data to be detected into a preset multimodal emotion analysis model, wherein the preset multimodal emotion analysis model includes a multimodal feature extraction network, a cross-attention feature interaction network, and a multimodal feature scoring network;

[0008] Extracting features of the multimodal emotion data to be detected through the multimodal feature extraction network to generate a multimodal emotion tensor;

[0009] Inputting the multimodal sentiment tensor into the cross-attention feature interaction network for feature interaction, and outputting multimodal interaction features;

[0010] The multimodal feature scoring network is used to perform feature scoring on the multimodal interaction features and the multimodal sentiment tensor to generate a target sentiment analysis score.

[0011] Optionally, the multimodal feature extraction network includes a language representation network, an audio processing network and a face detection network; the multimodal emotion data to be detected includes a text sequence to be detected, an audio to be detected and a video frame image to be detected; the multimodal emotion tensor includes a text tensor, an audio tensor and a video frame tensor; the step of extracting features from the multimodal emotion data to be detected through the multimodal feature extraction network to generate a multimodal emotion tensor includes:

[0012] Extracting text features from the text sequence to be detected through the language representation network to generate a text tensor;

[0013] Using the audio processing network to extract audio features from the audio to be detected, and generating an audio tensor;

[0014] The video frame image to be detected is input into the face detection network to extract video frame features, and a video frame tensor is output.

[0015] Optionally, the cross-attention feature interaction network includes a fully connected layer, an attention mechanism module, a cross-modal information interaction module and a splicing layer; the multimodal interaction feature includes a target audio text interaction feature and a target video frame text interaction feature; the step of inputting the multimodal sentiment tensor into the cross-attention feature interaction network for feature interaction and outputting the multimodal interaction feature includes:

[0016] Performing linear transformation on the text tensor, the audio tensor, and the video frame tensor respectively through a fully connected layer to generate a text feature matrix, an audio feature matrix, and a video frame feature matrix;

[0017] Using an attention mechanism module to perform feature superposition on the audio feature matrix and the audio tensor to generate audio forward normalized features;

[0018] Using an attention mechanism module to perform feature superposition on the video frame feature matrix and the video frame tensor to generate forward normalized features of the video frame;

[0019] Inputting the audio tensor, the query matrix corresponding to the audio forward normalized feature, and the text feature matrix into a cross-modal information interaction module for modal information interaction, and outputting initial audio-text interaction features;

[0020] Inputting the video frame tensor, the query matrix corresponding to the forward normalized features of the video frame, and the text feature matrix into a cross-modal information interaction module for modal information interaction, and outputting initial video frame text interaction features;

[0021] Splicing the initial audio-text interaction feature and the audio forward normalization feature through a splicing layer, and outputting an audio-text splicing feature;

[0022] Splicing the initial video frame text interaction feature and the video frame forward normalization feature through a splicing layer, and outputting a video frame text splicing feature;

[0023] A fully connected layer is used to perform linear transformation on the audio-text splicing features and the video-frame-text splicing features, respectively, and a target audio-text interaction feature and a target video-frame-text interaction feature are output.

[0024] Optionally, the attention mechanism module includes a self-attention mechanism layer, a normalization layer and a forward network layer; the audio feature matrix includes an audio query matrix, an audio key value matrix and an audio value matrix; the step of using the attention mechanism module to perform feature superposition on the audio feature matrix and the audio tensor to generate audio forward normalized features includes:

[0025] Performing matrix multiplication operation on the audio query matrix, the audio key value matrix and the audio value matrix through a self-attention mechanism layer, and outputting an audio self-attention feature matrix;

[0026] Performing an addition operation on the audio self-attention feature matrix and the audio tensor, and outputting a first audio superposition feature;

[0027] Normalizing the first audio superposition feature through a normalization layer to generate an audio normalization feature;

[0028] Using a forward network layer to perform nonlinear transformation on the audio normalization features and output audio nonlinear features;

[0029] Performing an addition operation on the audio nonlinear feature and the audio normalization feature, and outputting a second audio superposition feature;

[0030] The second audio superposition feature is normalized by using a normalization layer to generate an audio forward normalized feature.

[0031] Optionally, the cross-modal information interaction module includes a cross-attention mechanism layer, a normalization layer and a forward network layer; the step of inputting the audio tensor, the query matrix corresponding to the audio forward normalized feature and the text feature matrix into the cross-modal information interaction module for modal information interaction and outputting the initial audio-text interaction feature includes:

[0032] Performing matrix multiplication operation on the query matrix corresponding to the audio forward normalized feature and the text key value matrix and the text value matrix in the text feature matrix through a cross attention mechanism layer, and outputting an audio text cross attention matrix;

[0033] Performing an addition operation on the audio-text cross-attention matrix and the audio tensor, and outputting a first audio-text superposition feature;

[0034] Normalizing the first audio text superposition feature through a normalization layer to generate an audio text normalized feature;

[0035] Using a forward network layer to perform nonlinear transformation on the normalized features of the audio text, and outputting the nonlinear features of the audio text;

[0036] Performing an addition operation on the audio text nonlinear feature and the audio text normalized feature, and outputting a second audio text superposition feature;

[0037] The second audio-text superposition feature is normalized using a normalization layer to generate an initial audio-text interaction feature.

[0038] Optionally, the multimodal feature scoring network includes a feature fusion module and a sentiment analysis module; the step of using the multimodal feature scoring network to perform feature scoring on the multimodal interaction features and the multimodal sentiment tensor to generate a target sentiment analysis score includes:

[0039] The target audio-text interaction feature, the target video frame text interaction feature and the text tensor are spliced ​​by the feature fusion module to generate a multimodal splicing feature;

[0040] Performing two linear transformations on the multimodal splicing features and outputting multimodal fusion features;

[0041] The sentiment analysis module is used to perform linear transformation on the multimodal fusion features to generate a target sentiment analysis score.

[0042] Optionally, before the step of obtaining the multimodal emotion data to be detected and inputting the emotion data to be detected into a preset multimodal emotion analysis model, the step includes:

[0043] Acquire a multimodal emotion data set to be trained, input the multimodal emotion data set to be trained into an initial multimodal emotion analysis model, and output a multimodal emotion tensor set to be trained, a multimodal interaction feature set to be trained, a multimodal fusion feature set to be trained, and a emotion analysis score set to be trained;

[0044] A preset loss value function is used to calculate a target loss value according to the multimodal emotion tensor set to be trained, the multimodal interaction feature set to be trained, the multimodal fusion feature set to be trained, and the emotion analysis score set to be trained;

[0045] Determining whether the target loss value converges;

[0046] If the target loss value converges, the initial multimodal sentiment analysis model is used as the trained preset multimodal sentiment analysis model.

[0047] A second aspect of the present invention provides a multimodal sentiment analysis device, comprising:

[0048] An acquisition module, used for acquiring multimodal emotion data to be detected, and inputting the emotion data to be detected into a preset multimodal emotion analysis model, wherein the preset multimodal emotion analysis model includes a multimodal feature extraction network, a cross-attention feature interaction network, and a multimodal feature scoring network;

[0049] A generating module, used for performing feature extraction on the multimodal emotion data to be detected through the multimodal feature extraction network to generate a multimodal emotion tensor;

[0050] An output module, used for inputting the multimodal sentiment tensor into the cross-attention feature interaction network for feature interaction, and outputting multimodal interaction features;

[0051] A module is used to use the multimodal feature scoring network to perform feature scoring on the multimodal interaction features and the multimodal sentiment tensor to generate a target sentiment analysis score.

[0052] A third aspect of the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the multimodal sentiment analysis method as described in any one of the above items.

[0053] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the steps of the multimodal sentiment analysis method as described in any one of the above items.

[0054] It can be seen from the above technical solutions that the present invention has the following advantages:

[0055] The above technical scheme of the present invention provides a multimodal sentiment analysis method, which first obtains multimodal sentiment data to be detected, and inputs the sentiment data to be detected into a preset multimodal sentiment analysis model, wherein the preset multimodal sentiment analysis model includes a multimodal feature extraction network, a cross-attention feature interaction network and a multimodal feature scoring network; then, the multimodal feature extraction network is used to perform feature extraction on the multimodal sentiment data to be detected, and a multimodal sentiment tensor is generated; the multimodal sentiment tensor is input into the cross-attention feature interaction network for feature interaction, and a multimodal interaction feature is output; finally, a multimodal feature scoring network is used to perform feature scoring on the multimodal interaction feature and the multimodal sentiment tensor, and a target sentiment analysis score is generated; based on the above scheme, the process of performing feature interaction on the multimodal sentiment tensor through the cross-attention feature interaction network and outputting the multimodal interaction feature can pay more attention to the interaction between multimodal sequences, increase effective features, thereby reducing the influence of noise, and further improving the model analysis effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0057] Figure 1 A flowchart of the steps of a multimodal sentiment analysis method provided by an embodiment of the present invention;

[0058] Figure 2 A flowchart of another multimodal sentiment analysis method provided by an embodiment of the present invention;

[0059] Figure 3 A schematic diagram of the structure of a preset multimodal sentiment analysis model provided in an embodiment of the present invention;

[0060] Figure 4 A schematic diagram of the structure of an attention mechanism module and a cross-modal information interaction module provided in an embodiment of the present invention;

[0061] Figure 5 A structural block diagram of a multimodal sentiment analysis device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The embodiments of the present invention provide a multimodal sentiment analysis method and device, which are used to solve the technical problem that the existing multimodal sentiment analysis method causes a lot of noise in the fused features, resulting in poor model analysis effect.

[0063] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] See also Figure 1 , Figure 1 A flowchart of the steps of a multimodal sentiment analysis method provided by an embodiment of the present invention.

[0065] The present invention provides a multimodal sentiment analysis method, comprising:

[0066] Step 101: obtain multimodal emotion data to be detected, and input the emotion data to be detected into a preset multimodal emotion analysis model, wherein the preset multimodal emotion analysis model includes a multimodal feature extraction network, a cross-attention feature interaction network, and a multimodal feature scoring network.

[0067] The pre-built multimodal sentiment analysis model includes a multimodal feature extraction network, a cross-attention feature interaction network, and a multimodal feature scoring network.

[0068] It should be noted that the multimodal emotional data to be detected includes a text sequence to be detected, an audio to be detected, and a video frame image to be detected; wherein the text sequence to be detected is a text sequence obtained after the text to be detected is successively segmented and marked and masked. Specifically, the text to be detected is first segmented and marked by the marker (Tokenizer) in the pre-trained model to obtain multiple text sequences, and then the text sequences that do not meet the preset text sequence length in all text sequences are masked and filled to obtain the final multiple text sequences to be detected. For example, the pre-trained model used in the present invention is Bert-base-uncased. First, the Tokenizer in the pre-trained model is used to segment the text, and a start tag "[CLS]" is inserted before each sentence after the segmentation, and then an end tag "[SEP]" is inserted after each sentence. The present invention sets the preset text sequence length to 50, and the part of the text sequence whose sentence length is less than 50 is filled with a mask, and finally multiple text sequences to be detected are obtained.

[0069] Furthermore, the video frame image to be detected is the information of the frame extracted from the video to be detected at a set frequency. After obtaining multiple video frame images to be detected, a text sequence to be detected, an audio to be detected and a video frame image to be detected are input into a preset multimodal sentiment analysis model as a group of multimodal emotion data to be detected for processing; wherein, these three parts: the text to be detected, the audio to be detected and the video to be detected are derived from the video stream.

[0070] In this embodiment, multimodal emotion data to be detected is obtained, and the emotion data to be detected is input into a preset multimodal emotion analysis model, which includes a multimodal feature extraction network, a cross-attention feature interaction network and a multimodal feature scoring network.

[0071] Step 102: extract features from the multimodal emotion data to be detected through a multimodal feature extraction network to generate a multimodal emotion tensor.

[0072] The multimodal sentiment tensor includes text tensor, audio tensor and video frame tensor.

[0073] The multimodal feature extraction network includes a language representation network, an audio processing network, and a face detection network. The language representation network is the Bert model (Bidirectional Encoder Representations from Transformers). Bert is a pre-trained language model based on the Transformer architecture. Compared with the traditional unidirectional language model, Bert uses a bidirectional Transformer encoder, which can better understand the semantics and context in the context. Bert adopts a pre-training-fine-tuning strategy. In the pre-training stage, Bert uses a large-scale text corpus for unsupervised learning and learns text representation through tasks such as the Masked Language Model (MLM) and Next Sentence Prediction (NSP). The pre-trained BERT model can already have a good representation of the text. In the fine-tuning stage, Bert can be supervised on specific tasks. It only needs to be fine-tuned under the pre-trained parameters to adapt to the needs of specific fields or tasks, and can efficiently extract emotional features from texts. The audio processing network is Librosa, a Python library for analyzing audio and music. The face detection network is the face detection algorithm (OpenFace).

[0074] It should be noted that, for the text part, the present invention uses Bert to extract its features, and extracts features of the acquired text sequence to be detected through the BERT model, and outputs features with a tensor dimension of [50,768], that is, a text tensor, where 768 is the feature dimension of a single word in the text;

[0075] Furthermore, for the audio part, the present invention uses Librosa to perform feature extraction, and each speech after extraction is converted into a tensor with a feature dimension of [500,5], that is, an audio tensor, where 500 is the sequence length of the audio and 5 is the feature dimension.

[0076] Furthermore, for the video part, the present invention uses OpenFace (face detection algorithm) to perform feature extraction on the video frame image to be detected. Specifically, the face in the video frame image to be detected is detected by the face detection algorithm, and the face key points are detected. The facial features are calculated from the key point positions to obtain features with a tensor dimension of [375,20], that is, a video frame tensor, where 375 is the video sequence length and 20 is the feature dimension.

[0077] In this embodiment, feature extraction is performed on the multimodal emotion data to be detected through a multimodal feature extraction network to generate a multimodal emotion tensor.

[0078] Step 103: Input the multimodal sentiment tensor into the cross-attention feature interaction network for feature interaction, and output the multimodal interaction feature.

[0079] The multimodal interaction features include target audio text interaction features and target video frame text interaction features.

[0080] The cross-attention feature interaction network includes a fully connected layer, an attention mechanism module, a cross-modal information interaction module and a splicing layer; among them, the attention mechanism module (Transformer module) is mainly composed of a multi-head attention mechanism (Multi-HeadAttention) and a forward network (Feed Forward), which can capture the internal dependencies of the sequence and collect time information in the input of the time series.

[0081] Specifically, the text tensor, audio tensor and video frame tensor are linearly transformed through the fully connected layer to generate a text feature matrix, an audio feature matrix and a video frame feature matrix; wherein the text feature matrix includes a text query matrix, a text key-value matrix and a text value matrix; the audio feature matrix includes an audio query matrix, an audio key-value matrix and an audio value matrix; the video frame feature matrix includes a video frame query matrix, a video frame key-value matrix and a video frame value matrix.

[0082] Furthermore, an attention mechanism module is used to perform feature superposition on the audio feature matrix and the audio tensor to generate audio forward normalized features; an attention mechanism module is used to perform feature superposition on the video frame feature matrix and the video frame tensor to generate video frame forward normalized features; the query matrix and the text feature matrix corresponding to the audio tensor and the audio forward normalized features are input into the cross-modal information interaction module for modal information interaction, and the initial audio-text interaction features are output; the query matrix and the text feature matrix corresponding to the video frame tensor and the video frame forward normalized features are input into the cross-modal information interaction module for modal information interaction, and the initial video text interaction features are output. frame-text interaction features; the initial audio-text interaction features and the audio forward normalization features are spliced ​​through the splicing layer to output the audio-text splicing features; the initial video frame-text interaction features and the video frame forward normalization features are spliced ​​through the splicing layer to output the video frame-text splicing features; the fully connected layer is used to linearly transform the audio-text splicing features and the video frame-text splicing features respectively, and the target audio-text interaction features and the target video frame-text interaction features are output; among them, the query matrix corresponding to the audio forward normalization features and the query matrix corresponding to the video frame forward normalization features can both be obtained by linear transformation through the fully connected layer.

[0083] In this embodiment, the multimodal sentiment tensor is input into the cross-attention feature interaction network for feature interaction, and the multimodal interaction feature is output.

[0084] Step 104: Use a multimodal feature scoring network to perform feature scoring on the multimodal interaction features and the multimodal sentiment tensor to generate a target sentiment analysis score.

[0085] The multimodal feature scoring network includes a feature fusion module and a sentiment analysis module.

[0086] Specifically, the target audio-text interaction features, the target video frame text interaction features and the text tensor are spliced ​​through the feature fusion module to generate multimodal splicing features; the multimodal splicing features are linearly transformed twice to output multimodal fusion features; the sentiment analysis module is used to linearly transform the multimodal fusion features to generate the target sentiment analysis score.

[0087] It is worth mentioning that the feature fusion module splices the three modal features of the target audio-text interaction feature, the target video frame text interaction feature and the text tensor. The text tensor comes from the output of the Bert model. The present invention uses the first sequence feature in the Bert output, that is, the text tensor, as the overall text feature. The audio and video modal features, that is, the target audio-text interaction feature and the target video frame text interaction feature come from the Transformer module and the cross-modal emotion information interaction module. The outputs of the two modules are spliced ​​(Concat), and the features output after linear transformation can not only integrate audio and video into text features, but also retain single-modal features. In a hurry, the spliced ​​three-modal features (multi-modal splicing features) are linearly transformed twice to extract the final three-modal fusion features (multi-modal fusion features); the multi-modal fusion features are linearly transformed once to obtain the final emotion score (target emotion analysis score).

[0088] In this embodiment, a multimodal feature scoring network is used to perform feature scoring on multimodal interaction features and multimodal sentiment tensors to generate a target sentiment analysis score.

[0089] In an embodiment of the present invention, the present invention provides a multimodal sentiment analysis method, which first obtains multimodal sentiment data to be detected, and inputs the sentiment data to be detected into a preset multimodal sentiment analysis model, wherein the preset multimodal sentiment analysis model includes a multimodal feature extraction network, a cross-attention feature interaction network, and a multimodal feature scoring network; then, the multimodal feature extraction network is used to perform feature extraction on the multimodal sentiment data to be detected, and a multimodal sentiment tensor is generated; the multimodal sentiment tensor is input into the cross-attention feature interaction network for feature interaction, and a multimodal interaction feature is output; finally, a multimodal feature scoring network is used to perform feature scoring on the multimodal interaction feature and the multimodal sentiment tensor, and a target sentiment analysis score is generated; based on the above scheme, the process of performing feature interaction on the multimodal sentiment tensor through the cross-attention feature interaction network and outputting the multimodal interaction feature can pay more attention to the interaction between multimodal sequences, increase effective features, thereby reducing the influence of noise, and further improving the model analysis effect.

[0090] See also Figure 2 , Figure 2 A flowchart of the steps of another multimodal sentiment analysis method provided by an embodiment of the present invention.

[0091] The present invention provides a multimodal sentiment analysis method, comprising:

[0092] Step 201, obtain a multimodal sentiment data set to be trained, input the multimodal sentiment data set to be trained into an initial multimodal sentiment analysis model, and output a multimodal sentiment tensor set to be trained, a multimodal interaction feature set to be trained, a multimodal fusion feature set to be trained, and a sentiment analysis score set to be trained.

[0093] The multimodal emotion tensor set to be trained includes multiple text tensors to be trained, multiple audio tensors to be trained, and multiple video frame tensors to be trained.

[0094] The multimodal interaction feature set to be trained includes a plurality of audio-text interaction features to be trained and a plurality of video-frame-text interaction features to be trained.

[0095] The multimodal fusion features to be trained include multiple multimodal fusion features to be trained.

[0096] The sentiment analysis score set to be trained includes multiple sentiment analysis scores to be trained.

[0097] It should be noted that the multimodal emotion dataset to be trained includes multiple text sequences to be trained, multiple audios to be trained and multiple video frame images to be trained; wherein, the text sequence to be trained is a text sequence for network training obtained after the text to be trained is successively marked by word segmentation and mask filling; the video frame image to be trained is a video frame image for network training obtained by extracting frame information of the video to be trained at a set frequency. These three parts: the text sequence to be trained, the audio to be trained and the video frame image to be trained come from the video stream division.

[0098] In this embodiment, a multimodal emotion data set to be trained is obtained, the multimodal emotion data set to be trained is input into an initial multimodal emotion analysis model, and a multimodal emotion tensor set to be trained, a multimodal interaction feature set to be trained, a multimodal fusion feature set to be trained, and a emotion analysis score set to be trained are output.

[0099] Step 202: Calculate the target loss value using a preset loss value function based on the multimodal sentiment tensor set to be trained, the multimodal interaction feature set to be trained, the multimodal fusion feature set to be trained, and the sentiment analysis score set to be trained.

[0100] The preset loss value functions include the initial unimodal loss function, the initial unimodal and multimodal fusion feature scoring function, the loss function that maximizes the MI lower bound between unimodal features, the loss function that maximizes the MI lower bound between unimodal and multimodal fusion features, and the task loss function.

[0101] It should be noted that, in the network training stage, the mutual information maximization module proposed in the present invention adopts a preset loss value function to calculate the target loss value according to the multimodal emotion tensor set to be trained, the multimodal interaction feature set to be trained, the multimodal fusion feature set to be trained and the emotion analysis score set to be trained output by the untrained initial multimodal emotion analysis model; wherein, the mutual information maximization module is divided into two parts: maximizing the mutual information between single modalities and maximizing the mutual information between single modality and multimodal fusion features.

[0102] Specifically, the mutual information between single modalities is maximized, and the dependencies between the three modalities can be mined, and the modality-specific random noise irrelevant to the task can be filtered out. The input of this part is all the text tensors to be trained, all the audio text interaction features to be trained, and all the video frame text interaction features to be trained. The present invention selects an easy-to-handle MI lower bound and uses the variational counterpart q(y|x) to approximate the true value conditional distribution p(y|x), specifically:

[0103]

[0104] Where H(Y) is the differential entropy of Y; Y is the second sample set; X is the first sample set; I(X; Y) is the value of MI; E p(x,y( [logq(y|x)] is the expectation under the joint distribution p(x,y); q(y|x) is the variational counterpart; I BA is a single-modal loss value, which includes the video frame text loss value and audio-text loss

[0105] Furthermore, the present invention uses a Gaussian mixture model (GMM) to solve the problem, selects sentiment polarity (non-negative / negative) as the classification criterion, and establishes two normal distributions for the two categories. S ∈D train The estimated parameters are:

[0106]

[0107] Where c is the preset polarity category, c∈{pos,neg}; is the mean vector corresponding to the second sample set of category c, the mean vector corresponding to the second sample set includes the mean vector corresponding to the second audio text sample subset Yt,a of category c and the mean vector corresponding to the second video frame text sample subset Yt,v of category c; N c is the number of samples of the second sample set of category c, the number of samples of the second sample set including the number of samples of category c of the second audio text sample subset Yt,a of category c and the number of samples of the second video frame text sample subset Yt,v; is the i-th second sample in the second sample set in category c; is a covariance matrix of category c, the covariance matrix including a covariance matrix of category pos and a covariance matrix of category neg corresponding to the second audio text sample subset Yt,a, and a covariance matrix of category pos and a covariance matrix of category neg corresponding to the second video frame text sample subset Yt,v; is the transpose of the mean vector of category c; ⊙ is the element-by-element multiplication.

[0108] Furthermore, taking the lower bound of GMM entropy as an approximation, the entropy term of the MI lower bound is:

[0109]

[0110] Where H(Y) is the differential entropy of Y; det(∑ 1 ) is the determinant of the covariance matrix of category pos, which includes the determinant of the covariance matrix of category pos corresponding to the second audio text sample subset Yt,a and the determinant of the covariance matrix of category pos corresponding to the second video frame text sample subset Yt,v; det(∑ 2 ) is the determinant of the covariance matrix of category neg, which includes the determinant of the covariance matrix of category neg corresponding to the second audio text sample subset Yt,a and the determinant of the covariance matrix of category neg corresponding to the second video frame text sample subset Yt,v.

[0111] Based on the above foundation, the present invention implicitly assumes that the prior probabilities of the two categories are equal. Combining the above formula, the initial unimodal loss function is obtained:

[0112]

[0113] Wherein, c is a preset polarity category, c∈{pos,neg}; H(Y) is the differential entropy of Y; Y is a second sample set, and the second sample set includes a second audio text sample subset Yt,a and a second video frame text sample subset Yt,v; X is a first sample set; I(X;Y) is the value of MI; E p(x,y) [logq(y|x)] is the expectation under the joint distribution p(x,y); q(y|x) is the variational counterpart; I BA is the unimodal loss value, which includes the video frame text loss value and audio-text loss is the mean vector corresponding to the second sample set of category c, and the mean vector corresponding to the second sample set includes the mean vector corresponding to the second audio text sample subset Yt,a of category c and the mean vector corresponding to the second video frame text sample subset Yt,a of category c. t,vThe corresponding mean vector; N c is the number of samples of the second sample set of category c, the number of samples of the second sample set including the number of samples of category c of the second audio text sample subset Yt,a of category c and the number of samples of the second video frame text sample subset Yt,v; is the i-th second sample in the second sample set in category c; is a covariance matrix of category c, the covariance matrix including a covariance matrix of category pos and a covariance matrix of category neg corresponding to the second audio text sample subset Yt,a, and a covariance matrix of category pos and a covariance matrix of category neg corresponding to the second video frame text sample subset Yt,v; is the transpose of the mean vector of category c; ⊙ is element-by-element multiplication; det(∑ 1 ) is the determinant of the covariance matrix of category pos; det(∑ 2 ) is the determinant of the covariance matrix of category neg; x is the first sample, that is, the text tensor to be trained; y is the second sample, and the second sample includes the second audio text sample yt,a and the second video frame text sample yt,v.

[0114] Furthermore, each audio text interaction feature to be trained is classified according to the preset polarity category c, the text tensor set to be trained is used as the first sample set X, and the second audio text sample subset Yt,a is constructed using all the classified audio text interaction features to be trained. Each text tensor to be trained is used as the first sample, and each audio text interaction feature to be trained after classification is used as the second sample, and they are respectively substituted into the initial unimodal loss function to calculate multiple audio text loss values. Then take the average of all audio text loss values ​​to get the audio text loss mean

[0115] Further, we calculate After that, the text interaction features of each video frame to be trained are classified according to the preset polarity category c, the text tensor set to be trained is used as the first sample set X, and the second video frame text sample subset Yt,v is constructed using all the classified video frame text interaction features to be trained. Each text tensor to be trained is used as the first sample, and each classified video frame text interaction feature to be trained is used as the second sample, and they are substituted into the initial unimodal loss function to calculate the video frame text loss value Then take the average of all video frame text loss values ​​to get the video frame text loss mean

[0116] Furthermore, the loss function that maximizes the lower bound of MI between single modalities is used to calculate the loss value L that maximizes the lower bound of MI between single modalities based on the mean loss of video frame text and the mean loss of audio text.BA , the loss function for maximizing the lower bound of MI between the single modalities is specifically:

[0117] Furthermore, the mutual information between the unimodal and multimodal fusion features is maximized, and the optimization goal is to strengthen the dependency between the unimodal and fusion features, so as to supervise the multimodal features and not to deviate from the unimodal features. The generation path from the unimodal features to the fusion features, therefore, the present invention expects to have a path to construct the unimodal features from the fusion features. The input of this part is all the text tensors to be trained (the set of text tensors to be trained), all the audio text interaction features to be trained and all the video frame text interaction features to be trained, and all the multimodal fusion features to be trained (the set of multimodal fusion features to be trained). The present invention uses a scoring function acting on the normalized prediction vector and the true value vector to measure their correlation, and then combined with the predicted scoring function, it is possible to obtain the initial unimodal and multimodal fusion feature scoring function, specifically:

[0118]

[0119] in, To generate a multimodal fusion feature set to be trained The predicted unit-length vector includes a unit-length vector corresponding to the third text sample, a unit-length vector corresponding to the third audio-text interaction sample, and a unit-length vector corresponding to the third video frame text interaction sample; is the unit length vector of the i-th third sample in the third sample set, the third sample set includes a third text sample set, a third audio-text interaction sample set and a third video frame text interaction sample set, and the unit length vector of the third sample includes a unit length vector of the third text sample, a unit length vector of the third audio-text interaction sample and a unit length vector of the third video frame text interaction sample; To generate h according to the multimodal fusion feature set to be trained m predictions; is the i-th third sample, the third sample includes the third text sample Third audio text interaction sample Sample interaction with the third video frame text is the score predicted by the i-th third sample, where the score predicted by the third sample includes the score predicted by the third text sample, the score predicted by the third audio-text interaction sample, and the score predicted by the third video frame-text interaction sample; To generate h according to the multimodal fusion feature set to be trained m The transpose of the unit-length vector of predictions; ||·|| 2 is the Euclidean norm; It is a deep learning neural network that can generate Prediction of H m is a third sample set, the third sample set including a third text sample set The third audio text interaction sample collection A collection of text interaction samples with the third video frame Z is the fourth sample set, i.e., the multimodal fusion feature set to be trained; L N (Z,H m ) is the predicted score value, which includes the text fusion predicted score value Audio fusion prediction score Fusion prediction score with video frame is the predicted score and value corresponding to the third sample set, and the predicted score and value include the third text sample set Corresponding predicted scores and values, third audio text interaction sample set The corresponding predicted scores and values ​​and the set of text interaction samples in the third video frame The corresponding predicted scores and values; is the jth third sample in the third sample set.

[0120] Furthermore, the text tensor set to be trained is used as the third text sample set Each to-be-trained text tensor in the to-be-trained text tensor set is used as the third text sample The multimodal fusion feature set to be trained is used as the fourth sample set Z, and is substituted into the initial single-modal and multimodal fusion feature scoring functions respectively to calculate multiple text fusion prediction score values. Then take the average of all text fusion prediction scores to get the text fusion prediction score average

[0121] Further, all the audio-text interaction features to be trained are used to construct a third audio-text interaction sample set Each audio-text interaction feature to be trained is used as the third audio-text interaction sample The multimodal fusion feature set to be trained is used as the fourth sample set Z, and is substituted into the initial single-modal and multimodal fusion feature scoring functions respectively to calculate multiple audio fusion prediction score values. Then take the average of all audio fusion prediction scores to get the average audio fusion prediction score

[0122] Furthermore, all the video frame text interaction features to be trained are used to construct a third video frame text interaction sample set The text interaction features of each video frame to be trained are used as the text interaction samples of the third video frame. The multimodal fusion feature set to be trained is used as the fourth sample set Z, and is substituted into the initial single-modal and multimodal fusion feature scoring functions respectively to calculate the fusion prediction score values ​​of multiple video frames. Then take the average of all the video frame fusion prediction scores to get the video frame fusion prediction score average

[0123] Furthermore, the MI lower bound between the unimodal and multimodal fusion features is used to maximize the loss function and predict the mean score according to the text fusion. Audio fusion prediction score mean and the mean score of video frame fusion prediction Calculate the MI lower bound between single-modal and multi-modal fusion features to maximize the loss value L CPC , where the MI lower bound maximization loss function between the single modality and multimodal fusion features is specifically:

[0124] Furthermore, L is obtained through the mutual information maximization module CPC and L BA Two loss values, and then use the task loss function according to the sentiment analysis score set to be trained Calculate the task loss value L task , where the task loss function is specifically: Among them, y is the true sentiment analysis score set, and MAE(·) is the mean absolute error loss function.

[0125] Furthermore, the task loss value L obtained through the above steps is task , the MI lower bound between single modalities maximizes the loss value L BA , the MI lower bound between single-modal and multi-modal fusion features maximizes the loss value L CPC Perform weighted sum to get the target loss value. The specific process is as follows:

[0126] L=L task +αL BA +βL CPC ;

[0127] Where L is the target loss value; α is the first hyperparameter, set to 0.1; β is the second hyperparameter, set to 0.1; L task is the task loss value; L BA Maximize the loss value for the MI lower bound between single modalities; L CPC Maximize the loss value for the MI lower bound between multimodal fusion features.

[0128] In this embodiment, a preset loss value function is used to calculate the target loss value based on the multimodal sentiment tensor set to be trained, the multimodal interaction feature set to be trained, the multimodal fusion feature set to be trained, and the sentiment analysis score set to be trained.

[0129] Step 203: Determine whether the target loss value converges.

[0130] In this embodiment, it is determined whether the target loss value converges.

[0131] Step 204: If the target loss value converges, the initial multimodal sentiment analysis model is used as the trained preset multimodal sentiment analysis model.

[0132] It should be noted that if the target loss value converges, the initial multimodal sentiment analysis model is used as the trained preset multimodal sentiment analysis model; if the target loss value does not converge, the model parameters of the initial multimodal sentiment analysis model are updated according to the target loss value, a new initial multimodal sentiment analysis model is determined, and the process jumps to execute step 201 until the target loss value converges, and the initial multimodal sentiment analysis model determined when the target loss value converges is used as the trained preset multimodal sentiment analysis model.

[0133] In this embodiment, if the target loss value converges, the initial multimodal sentiment analysis model is used as the trained preset multimodal sentiment analysis model.

[0134] Step 205: obtain the multimodal emotion data to be detected, and input the emotion data to be detected into a preset multimodal emotion analysis model, wherein the preset multimodal emotion analysis model includes a multimodal feature extraction network, a cross-attention feature interaction network, and a multimodal feature scoring network.

[0135] See also Figure 3 The preset multimodal sentiment analysis model proposed in the present invention includes a multimodal feature extraction network, a cross-attention feature interaction network and a multimodal feature scoring network; wherein the multimodal feature extraction network includes a language representation network (Bert), an audio processing network (Librosa) and a face detection network (OpenFace); the cross-attention feature interaction network includes a fully connected layer, an attention mechanism module, a cross-modal information interaction module and a splicing layer; the attention mechanism module includes a self-attention mechanism layer, a normalization layer and a forward network layer; the cross-modal information interaction module includes a cross-attention mechanism layer, a normalization layer and a forward network layer; the multimodal feature scoring network includes a feature fusion module and a sentiment analysis module.

[0136] In this embodiment, multimodal emotion data to be detected is obtained, and the emotion data to be detected is input into a preset multimodal emotion analysis model, which includes a multimodal feature extraction network, a cross-attention feature interaction network and a multimodal feature scoring network.

[0137] Step 206: extract features from the multimodal emotion data to be detected through a multimodal feature extraction network to generate a multimodal emotion tensor.

[0138] The multimodal feature extraction network includes a language representation network, an audio processing network, and a face detection network.

[0139] The multimodal emotion data to be detected includes text sequences to be detected, audio to be detected, and video frame images to be detected.

[0140] The multimodal sentiment tensor includes text tensor, audio tensor and video frame tensor.

[0141] Further, step 206 may include the following sub-steps:

[0142] S61, extracting text features from the text sequence to be detected through a language representation network to generate a text tensor;

[0143] S62, using an audio processing network to extract audio features of the audio to be detected, and generating an audio tensor;

[0144] S63: Input the video frame image to be detected into the face detection network to extract video frame features, and output the video frame tensor.

[0145] In this embodiment, feature extraction is performed on the multimodal emotion data to be detected through a multimodal feature extraction network to generate a multimodal emotion tensor.

[0146] Step 207: input the multimodal sentiment tensor into the cross-attention feature interaction network for feature interaction, and output the multimodal interaction feature.

[0147] The cross-attention feature interaction network includes a fully connected layer, an attention mechanism module, a cross-modal information interaction module and a splicing layer.

[0148] The multimodal interaction features include target audio text interaction features and target video frame text interaction features.

[0149] Further, step 207 may include the following sub-steps:

[0150] S71, linearly transform the text tensor, the audio tensor, and the video frame tensor respectively through a fully connected layer to generate a text feature matrix, an audio feature matrix, and a video frame feature matrix;

[0151] S72, using an attention mechanism module to perform feature superposition on the audio feature matrix and the audio tensor to generate audio forward normalized features;

[0152] Further, S72 may include the following sub-steps:

[0153] S721, performing matrix multiplication operation on the audio query matrix, the audio key value matrix and the audio value matrix through the self-attention mechanism layer, and outputting the audio self-attention feature matrix;

[0154] S722, performing an addition operation on the audio self-attention feature matrix and the audio tensor, and outputting a first audio superposition feature;

[0155] S723, normalizing the first audio superposition feature through a normalization layer to generate an audio normalization feature;

[0156] S724, using a forward network layer to perform nonlinear transformation on the audio normalization feature, and outputting the audio nonlinear feature;

[0157] S725, performing an addition operation on the audio nonlinear feature and the audio normalization feature, and outputting a second audio superposition feature;

[0158] S726. Use a normalization layer to normalize the second audio superposition feature to generate an audio forward normalized feature.

[0159] It should be noted that the multi-head attention mechanism is essentially multiple parallel attention mechanisms. The attention mechanism module (Transformer module) uses the self-attention mechanism, which is composed of three matrices: Q (query), K (key value), and V (value). The three matrices are obtained by linear transformation (fully connected layer) of the input, that is, the text tensor, audio tensor, and video frame tensor are respectively linearly transformed (fully connected layer) to obtain the corresponding Q (query), K (key value), and V (value) matrices; the linear transformation process is specifically as follows:

[0160]

[0161] Where Q is the query matrix, which includes the text query matrix, the audio query matrix and the video frame query matrix; W q is the weight of the first fully connected layer; W k is the weight of the second fully connected layer; W v is the weight of the third fully connected layer; X is the multimodal sentiment tensor; K is the key-value matrix, which includes the text key-value matrix, the audio key-value matrix and the video frame key-value matrix; V is the value matrix, which includes the text value matrix, the audio value matrix and the video frame value matrix.

[0162] Furthermore, the audio query matrix, audio key matrix, and audio value matrix are multiplied by the self-attention mechanism layer to output the audio self-attention feature matrix. Figure 4 In (a), in the Transformer module, the input and output of the multi-head attention mechanism and the input and output of the forward network constitute two residual networks, and the outputs of the two residual networks are subjected to layer normalization (LayerNorm); exemplarily: the feature processing process of the self-attention mechanism layer is: matrix multiplication is performed on Q and K to calculate the similarity matrix S of the Q and K vectors, S is divided by a specific constant α and activated by the Softmax function to obtain the attention score matrix S', and finally S' is matrix multiplied with V to obtain the attention mechanism output, that is, the self-attention feature matrix. The process can be expressed as:

[0163]

[0164] Among them, S is the similarity matrix; Q is the query matrix; K is the key-value matrix; T is the transpose operation; α is a specific constant; S' is the attention score matrix; Attention(Q,K,V) is the self-attention feature matrix; V is the value matrix; in_dim is the feature dimension of the input multimodal sentiment tensor.

[0165] S73, using an attention mechanism module to perform feature superposition on the video frame feature matrix and the video frame tensor to generate a forward normalized feature of the video frame;

[0166] It is worth mentioning that the principle of the steps of using the attention mechanism module to perform feature superposition on the video frame feature matrix and the video frame tensor is consistent with the principle of the steps of using the attention mechanism module to perform feature superposition on the audio feature matrix and the audio tensor, and the present invention will not go into details.

[0167] S74, inputting the audio tensor, the query matrix corresponding to the audio forward normalized feature, and the text feature matrix into the cross-modal information interaction module for modal information interaction, and outputting the initial audio-text interaction feature;

[0168] Further, S74 may include the following sub-steps:

[0169] S741, performing matrix multiplication operation on the text key value matrix and the text value matrix in the query matrix and the text feature matrix corresponding to the audio forward normalized feature through the cross attention mechanism layer, and outputting the audio text cross attention matrix;

[0170] S742, performing an addition operation on the audio text cross attention matrix and the audio tensor, and outputting a first audio text superposition feature;

[0171] S743, normalizing the first audio text superposition feature through a normalization layer to generate an audio text normalized feature;

[0172] S744, using a forward network layer to perform nonlinear transformation on the normalized features of the audio text, and outputting the nonlinear features of the audio text;

[0173] S745, performing an addition operation on the audio text nonlinear feature and the audio text normalization feature, and outputting a second audio text superposition feature;

[0174] S746. Use a normalization layer to normalize the second audio-text superposition feature to generate an initial audio-text interaction feature.

[0175] Please note that Figure 4 In (b), the cross-modal emotional information interaction module is also composed of a multi-head attention mechanism and a forward network, which enables one modality to obtain information from another modality. Since text is absolutely dominant among the three modalities, there is no need to make text features accept other modal information in the present invention, so only audio and video need to obtain text features.

[0176] Furthermore, unlike the Transformer module, in the multi-head attention mechanism, this module uses the cross-attention mechanism, such as Figure 4 In (b), this attention mechanism is the same as the self-attention mechanism, which is composed of Q, K, and V matrices, and the operation method is the same. The difference is that the Q, K, and V matrices of the self-attention mechanism are all derived from modality A, while the Q of the cross-modal self-attention mechanism is derived from modality A, and K and V are derived from another modality B. Therefore, in the self-attention mechanism, the similarity matrix S represents the similarity between different sequences in the same modality input, while the similarity matrix S in the cross-modal self-attention mechanism expresses the similarity between different sequences between different modality inputs.

[0177] S75, inputting the video frame tensor, the query matrix corresponding to the forward normalized features of the video frame, and the text feature matrix into the cross-modal information interaction module for modal information interaction, and outputting the initial video frame text interaction features;

[0178] It is worth mentioning that the principle of the step of inputting the query matrix and text feature matrix corresponding to the video frame tensor and the forward normalized features of the video frame into the cross-modal information interaction module for modal information interaction is consistent with the principle of the step of inputting the query matrix and text feature matrix corresponding to the audio tensor and the forward normalized features of the audio into the cross-modal information interaction module for modal information interaction, and the present invention will not go into details.

[0179] S76, concatenating the initial audio-text interaction feature and the audio forward normalization feature through a concatenation layer, and outputting an audio-text concatenation feature;

[0180] S77, splicing the initial video frame text interaction feature and the video frame forward normalization feature through the splicing layer, and outputting the video frame text splicing feature;

[0181] S78. Use a fully connected layer to perform linear transformation on the audio text splicing features and the video frame text splicing features, and output target audio text interaction features and target video frame text interaction features.

[0182] In this embodiment, the multimodal sentiment tensor is input into the cross-attention feature interaction network for feature interaction, and the multimodal interaction feature is output.

[0183] Step 208: Use a multimodal feature scoring network to perform feature scoring on the multimodal interaction features and the multimodal sentiment tensor to generate a target sentiment analysis score.

[0184] The multimodal feature scoring network includes a feature fusion module and a sentiment analysis module.

[0185] Further, step 208 may include the following sub-steps:

[0186] S81, splicing the target audio text interaction feature, the target video frame text interaction feature and the text tensor through a feature fusion module to generate a multimodal splicing feature;

[0187] S82, performing two linear transformations on the multimodal splicing features, and outputting multimodal fusion features;

[0188] S83. Use the sentiment analysis module to perform linear transformation on the multimodal fusion features to generate a target sentiment analysis score.

[0189] In this embodiment, a multimodal feature scoring network is used to perform feature scoring on multimodal interaction features and multimodal sentiment tensors to generate a target sentiment analysis score.

[0190] It is worth mentioning that, as shown in Table 1, the TFN, LMF, MFN, MULT, and MISA data in the table are from Mao et al. (2022). The scores measured by the preset multimodal sentiment analysis model proposed in the present invention on the preprocessed TFN, LMF, MFN, MULT, and MISA data sets are shown in Table 1. After experimental verification, the preset multimodal sentiment analysis model proposed in the present invention has achieved certain results.

[0191] Table 1 uses the preset multimodal sentiment analysis model to output the corresponding scores according to each data set

[0192]

[0193] As a comparison of technical effects, it can be combined with existing technologies for reference. Human emotional expression is usually multimodal, which contains a mixture of natural language, facial expressions and voice behaviors. The general multimodal feature fusion method is to further process the features of the three modalities into one-dimensional tensors with the same feature dimension through LSTM or Transformer, and then directly splice the feature tensors of the three modalities. Due to the small amount of feature information in video and audio, the video and audio will bring in more noise after the same-dimensional features are spliced, which will affect the fusion results. And if the fusion process is not supervised, valuable emotional information will be lost in the feature fusion process. In the existing multimodal sentiment analysis methods, due to the small amount of feature information and high noise in the video and audio modalities, the fusion features have a lot of noise, and valuable emotional information is lost in the fusion process.

[0194] In response to the above problems, the present invention uses a cross-attention mechanism, which focuses on the interaction between multimodal sequences, and increases the effective features in the video and audio features by integrating the dominant text features into the video and audio respectively, thereby reducing the impact of noise. The present invention maintains the effective information in each unimodal feature and multimodal feature by hierarchically maximizing the mutual information (MI) between two modalities and the mutual information (MI) between the multimodal fusion results and the unimodal input. Mutual information (MI) is a concept in the information theory topic, and MI measures the dependency between pairs of multidimensional variables. Maximizing mutual information (MI) has been proven to effectively remove redundant information that is irrelevant to downstream tasks. The present invention can effectively solve the problem of noise influence on video and audio, as well as the problem of loss of important information during the fusion process, and can achieve excellent results in the field of sentiment analysis.

[0195] Furthermore, the present invention integrates text information into video and audio features through a cross-attention mechanism to improve the effectiveness of video and audio features. In order to prevent the loss of unimodal information in modal interaction, the present invention fuses the video and audio features after the interaction with the features before the interaction. At the same time, by maximizing mutual information (MI), unimodal feature extraction and multimodal fusion are supervised, so that each unimodality can be more dependent, and valuable emotional information can be fully retained in multimodal features by increasing the dependency between unimodal features and multimodal features. The present invention can effectively reduce the impact of video and audio modal noise, and retain valuable emotional information in modal fusion, thereby improving the accuracy of sentiment analysis. The present invention conducted experiments on the MOSI and MOSEI datasets to evaluate the effectiveness of the method proposed in the present invention. The dataset contains 23,453 data in three modalities: language, vision, and sound. The emotional intensity of the target is marked with [-3, 3]. It is a dataset suitable for multimodal sentiment analysis.

[0196] In summary, the present invention enhances the effectiveness of video and audio modalities through the cross-attention mechanism, reduces the loss of emotional information in the fusion process through mutual information (MI), improves the accuracy of sentiment analysis, and proposes a new multimodal information fusion scheme. The scheme mainly includes 6 modules: data feature extraction module, Transformer module, cross-modal sentiment information interaction module, fusion module, sentiment analysis module and mutual information maximization (MI) module. The data feature extraction module extracts features from the three modal information in the video data respectively; the Transformer module further extracts the three modal features and strengthens the time information; the cross-modal emotion information interaction module enables the visual and sound modalities to obtain text information through the cross-attention mechanism, thereby strengthening the information of the visual and sound modalities; the fusion module fuses the emotion information of the three modalities together so that each trimodal data can be fully expressed through a one-dimensional tensor; the emotion analysis module scores the emotion through the fused multimodal features to determine the emotion category; the mutual information maximization (MI) module estimates and improves the lower bound of the mutual information (MI), thereby enhancing the dependency between modalities and between single modality and multimodality, removing redundant feature information and retaining valuable emotion information.

[0197] In an embodiment of the present invention, the present invention provides a multimodal sentiment analysis method, which first obtains multimodal sentiment data to be detected, and inputs the sentiment data to be detected into a preset multimodal sentiment analysis model, wherein the preset multimodal sentiment analysis model includes a multimodal feature extraction network, a cross-attention feature interaction network, and a multimodal feature scoring network; then, the multimodal feature extraction network is used to perform feature extraction on the multimodal sentiment data to be detected, and a multimodal sentiment tensor is generated; the multimodal sentiment tensor is input into the cross-attention feature interaction network for feature interaction, and a multimodal interaction feature is output; finally, a multimodal feature scoring network is used to perform feature scoring on the multimodal interaction feature and the multimodal sentiment tensor, and a target sentiment analysis score is generated; based on the above scheme, the process of performing feature interaction on the multimodal sentiment tensor through the cross-attention feature interaction network and outputting the multimodal interaction feature can pay more attention to the interaction between multimodal sequences, increase effective features, thereby reducing the influence of noise, and further improving the model analysis effect.

[0198] See also Figure 5 , Figure 5 A structural block diagram of a multimodal sentiment analysis device provided in an embodiment of the present invention.

[0199] The present invention provides a multimodal sentiment analysis device, comprising:

[0200] An acquisition module 501 is used to acquire multimodal emotion data to be detected, and input the emotion data to be detected into a preset multimodal emotion analysis model, wherein the preset multimodal emotion analysis model includes a multimodal feature extraction network, a cross-attention feature interaction network, and a multimodal feature scoring network;

[0201] A generating module 502 is used to extract features from the multimodal emotion data to be detected through a multimodal feature extraction network to generate a multimodal emotion tensor;

[0202] An output module 503 is used to input the multimodal sentiment tensor into the cross-attention feature interaction network for feature interaction, and output the multimodal interaction feature;

[0203] Module 504 is used to perform feature scoring on the multimodal interaction features and the multimodal sentiment tensor using a multimodal feature scoring network to generate a target sentiment analysis score.

[0204] Furthermore, the multimodal feature extraction network includes a language representation network, an audio processing network and a face detection network; the multimodal emotion data to be detected includes a text sequence to be detected, an audio to be detected and a video frame image to be detected; the multimodal emotion tensor includes a text tensor, an audio tensor and a video frame tensor; the acquisition module 501 is specifically used for:

[0205] The text features of the text sequence to be detected are extracted through the language representation network to generate a text tensor;

[0206] An audio processing network is used to extract audio features from the audio to be detected and generate an audio tensor;

[0207] The video frame image to be detected is input into the face detection network for video frame feature extraction, and the video frame tensor is output.

[0208] Furthermore, the cross-attention feature interaction network includes a fully connected layer, an attention mechanism module, a cross-modal information interaction module and a splicing layer; the multimodal interaction feature includes a target audio text interaction feature and a target video frame text interaction feature; the generation module 502 includes:

[0209] The first submodule is used to perform linear transformation on the text tensor, the audio tensor and the video frame tensor respectively through a fully connected layer to generate a text feature matrix, an audio feature matrix and a video frame feature matrix;

[0210] The second submodule is used to use the attention mechanism module to perform feature superposition on the audio feature matrix and the audio tensor to generate audio forward normalized features;

[0211] The third submodule is used to use the attention mechanism module to perform feature superposition on the video frame feature matrix and the video frame tensor to generate forward normalized features of the video frame;

[0212] The fourth submodule is used to input the audio tensor, the query matrix corresponding to the audio forward normalized feature, and the text feature matrix into the cross-modal information interaction module for modal information interaction, and output the initial audio-text interaction feature;

[0213] The fifth submodule is used to input the video frame tensor, the query matrix corresponding to the forward normalized features of the video frame, and the text feature matrix into the cross-modal information interaction module for modal information interaction, and output the initial video frame text interaction features;

[0214] The sixth submodule is used to splice the initial audio-text interaction features and the audio forward normalization features through a splicing layer, and output the audio-text splicing features;

[0215] The seventh submodule is used to splice the initial video frame text interaction feature and the video frame forward normalization feature through a splicing layer, and output the video frame text splicing feature;

[0216] The eighth submodule is used to use a fully connected layer to perform linear transformation on the audio text splicing features and the video frame text splicing features respectively, and output target audio text interaction features and target video frame text interaction features.

[0217] Furthermore, the attention mechanism module includes a self-attention mechanism layer, a normalization layer and a forward network layer; the audio feature matrix includes an audio query matrix, an audio key value matrix and an audio value matrix; the second submodule is specifically used for:

[0218] Perform matrix multiplication on the audio query matrix, audio key matrix, and audio value matrix through the self-attention mechanism layer, and output the audio self-attention feature matrix;

[0219] Performing an addition operation on the audio self-attention feature matrix and the audio tensor, and outputting a first audio superposition feature;

[0220] Normalizing the first audio superposition feature through a normalization layer to generate an audio normalization feature;

[0221] The forward network layer is used to perform nonlinear transformation on the audio normalization features and output the audio nonlinear features;

[0222] Performing an addition operation on the audio nonlinear feature and the audio normalization feature, and outputting a second audio superposition feature;

[0223] The normalization layer is used to normalize the second audio superposition feature to generate an audio forward normalized feature.

[0224] Furthermore, the cross-modal information interaction module includes a cross-attention mechanism layer, a normalization layer and a forward network layer; the fourth submodule is specifically used for:

[0225] Perform matrix multiplication on the query matrix corresponding to the audio forward normalized features and the text key value matrix and text value matrix in the text feature matrix through the cross attention mechanism layer, and output the audio text cross attention matrix;

[0226] Performing an addition operation on the audio-text cross attention matrix and the audio tensor, and outputting the first audio-text superposition feature;

[0227] Normalizing the first audio text superposition feature through a normalization layer to generate an audio text normalized feature;

[0228] The forward network layer is used to perform nonlinear transformation on the normalized features of the audio text and output the nonlinear features of the audio text;

[0229] Performing an addition operation on the audio text nonlinear feature and the audio text normalized feature, and outputting a second audio text superposition feature;

[0230] The normalization layer is used to normalize the second audio-text superposition features to generate initial audio-text interaction features.

[0231] Furthermore, the multimodal feature scoring network includes a feature fusion module and a sentiment analysis module; the module 504 is used to:

[0232] The target audio text interaction features, the target video frame text interaction features and the text tensor are spliced ​​through the feature fusion module to generate multimodal splicing features;

[0233] Perform two linear transformations on the multimodal splicing features and output multimodal fusion features;

[0234] The sentiment analysis module is used to perform linear transformation on the multimodal fusion features to generate the target sentiment analysis score.

[0235] In an optional embodiment, the device further comprises:

[0236] An initial module is used to obtain a multimodal emotion data set to be trained, input the multimodal emotion data set to be trained into an initial multimodal emotion analysis model, and output a multimodal emotion tensor set to be trained, a multimodal interaction feature set to be trained, a multimodal fusion feature set to be trained, and a emotion analysis score set to be trained;

[0237] A loss value calculation module, used for calculating a target loss value according to the multimodal emotion tensor set to be trained, the multimodal interaction feature set to be trained, the multimodal fusion feature set to be trained and the emotion analysis score set to be trained by using a preset loss value function;

[0238] A judgment module is used to judge whether the target loss value converges;

[0239] The convergence module is used to use the initial multimodal sentiment analysis model as a trained preset multimodal sentiment analysis model if the target loss value converges.

[0240] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, modules and sub-modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0241] An embodiment of the present invention also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory; when the computer program is executed by the processor, the processor executes the steps of the multimodal sentiment analysis method as described in any of the above embodiments.

[0242] An embodiment of the present invention further provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the steps of the multimodal sentiment analysis method as described in any of the above embodiments are implemented.

[0243] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0244] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0245] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal sentiment analysis method, characterized in that: include: Acquire multimodal emotion data to be detected, and input the emotion data to be detected into a preset multimodal emotion analysis model, wherein the preset multimodal emotion analysis model includes a multimodal feature extraction network, a cross-attention feature interaction network, and a multimodal feature scoring network; Extracting features of the multimodal emotion data to be detected through the multimodal feature extraction network to generate a multimodal emotion tensor; Inputting the multimodal sentiment tensor into the cross-attention feature interaction network for feature interaction, and outputting multimodal interaction features; The multimodal feature scoring network is used to perform feature scoring on the multimodal interaction features and the multimodal sentiment tensor to generate a target sentiment analysis score.

2. The multimodal sentiment analysis method according to claim 1, characterized in that: The multimodal feature extraction network includes a language representation network, an audio processing network and a face detection network; the multimodal emotion data to be detected includes a text sequence to be detected, an audio to be detected and a video frame image to be detected; the multimodal emotion tensor includes a text tensor, an audio tensor and a video frame tensor; the step of extracting features from the multimodal emotion data to be detected through the multimodal feature extraction network to generate a multimodal emotion tensor includes: Extracting text features from the text sequence to be detected through the language representation network to generate a text tensor; Using the audio processing network to extract audio features from the audio to be detected, and generating an audio tensor; The video frame image to be detected is input into the face detection network to extract video frame features, and a video frame tensor is output.

3. The multimodal sentiment analysis method according to claim 2, characterized in that: The cross-attention feature interaction network includes a fully connected layer, an attention mechanism module, a cross-modal information interaction module and a splicing layer; the multimodal interaction feature includes a target audio text interaction feature and a target video frame text interaction feature; the step of inputting the multimodal sentiment tensor into the cross-attention feature interaction network for feature interaction and outputting the multimodal interaction feature includes: Performing linear transformation on the text tensor, the audio tensor, and the video frame tensor respectively through a fully connected layer to generate a text feature matrix, an audio feature matrix, and a video frame feature matrix; Using an attention mechanism module to perform feature superposition on the audio feature matrix and the audio tensor to generate audio forward normalized features; Using an attention mechanism module to perform feature superposition on the video frame feature matrix and the video frame tensor to generate forward normalized features of the video frame; Inputting the audio tensor, the query matrix corresponding to the audio forward normalized feature, and the text feature matrix into a cross-modal information interaction module for modal information interaction, and outputting initial audio-text interaction features; Inputting the video frame tensor, the query matrix corresponding to the forward normalized features of the video frame, and the text feature matrix into a cross-modal information interaction module for modal information interaction, and outputting initial video frame text interaction features; Splicing the initial audio-text interaction feature and the audio forward normalization feature through a splicing layer, and outputting an audio-text splicing feature; Splicing the initial video frame text interaction feature and the video frame forward normalization feature through a splicing layer, and outputting a video frame text splicing feature; A fully connected layer is used to perform linear transformation on the audio-text splicing features and the video-frame-text splicing features, respectively, and a target audio-text interaction feature and a target video-frame-text interaction feature are output.

4. The multimodal sentiment analysis method according to claim 3, characterized in that: The attention mechanism module includes a self-attention mechanism layer, a normalization layer and a forward network layer; the audio feature matrix includes an audio query matrix, an audio key value matrix and an audio value matrix; The step of using the attention mechanism module to perform feature superposition on the audio feature matrix and the audio tensor to generate audio forward normalized features includes: Performing matrix multiplication operation on the audio query matrix, the audio key value matrix and the audio value matrix through a self-attention mechanism layer, and outputting an audio self-attention feature matrix; Performing an addition operation on the audio self-attention feature matrix and the audio tensor, and outputting a first audio superposition feature; Normalizing the first audio superposition feature through a normalization layer to generate an audio normalization feature; Using a forward network layer to perform nonlinear transformation on the audio normalization features and output audio nonlinear features; Performing an addition operation on the audio nonlinear feature and the audio normalization feature, and outputting a second audio superposition feature; The second audio superposition feature is normalized by using a normalization layer to generate an audio forward normalized feature.

5. The multimodal sentiment analysis method according to claim 3, characterized in that: The cross-modal information interaction module includes a cross-attention mechanism layer, a normalization layer and a forward network layer; the step of inputting the audio tensor, the query matrix corresponding to the audio forward normalized feature and the text feature matrix into the cross-modal information interaction module for modal information interaction and outputting the initial audio-text interaction feature includes: Performing matrix multiplication operation on the query matrix corresponding to the audio forward normalized feature and the text key value matrix and the text value matrix in the text feature matrix through a cross attention mechanism layer, and outputting an audio text cross attention matrix; Performing an addition operation on the audio-text cross-attention matrix and the audio tensor, and outputting a first audio-text superposition feature; Normalizing the first audio text superposition feature through a normalization layer to generate an audio text normalized feature; Using a forward network layer to perform nonlinear transformation on the normalized features of the audio text, and outputting the nonlinear features of the audio text; Performing an addition operation on the audio text nonlinear feature and the audio text normalized feature, and outputting a second audio text superposition feature; The second audio-text superposition feature is normalized using a normalization layer to generate an initial audio-text interaction feature.

6. The multimodal sentiment analysis method according to claim 3, characterized in that: The multimodal feature scoring network includes a feature fusion module and a sentiment analysis module; the step of using the multimodal feature scoring network to perform feature scoring on the multimodal interaction features and the multimodal sentiment tensor to generate a target sentiment analysis score includes: The target audio-text interaction feature, the target video frame text interaction feature and the text tensor are spliced ​​by the feature fusion module to generate a multimodal splicing feature; Performing two linear transformations on the multimodal splicing features and outputting multimodal fusion features; The sentiment analysis module is used to perform linear transformation on the multimodal fusion features to generate a target sentiment analysis score.

7. The multimodal sentiment analysis method according to claim 1, characterized in that: Before the step of obtaining the multimodal emotion data to be detected and inputting the emotion data to be detected into a preset multimodal emotion analysis model, the method includes: Acquire a multimodal emotion data set to be trained, input the multimodal emotion data set to be trained into an initial multimodal emotion analysis model, and output a multimodal emotion tensor set to be trained, a multimodal interaction feature set to be trained, a multimodal fusion feature set to be trained, and a emotion analysis score set to be trained; A preset loss value function is used to calculate a target loss value according to the multimodal emotion tensor set to be trained, the multimodal interaction feature set to be trained, the multimodal fusion feature set to be trained, and the emotion analysis score set to be trained; Determining whether the target loss value converges; If the target loss value converges, the initial multimodal sentiment analysis model is used as the trained preset multimodal sentiment analysis model.

8. A multimodal sentiment analysis device, characterized in that: include: An acquisition module, used for acquiring multimodal emotion data to be detected, and inputting the emotion data to be detected into a preset multimodal emotion analysis model, wherein the preset multimodal emotion analysis model includes a multimodal feature extraction network, a cross-attention feature interaction network, and a multimodal feature scoring network; A generating module, used for performing feature extraction on the multimodal emotion data to be detected through the multimodal feature extraction network to generate a multimodal emotion tensor; An output module, used for inputting the multimodal sentiment tensor into the cross-attention feature interaction network for feature interaction, and outputting multimodal interaction features; A module is used to use the multimodal feature scoring network to perform feature scoring on the multimodal interaction features and the multimodal sentiment tensor to generate a target sentiment analysis score.

9. A computer device, characterized in that: It includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the multimodal sentiment analysis method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the multimodal sentiment analysis method as described in any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • Multi-modal sentiment analysis method fusing multiple features and attention mechanism

    CN116028846A

  • Multi-modal sentiment analysis method and device and electronic equipment

    CN116662482A

  • Multi-modal sentiment analysis system and method based on loop generation

    CN116861181A

  • Multi-modal emotion recognition method and system for guiding attention fusion based on text modal, and storage medium

    CN117786596A

Cited By

  • Multi-modal sentiment analysis and interactive adjustment method and system based on large model

    CN119475099A

  • Multi-modal sentiment analysis method and device based on mutual information and cross-modal text enhancement

    CN120372441A