Multimodal sentiment analysis method, device, electronic device and storage medium

By extracting and enhancing the feature of speech, text and images in the video, and combining the cross-modal fusion model for feature fusion, the problem of poor feature fusion between modes in multimodal sentiment analysis is solved, and the sentiment analysis is improved Accuracy, suitable for areas such as mental health status detection.

CN117828523BActive Publication Date: 2024-11-19HUBEI UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311779510.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-20
Publication Date
2024-11-19
Estimated Expiration
2043-12-20

AI Technical Summary

Technical Problem

The existing multimodal sentiment analysis methods ignore the independence and correlation between modals when fusing text, speech and image features, resulting in poor accuracy of sentiment analysis, especially due to poor feature fusion effect due to different information density.

Method used

By extracting the feature of speech, text and character images in the video respectively, feature enhancement and cross-modal feature fusion are performed, feature expression is enhanced using LSTM network and Hadamar product methods, and feature fusion is performed through cross-modal fusion model. Finally, a multimodal sentiment analysis model was input for sentiment analysis.

Benefits of technology

Improve the accuracy of multimodal emotion analysis, reduce the loss of important information, take into account the consistency and differences of different modal characteristics, enhance the expression of emotional states, and is suitable for fields such as mental health status detection and psychological disease warning. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117828523B_ABST
    Figure CN117828523B_ABST
Patent Text Reader

Abstract

The present invention discloses a multimodal sentiment analysis method, device, electronic device and storage medium, and relates to the field of sentiment analysis technology. The method comprises: extracting features of speech, text and character images in a video to obtain unimodal speech features, unimodal text features and unimodal character features; based on any two of the three unimodal features, enhancing the other feature to obtain unimodal speech enhancement features, unimodal text enhancement features and unimodal character enhancement features; fusing the unimodal text enhancement features with the unimodal speech enhancement features and the unimodal character enhancement features to obtain a first cross-modal feature and a second cross-modal feature; inputting the first cross-modal feature, the second cross-modal feature and the unimodal text enhancement features into a multimodal sentiment analysis model to perform sentiment analysis and obtain sentiment analysis results. The method, device, electronic device and storage medium disclosed in the present invention can improve the accuracy of sentiment analysis.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of sentiment analysis, and specifically relates to a multimodal sentiment analysis method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of social media, people's ways of expressing themselves on the Internet have become more and more diverse. On platforms such as Twitter, Weibo, and Taobao, many people will add a few pictures to their text comments. Short video software such as Douyin and Kuaishou are also loved by many people. Users use short videos to record their lives and express their feelings, resulting in a large amount of multi-modal data including text, pictures, animated images, audio and video. The process of using two or more modal data to predict emotional information is called multimodal sentiment analysis (MSA). Compared with traditional single-modal sentiment representation, using multimodal data for sentiment analysis is more conducive to understanding users' attitudes and opinions on certain events. Single-modal sentiment analysis is difficult to apply in complex multimedia environments, so it is necessary to integrate multiple modalities for fusion analysis.

[0003] With the continuous development of deep neural networks, multimodal sentiment analysis has achieved good research results. Most existing technologies use deep learning methods to perform multimodal sentiment analysis, such as transfer learning, contrastive learning, and multi-task learning. However, on the one hand, most of these methods focus on the fusion mechanism of multimodal data, while ignoring the independence and correlation of different modal data, which limits the performance of the model; on the other hand, due to the differences in information density between the three modalities of text, speech, and image, text information has high-level semantic features, while audio and visual information are low-level semantic features, which leads to poor results when we perform feature fusion, and thus poor accuracy of sentiment analysis results.

[0004] Therefore, how to provide an effective solution to improve the accuracy of sentiment analysis has become a difficult problem to be solved urgently in the prior art. Summary of the invention

[0005] The purpose of the present invention is to provide a multimodal sentiment analysis method, device, electronic device and storage medium to solve the above-mentioned problems existing in the prior art.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a multimodal sentiment analysis method, comprising:

[0008] Extract features of speech, text and character images in the video respectively to obtain unimodal speech features, unimodal text features and unimodal character features;

[0009] Based on any two of the unimodal speech features, the unimodal text features, and the unimodal character features, feature enhancement is performed on another feature other than the any two features to obtain unimodal speech enhancement features, unimodal text enhancement features, and unimodal character enhancement features;

[0010] The unimodal text enhancement feature is respectively fused with the unimodal speech enhancement feature and the unimodal person enhancement feature to obtain a first cross-modal feature and a second cross-modal feature;

[0011] The first cross-modal feature, the second cross-modal feature and the unimodal text enhancement feature are input into a multimodal sentiment analysis model to perform sentiment analysis and obtain a multimodal sentiment analysis result.

[0012] Based on the above disclosed content, the present invention obtains unimodal speech features, unimodal text features and unimodal character features by respectively extracting features of speech, text and character images in a video; based on any two of the unimodal speech features, unimodal text features and unimodal character features, feature enhancement is performed on another feature other than any two features to obtain unimodal speech enhancement features, unimodal text enhancement features and unimodal character enhancement features; the unimodal text enhancement features are respectively feature-fused with the unimodal speech enhancement features and the unimodal character enhancement features to obtain a first cross-modal feature and a second cross-modal feature; the first cross-modal feature, the second cross-modal feature and the unimodal text enhancement features are input into a multimodal sentiment analysis model to perform sentiment analysis to obtain a multimodal sentiment analysis result. In this way, in the process of sentiment analysis, multimodal features are used as the basis to reduce the loss of important information in the process of multimodal sentiment analysis. At the same time, by performing feature enhancement and feature fusion on multimodal features, while taking into account the consistency and difference of different modal features, the characteristic expression of emotional state can be enhanced, thereby obtaining more accurate sentiment analysis results and improving the accuracy of sentiment analysis. It can be widely used in various fields such as detection and evaluation of mental health status and early warning of mental illness.

[0013] Through the above-mentioned design, the present invention can reduce the loss of important information in the process of multimodal sentiment analysis based on multimodal features, and at the same time, by performing feature enhancement and feature fusion on multimodal features, while taking into account the consistency and difference of different modal features, it can enhance the characteristic expression of emotional state, thereby obtaining more accurate sentiment analysis results, improving the accuracy of sentiment analysis, and can be widely used in various fields such as detection and evaluation of mental health status, early warning of mental illness, etc.

[0014] In a possible design, based on any two of the unimodal speech features, the unimodal text features, and the unimodal character features, feature enhancement is performed on another feature other than the any two features to obtain unimodal speech enhancement features, unimodal text enhancement features, and unimodal character enhancement features, including:

[0015] Inputting the unimodal speech feature, the unimodal text feature and the unimodal character feature into the LSTM network respectively, and obtaining the hidden layer output corresponding to the unimodal speech feature, the hidden layer output corresponding to the unimodal text feature and the hidden layer output corresponding to the unimodal character feature;

[0016] Performing a Hadamard product on the unimodal character feature using the hidden layer output corresponding to the unimodal speech feature and the hidden layer output corresponding to the unimodal text feature to obtain a unimodal character enhancement feature corresponding to the unimodal character feature;

[0017] Using the hidden layer output corresponding to the unimodal text feature and the hidden layer output corresponding to the unimodal character feature, performing a Hadamard product on the unimodal speech feature to obtain a unimodal speech enhancement feature corresponding to the unimodal speech feature;

[0018] The Hadamard product is performed on the unimodal text features using the hidden layer output corresponding to the unimodal speech features and the hidden layer output corresponding to the unimodal character features to obtain unimodal text enhancement features corresponding to the unimodal text features.

[0019] In a possible design, the unimodal text enhancement feature is respectively fused with the unimodal speech enhancement feature and the unimodal person enhancement feature to obtain a first cross-modal feature and a second cross-modal feature, including:

[0020] The unimodal text enhancement feature is respectively fused with the unimodal speech enhancement feature and the unimodal person enhancement feature through a cross-modal fusion model to obtain a first cross-modal feature and a second cross-modal feature;

[0021] The cross-modal fusion model includes a position encoding layer, a symmetric attention mechanism layer, a first residual connection and normalization layer, a feedforward network processing layer, and a second residual connection and normalization layer;

[0022] The position encoding layer is used to construct the position relationship between the unimodal text enhancement feature, the unimodal speech enhancement feature and the unimodal person enhancement feature in the sequence;

[0023] The symmetric attention mechanism layer is used to use the unimodal text enhancement feature as a query vector, the unimodal speech enhancement feature as a key vector and a value vector, perform a dot product operation using the query vector and the key vector, and after obtaining the attention weight through the softmax function, multiply the attention weight by the value vector to obtain the first symmetric attention mechanism output information;

[0024] The symmetric attention mechanism layer is also used to use the unimodal text enhancement feature as a query vector, the unimodal person enhancement feature as a key vector and a value vector, perform a dot product operation using the query vector and the key vector, and after obtaining the attention weight through the softmax function, multiply the attention weight by the value vector to obtain the second symmetric attention mechanism output information;

[0025] The first residual connection and normalization layer is used to perform residual connection and normalization processing on the unimodal text enhancement feature and the output information of the first symmetric attention mechanism to obtain a third cross-modal feature;

[0026] The first residual connection and normalization layer is further used to perform residual connection and normalization processing on the unimodal text enhancement feature and the output information of the second symmetric attention mechanism to obtain a fourth cross-modal feature;

[0027] The feedforward network processing layer is used to perform enhancement processing on the third cross-modal feature and the fourth cross-modal feature through a linear rectification function and a linear activation function respectively, so as to obtain a fifth cross-modal feature corresponding to the third cross-modal feature and a sixth cross-modal feature corresponding to the fourth cross-modal feature;

[0028] The second residual connection and normalization layer is used to perform residual connection and normalization processing on the third cross-modal feature and the fifth cross-modal feature to obtain the first cross-modal feature;

[0029] The second residual connection and normalization layer is also used to perform residual connection and normalization processing on the fourth cross-modal feature and the sixth cross-modal feature to obtain the second cross-modal feature.

[0030] In a possible design, the first cross-modal feature, the second cross-modal feature, and the unimodal text enhancement feature are input into a multimodal sentiment analysis model for sentiment analysis to obtain a multimodal sentiment analysis result, including:

[0031] Adjusting the weights of the first cross-modal feature, the second cross-modal feature, and the unimodal text enhancement feature through an attention mechanism;

[0032] The first cross-modal feature after adjusting the weight, the second cross-modal feature after adjusting the weight, and the unimodal text enhancement feature after adjusting the weight are input into a multimodal sentiment analysis model for sentiment analysis to obtain a multimodal sentiment analysis result.

[0033] In one possible design, before inputting the first cross-modal feature, the second cross-modal feature, and the unimodal text enhancement feature into a multimodal sentiment analysis model for sentiment analysis, the method further includes:

[0034] The first cross-modal feature, the second cross-modal feature, and the unimodal text enhancement feature are concatenated.

[0035] In a possible design, the feature extraction of speech, text and character image in the video is performed respectively to obtain unimodal speech features, unimodal text features and unimodal character features, including:

[0036] Extract facial features of the person images in the video to obtain the facial features of the person;

[0037] Extract action features from character images in the video to obtain character action features;

[0038] splicing the character facial features and the character action features to obtain the unimodal character features;

[0039] Extracting features of text in the video to obtain the unimodal text features;

[0040] Feature extraction is performed on the speech in the video to obtain the unimodal speech feature.

[0041] In a possible design, extracting facial features from a person image in a video includes:

[0042] Extract facial features from the characters in the video using the OpenFace model;

[0043] The step of extracting action features from a person image in a video includes:

[0044] The ViTPose model is used to extract action features from character images in the video;

[0045] The feature extraction of the text in the video includes:

[0046] Extract features from text in the video using the BERT model;

[0047] The feature extraction of speech in the video includes:

[0048] The LibROSA speech toolkit is used to extract the speech features in the video.

[0049] In a second aspect, the present invention provides a multimodal sentiment analysis device, comprising:

[0050] A feature extraction unit is used to extract features of speech, text and character images in the video respectively, to obtain unimodal speech features, unimodal text features and unimodal character features;

[0051] A feature enhancement unit, configured to enhance the feature of any two of the unimodal speech feature, the unimodal text feature and the unimodal character feature, to obtain a unimodal speech enhancement feature, a unimodal text enhancement feature and a unimodal character enhancement feature;

[0052] A feature fusion unit, used to fuse the unimodal text enhancement feature with the unimodal speech enhancement feature and the unimodal person enhancement feature to obtain a first cross-modal feature and a second cross-modal feature;

[0053] The sentiment analysis unit is used to input the first cross-modal feature, the second cross-modal feature and the unimodal text enhancement feature into a multimodal sentiment analysis model to perform sentiment analysis and obtain a multimodal sentiment analysis result.

[0054] In a third aspect, the present invention provides an electronic device comprising a memory, a processor and a transceiver which are communicatively connected in sequence, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program to execute the multimodal sentiment analysis method as described in the first aspect or any possible design of the first aspect.

[0055] In a fourth aspect, the present invention provides a computer-readable storage medium having instructions stored thereon. When the instructions are executed on a computer, the multimodal sentiment analysis method described in the first aspect or any possible design of the first aspect is executed.

[0056] In a fifth aspect, the present invention provides a computer program product comprising instructions, which, when executed on a computer, causes the computer to execute the multimodal sentiment analysis method as described in the first aspect or any possible design of the first aspect.

[0057] Beneficial effects:

[0058] The multimodal sentiment analysis method, device, electronic device and storage medium provided by the present invention can reduce the loss of important information in the multimodal sentiment analysis process based on multimodal features, and at the same time, by performing feature enhancement and feature fusion on multimodal features, while taking into account the consistency and difference of different modal features, it can enhance the characteristic expression of emotional state, thereby obtaining more accurate sentiment analysis results, improving the accuracy of sentiment analysis, and can be widely used in various fields such as detecting and evaluating mental health status and early warning of mental illness. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A flowchart of a multimodal sentiment analysis method provided in an embodiment of the present application;

[0060] Figure 2 A flowchart of the specific analysis process of multimodal sentiment analysis provided in the embodiment of the present application;

[0061] Figure 3 A schematic block diagram of a multimodal sentiment analysis device provided in an embodiment of the present application;

[0062] Figure 4 A schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in combination with the drawings and the description of the embodiments or the prior art. Obviously, the following description of the structure of the drawings is only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention.

[0064] It should be understood that although the terms first, second, etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are only used to distinguish one unit from another unit. For example, a first unit can be referred to as a second unit, and similarly, a second unit can be referred to as a first unit without departing from the scope of the exemplary embodiments of the present invention.

[0065] It should be understood that the term "and / or" that may appear in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can represent three situations: A exists alone, B exists alone, and A and B exist at the same time. The term " / and" that may appear in this article describes another type of association object relationship, indicating that two relationships may exist. For example, A / and B can represent two situations: A exists alone, and A and B exist alone. In addition, the character " / " that may appear in this article generally indicates that the previous and next associated objects are in an "or" relationship.

[0066] In order to improve the accuracy of sentiment analysis, the embodiments of the present application provide a multimodal sentiment analysis method, device, electronic device and storage medium. The multimodal sentiment analysis method, device, electronic device and storage medium can more accurately implement sentiment analysis and improve the accuracy of sentiment analysis.

[0067] The multimodal sentiment analysis method provided in the embodiments of the present application can be applied to a user terminal or a server, and the user terminal may be, but is not limited to, a personal computer, a smart phone, a tablet computer, a laptop computer, a personal digital assistant, etc.

[0068] The multimodal sentiment analysis method provided in the embodiments of the present application will be described in detail below.

[0069] like Figure 1 and Figure 2 As shown, it is a flowchart of the multimodal sentiment analysis method provided in the first aspect of the embodiment of the present application. The multimodal sentiment analysis method may include but is not limited to the following steps S101-S104.

[0070] Step S101. Feature extraction is performed on the speech, text and character image in the video respectively to obtain unimodal speech features, unimodal text features and unimodal character features.

[0071] In the embodiment of the present application, when performing feature characterization, facial features of the person can be extracted from the person image in the video to obtain facial features of the person, and action features of the person can be extracted from the person image in the video to obtain action features of the person. The facial features of the person and the facial features of the person are then passed into a linear transformation layer to be converted to the same latitude and spliced ​​to obtain a single-modal person feature. Among them, the facial features of the person can include, but are not limited to, information such as head direction, gaze direction, facial landmarks, and facial action units.

[0072] When extracting facial features, the OpenFace model or the Facet model can be used, but is not limited to. When extracting action features, the ViTPose model or the Recurrent Neural Network (RNN) model can be used, but is not limited to. When extracting features from text, the BERT (Bidirectional Encoder Representations from Transformers) model or the ELMO (Embeddings from Language Models) model can be used, but is not limited to. When extracting features from speech, the LibROSA speech toolkit can be used for feature extraction.

[0073] Step S102. Based on any two of the unimodal speech features, the unimodal text features and the unimodal character features, feature enhancement is performed on another feature other than the two features to obtain unimodal speech enhancement features, unimodal text enhancement features and unimodal character enhancement features.

[0074] Specifically, the unimodal speech features, unimodal text features and unimodal character features can be respectively input into the LSTM network to obtain the hidden layer output corresponding to the unimodal speech features, the hidden layer output corresponding to the unimodal text features and the hidden layer output corresponding to the unimodal character features. Then, the hidden layer output corresponding to the unimodal speech features and the hidden layer output corresponding to the unimodal text features are used to perform a Hadamard product on the unimodal character features to obtain the unimodal character enhancement features corresponding to the unimodal character features. The hidden layer output corresponding to the unimodal text features and the hidden layer output corresponding to the unimodal character features are used to perform a Hadamard product on the unimodal speech features to obtain the unimodal speech enhancement features corresponding to the unimodal speech features. The hidden layer output corresponding to the unimodal speech features and the hidden layer output corresponding to the unimodal character features are used to perform a Hadamard product on the unimodal text features to obtain the unimodal text enhancement features corresponding to the unimodal text features.

[0075] The unimodal speech feature is represented as Fa, the unimodal text feature is represented as Ft, the unimodal character feature is represented as Fv, the hidden layer output corresponding to the unimodal speech feature is represented as ha, the hidden layer output corresponding to the unimodal text feature is represented as ht, and the hidden layer output corresponding to the unimodal character feature is represented as hv. The calculation formula for feature enhancement can be expressed as: Among them, Xa represents the unimodal speech enhancement feature, Xt represents the unimodal text enhancement feature, and Xv represents the unimodal person enhancement feature.

[0076] Step S103: Fusing the unimodal text enhancement feature with the unimodal speech enhancement feature and the unimodal person enhancement feature to obtain a first cross-modal feature and a second cross-modal feature.

[0077] In an embodiment of the present application, the unimodal text enhancement feature can be fused with the unimodal speech enhancement feature and the unimodal person enhancement feature respectively through a cross-modal fusion model to obtain a first cross-modal feature and a second cross-modal feature. The cross-modal fusion model includes a position encoding layer, a symmetric attention mechanism layer, a first residual connection and normalization layer, a feedforward network processing layer, and a second residual connection and normalization layer.

[0078] Among them, the position encoding layer is used to construct the position relationship between the unimodal text enhancement features, the unimodal speech enhancement features and the unimodal person enhancement features in the sequence.

[0079] The symmetric attention mechanism layer is used to use the unimodal text enhancement feature as the query vector and the unimodal speech enhancement feature as the key vector and value vector. The query vector and the key vector are used for dot product operation. After obtaining the attention weight through the softmax function, the attention weight is multiplied by the value vector to obtain the first symmetric attention mechanism output information.

[0080] The output information of the first symmetric attention mechanism can be expressed as Where Qt represents the query vector corresponding to the unimodal text enhancement feature, Ka represents the key vector corresponding to the unimodal speech enhancement feature, Va represents the value vector corresponding to the unimodal speech enhancement feature, T represents transposition, and dk is a constant.

[0081] The symmetric attention mechanism layer is also used to use the unimodal text enhancement features as the query vector and the unimodal person enhancement features as the key vector and value vector, perform a dot product operation using the query vector and the key vector, and after obtaining the attention weight through the softmax function, multiply the attention weight by the value vector to obtain the second symmetric attention mechanism output information.

[0082] The output information of the second symmetric attention mechanism can be expressed as Where Kv represents the key vector corresponding to the unimodal character enhancement feature, and Vv represents the value vector corresponding to the unimodal character enhancement feature.

[0083] The first residual connection and normalization layer is used to perform residual connection and normalization processing on the unimodal text enhancement feature and the output information of the first symmetric attention mechanism to obtain the third cross-modal feature. The third cross-modal feature can be expressed as X′at=LayerNorm(Xt+Attention(Qt,Ka,Va)).

[0084] The first residual connection and normalization layer is also used to perform residual connection and normalization processing on the unimodal text enhancement feature and the output information of the second symmetric attention mechanism to obtain the fourth cross-modal feature. The fourth cross-modal feature can be expressed as X′vt=LayerNorm(Xt+Attention(Qt,Kv,Vv)).

[0085] The feedforward network processing layer is used to enhance the third cross-modal feature and the fourth cross-modal feature through a linear rectification function and a linear activation function, respectively, to obtain a fifth cross-modal feature corresponding to the third cross-modal feature and a sixth cross-modal feature corresponding to the fourth cross-modal feature.

[0086] The fifth cross-modal feature can be expressed as FFN(X′at)=max(0, X′atW1+b1)W2+b2, and the sixth cross-modal feature can be expressed as FFN(X′vt)=max(0, X′vtW1+b1)W2+b2, where W1 and W2 are the weight matrices of the feedforward layer, and b1 and b2 are the feedforward layer biases. Since the symmetric attention mechanism is not well adapted to complex processes, the ability of modal expression of emotional state is enhanced through the feedforward network layer.

[0087] The second residual connection and normalization layer is used to perform residual connection and normalization processing on the third cross-modal feature and the fifth cross-modal feature to obtain the first cross-modal feature. The second residual connection and normalization layer is also used to perform residual connection and normalization processing on the fourth cross-modal feature and the sixth cross-modal feature to obtain the second cross-modal feature. The first cross-modal feature can be expressed as Xat=LayerNorm(X′at+FFN(X′at)), and the second cross-modal feature can be expressed as Xvt=LayerNorm(X′vt+FFN(X′vt)).

[0088] Through cross-modal feature fusion processing, the interactive fusion of different single-modal features can be achieved, enhancing the correlation and complementarity between modalities.

[0089] Step S104: Input the first cross-modal feature, the second cross-modal feature and the unimodal text enhancement feature into a multimodal sentiment analysis model to perform sentiment analysis and obtain a multimodal sentiment analysis result.

[0090] In order to ensure the dominance of text features and obtain the fusion of single-modal internal information and inter-modal interactive information, the embodiment of the present application finally uses cross-modal features and single-modal text enhancement features as the final multimodal feature representation, and through linear transformation operations, the first cross-modal feature, the second cross-modal feature and the single-modal text enhancement feature have the same feature dimension, and then the first cross-modal feature, the second cross-modal feature and the single-modal text enhancement feature are spliced. The spliced ​​features can be expressed as Z = Concat (Xat, Xt, Xvt).

[0091] When performing sentiment analysis, the weights of the first cross-modal feature, the second cross-modal feature, and the unimodal text enhancement feature can be adjusted through the attention mechanism to identify key features. Then, the weighted first cross-modal feature, the weighted second cross-modal feature, and the weighted unimodal text enhancement feature are input into a pre-trained multimodal sentiment analysis model for sentiment analysis to perform sentiment analysis and obtain multimodal sentiment analysis results. By adjusting the weights of the first cross-modal feature, the second cross-modal feature, and the unimodal text enhancement feature through the attention mechanism, the weight of useful information is increased and the weight of other information is reduced, thereby further improving the accuracy of the model.

[0092] The multimodal sentiment analysis method provided by the present invention obtains unimodal speech features, unimodal text features and unimodal character features by respectively extracting features from speech, text and character images in a video; based on any two of the unimodal speech features, unimodal text features and unimodal character features, a feature enhancement is performed on another feature other than the any two features to obtain unimodal speech enhancement features, unimodal text enhancement features and unimodal character enhancement features; the unimodal text enhancement features are respectively feature-fused with the unimodal speech enhancement features and the unimodal character enhancement features to obtain a first cross-modal feature and a second cross-modal feature; the first cross-modal feature, the second cross-modal feature and the unimodal text enhancement features are input into a multimodal sentiment analysis model to perform sentiment analysis to obtain a multimodal sentiment analysis result. In this way, in the process of sentiment analysis, based on multimodal features, the loss of important information in the process of multimodal sentiment analysis is reduced. At the same time, by performing feature enhancement and feature fusion on multimodal features, while taking into account the consistency and difference of different modal features, the feature expression of emotional state can be enhanced, thereby obtaining a more accurate sentiment analysis result and improving the accuracy of sentiment analysis. In addition, when performing sentiment analysis, the weights of the first cross-modal feature, the second cross-modal feature and the unimodal text enhancement feature can be adjusted through the attention mechanism, which increases the weight of useful information and reduces the weight of other information, which can further improve the accuracy of the model. The multimodal sentiment analysis method provided by the present invention can be widely used in various fields such as detecting and evaluating mental health status and early warning of mental illness.

[0093] See also Figure 3 A second aspect of an embodiment of the present application provides a multimodal sentiment analysis device, the multimodal sentiment analysis device comprising:

[0094] A feature extraction unit is used to extract features of speech, text and character images in the video respectively, to obtain unimodal speech features, unimodal text features and unimodal character features;

[0095] A feature enhancement unit, configured to enhance the feature of any two of the unimodal speech feature, the unimodal text feature and the unimodal character feature, to obtain a unimodal speech enhancement feature, a unimodal text enhancement feature and a unimodal character enhancement feature;

[0096] A feature fusion unit, used to fuse the unimodal text enhancement feature with the unimodal speech enhancement feature and the unimodal person enhancement feature to obtain a first cross-modal feature and a second cross-modal feature;

[0097] The sentiment analysis unit is used to input the first cross-modal feature, the second cross-modal feature and the unimodal text enhancement feature into a multimodal sentiment analysis model to perform sentiment analysis and obtain a multimodal sentiment analysis result.

[0098] The working process, working details and technical effects of the multimodal sentiment analysis device provided in the second aspect of this embodiment can be found in the first aspect of the embodiment and will not be repeated here.

[0099] like Figure 4 As shown, the third aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor and a transceiver that are communicatively connected in sequence, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program to execute the multimodal sentiment analysis method as described in the first aspect of the embodiment.

[0100] For specific examples, the memory may include but is not limited to random access memory (RAM), read-only memory (ROM), flash memory, first-in-first-out memory (FIFO) and / or first-in-last-out memory (FILO), etc.; the processor may be but is not limited to a microprocessor of the STM32F105 series, an ARM (Advanced RISC-Machines), an X86 or other architecture processor, or a processor with an integrated NPU (neural-network processing units); the transceiver may be but is not limited to a WiFi (Wireless Fidelity) wireless transceiver, a Bluetooth wireless transceiver, a General Packet Radio Service (GPRS) wireless transceiver, a ZigBee protocol (a low-power local area network protocol based on the IEEE802.15.4 standard, ZigBee) wireless transceiver, a 3G transceiver, a 4G transceiver and / or a 5G transceiver, etc.

[0101] The fourth aspect of this embodiment provides a computer-readable storage medium storing instructions including the multimodal sentiment analysis method described in the first aspect of the embodiment, that is, the computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the multimodal sentiment analysis method described in the first aspect is executed. The computer-readable storage medium refers to a carrier for storing data, which may include but is not limited to a floppy disk, an optical disk, a hard disk, a flash memory, a USB flash drive, and / or a memory stick, etc., and the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0102] The fifth aspect of this embodiment provides a computer program product comprising instructions, which, when executed on a computer, causes the computer to execute the multimodal sentiment analysis method as described in the first aspect of the embodiment, wherein the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0103] It should be understood that certain details are provided in the following description to facilitate a complete understanding of the example embodiments. However, it should be understood by those of ordinary skill in the art that the example embodiments can be implemented without these certain details. For example, the system can be shown in a block diagram to avoid obscuring the example with unnecessary details. In other examples, well-known processes, structures, and techniques may not be shown in unnecessary detail to avoid obscuring the example embodiments.

[0104] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multimodal sentiment analysis method, characterized in that: include: Extract features of speech, text and character images in the video respectively to obtain unimodal speech features, unimodal text features and unimodal character features; Based on any two of the unimodal speech features, the unimodal text features, and the unimodal character features, feature enhancement is performed on another feature other than the any two features to obtain unimodal speech enhancement features, unimodal text enhancement features, and unimodal character enhancement features; The unimodal text enhancement feature is respectively fused with the unimodal speech enhancement feature and the unimodal person enhancement feature to obtain a first cross-modal feature and a second cross-modal feature; Inputting the first cross-modal feature, the second cross-modal feature and the unimodal text enhancement feature into a multimodal sentiment analysis model for sentiment analysis to obtain a multimodal sentiment analysis result; The method of enhancing the feature of any two of the unimodal speech feature, the unimodal text feature and the unimodal character feature to obtain unimodal speech enhancement feature, unimodal text enhancement feature and unimodal character enhancement feature comprises: Inputting the unimodal speech feature, the unimodal text feature and the unimodal character feature into the LSTM network respectively, and obtaining the hidden layer output corresponding to the unimodal speech feature, the hidden layer output corresponding to the unimodal text feature and the hidden layer output corresponding to the unimodal character feature; Performing a Hadamard product on the unimodal character feature using the hidden layer output corresponding to the unimodal speech feature and the hidden layer output corresponding to the unimodal text feature to obtain a unimodal character enhancement feature corresponding to the unimodal character feature; Using the hidden layer output corresponding to the unimodal text feature and the hidden layer output corresponding to the unimodal character feature, performing a Hadamard product on the unimodal speech feature to obtain a unimodal speech enhancement feature corresponding to the unimodal speech feature; Using the hidden layer output corresponding to the unimodal speech feature and the hidden layer output corresponding to the unimodal character feature, performing a Hadamard product on the unimodal text feature to obtain a unimodal text enhancement feature corresponding to the unimodal text feature; The step of fusing the unimodal text enhancement feature with the unimodal speech enhancement feature and the unimodal person enhancement feature to obtain a first cross-modal feature and a second cross-modal feature includes: The unimodal text enhancement feature is respectively fused with the unimodal speech enhancement feature and the unimodal person enhancement feature through a cross-modal fusion model to obtain a first cross-modal feature and a second cross-modal feature; The cross-modal fusion model includes a position encoding layer, a symmetric attention mechanism layer, a first residual connection and normalization layer, a feedforward network processing layer, and a second residual connection and normalization layer; The position encoding layer is used to construct the position relationship between the unimodal text enhancement feature, the unimodal speech enhancement feature and the unimodal person enhancement feature in the sequence; The symmetric attention mechanism layer is used to use the unimodal text enhancement feature as a query vector, the unimodal speech enhancement feature as a key vector and a value vector, perform a dot product operation using the query vector and the key vector, and after obtaining the attention weight through the softmax function, multiply the attention weight by the value vector to obtain the first symmetric attention mechanism output information; The symmetric attention mechanism layer is also used to use the unimodal text enhancement feature as a query vector, the unimodal person enhancement feature as a key vector and a value vector, perform a dot product operation using the query vector and the key vector, and after obtaining the attention weight through the softmax function, multiply the attention weight by the value vector to obtain the second symmetric attention mechanism output information; The first residual connection and normalization layer is used to perform residual connection and normalization processing on the unimodal text enhancement feature and the output information of the first symmetric attention mechanism to obtain a third cross-modal feature; The first residual connection and normalization layer is further used to perform residual connection and normalization processing on the unimodal text enhancement feature and the output information of the second symmetric attention mechanism to obtain a fourth cross-modal feature; The feedforward network processing layer is used to perform enhancement processing on the third cross-modal feature and the fourth cross-modal feature through a linear rectification function and a linear activation function respectively, so as to obtain a fifth cross-modal feature corresponding to the third cross-modal feature and a sixth cross-modal feature corresponding to the fourth cross-modal feature; The second residual connection and normalization layer is used to perform residual connection and normalization processing on the third cross-modal feature and the fifth cross-modal feature to obtain the first cross-modal feature; The second residual connection and normalization layer is also used to perform residual connection and normalization processing on the fourth cross-modal feature and the sixth cross-modal feature to obtain the second cross-modal feature.

2. The multimodal sentiment analysis method according to claim 1, characterized in that: The step of inputting the first cross-modal feature, the second cross-modal feature, and the unimodal text enhancement feature into a multimodal sentiment analysis model for sentiment analysis to obtain a multimodal sentiment analysis result includes: Adjusting the weights of the first cross-modal feature, the second cross-modal feature, and the unimodal text enhancement feature through an attention mechanism; The first cross-modal feature after adjusting the weight, the second cross-modal feature after adjusting the weight, and the unimodal text enhancement feature after adjusting the weight are input into a multimodal sentiment analysis model for sentiment analysis to obtain a multimodal sentiment analysis result.

3. The multimodal sentiment analysis method according to claim 1, characterized in that: Before inputting the first cross-modal feature, the second cross-modal feature, and the unimodal text enhancement feature into a multimodal sentiment analysis model for sentiment analysis, the method further includes: The first cross-modal feature, the second cross-modal feature, and the unimodal text enhancement feature are concatenated.

4. The multimodal sentiment analysis method according to claim 1, characterized in that: The feature extraction of the speech, text and character image in the video is performed respectively to obtain unimodal speech features, unimodal text features and unimodal character features, including: Extract facial features of the person images in the video to obtain the facial features of the person; Extract action features from character images in the video to obtain character action features; splicing the character facial features and the character action features to obtain the unimodal character features; Extracting features of text in the video to obtain the unimodal text features; Feature extraction is performed on the speech in the video to obtain the unimodal speech feature.

5. The multimodal sentiment analysis method according to claim 4, characterized in that: The facial feature extraction of the character image in the video includes: Extract facial features from the characters in the video using the OpenFace model; The step of extracting action features from a person image in a video includes: The ViTPose model is used to extract action features from character images in the video; The feature extraction of the text in the video includes: Extract features from text in the video using the BERT model; The feature extraction of speech in the video includes: The LibROSA speech toolkit is used to extract the speech features in the video.

6. A multimodal sentiment analysis device, characterized in that: include: A feature extraction unit is used to extract features of speech, text and character images in the video respectively, to obtain unimodal speech features, unimodal text features and unimodal character features; A feature enhancement unit, configured to enhance the feature of any two of the unimodal speech feature, the unimodal text feature and the unimodal character feature, to obtain a unimodal speech enhancement feature, a unimodal text enhancement feature and a unimodal character enhancement feature; A feature fusion unit, used to fuse the unimodal text enhancement feature with the unimodal speech enhancement feature and the unimodal person enhancement feature to obtain a first cross-modal feature and a second cross-modal feature; A sentiment analysis unit, configured to input the first cross-modal feature, the second cross-modal feature, and the unimodal text enhancement feature into a multimodal sentiment analysis model to perform sentiment analysis and obtain a multimodal sentiment analysis result; When the feature enhancement unit performs feature enhancement on another feature other than any two features of the unimodal speech feature, the unimodal text feature and the unimodal character feature based on any two features of the unimodal speech feature, the unimodal text feature and the unimodal character feature to obtain the unimodal speech enhancement feature, the unimodal text enhancement feature and the unimodal character enhancement feature, the feature enhancement unit is specifically used to: Inputting the unimodal speech feature, the unimodal text feature and the unimodal character feature into the LSTM network respectively, and obtaining the hidden layer output corresponding to the unimodal speech feature, the hidden layer output corresponding to the unimodal text feature and the hidden layer output corresponding to the unimodal character feature; Performing a Hadamard product on the unimodal character feature using the hidden layer output corresponding to the unimodal speech feature and the hidden layer output corresponding to the unimodal text feature to obtain a unimodal character enhancement feature corresponding to the unimodal character feature; Using the hidden layer output corresponding to the unimodal text feature and the hidden layer output corresponding to the unimodal character feature, performing a Hadamard product on the unimodal speech feature to obtain a unimodal speech enhancement feature corresponding to the unimodal speech feature; Using the hidden layer output corresponding to the unimodal speech feature and the hidden layer output corresponding to the unimodal character feature, performing a Hadamard product on the unimodal text feature to obtain a unimodal text enhancement feature corresponding to the unimodal text feature; When the feature fusion unit is used to fuse the unimodal text enhancement feature with the unimodal speech enhancement feature and the unimodal person enhancement feature to obtain the first cross-modal feature and the second cross-modal feature, it is specifically used to: The unimodal text enhancement feature is respectively fused with the unimodal speech enhancement feature and the unimodal person enhancement feature through a cross-modal fusion model to obtain a first cross-modal feature and a second cross-modal feature; The cross-modal fusion model includes a position encoding layer, a symmetric attention mechanism layer, a first residual connection and normalization layer, a feedforward network processing layer, and a second residual connection and normalization layer; The position encoding layer is used to construct the position relationship between the unimodal text enhancement feature, the unimodal speech enhancement feature and the unimodal person enhancement feature in the sequence; The symmetric attention mechanism layer is used to use the unimodal text enhancement feature as a query vector, the unimodal speech enhancement feature as a key vector and a value vector, perform a dot product operation using the query vector and the key vector, and after obtaining the attention weight through the softmax function, multiply the attention weight by the value vector to obtain the first symmetric attention mechanism output information; The symmetric attention mechanism layer is also used to use the unimodal text enhancement feature as a query vector, the unimodal person enhancement feature as a key vector and a value vector, perform a dot product operation using the query vector and the key vector, and after obtaining the attention weight through the softmax function, multiply the attention weight by the value vector to obtain the second symmetric attention mechanism output information; The first residual connection and normalization layer is used to perform residual connection and normalization processing on the unimodal text enhancement feature and the output information of the first symmetric attention mechanism to obtain a third cross-modal feature; The first residual connection and normalization layer is further used to perform residual connection and normalization processing on the unimodal text enhancement feature and the output information of the second symmetric attention mechanism to obtain a fourth cross-modal feature; The feedforward network processing layer is used to perform enhancement processing on the third cross-modal feature and the fourth cross-modal feature through a linear rectification function and a linear activation function respectively, so as to obtain a fifth cross-modal feature corresponding to the third cross-modal feature and a sixth cross-modal feature corresponding to the fourth cross-modal feature; The second residual connection and normalization layer is used to perform residual connection and normalization processing on the third cross-modal feature and the fifth cross-modal feature to obtain the first cross-modal feature; The second residual connection and normalization layer is also used to perform residual connection and normalization processing on the fourth cross-modal feature and the sixth cross-modal feature to obtain the second cross-modal feature.

7. An electronic device, characterized in that: It includes a memory, a processor and a transceiver which are communicatively connected in sequence, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program to execute the multimodal sentiment analysis method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the multimodal sentiment analysis method according to any one of claims 1 to 5 is executed.