A sentiment classification method based on a multi-modal attention mechanism
By constructing a heterogeneous multimodal interaction graph and residual connections, this method solves the problem that existing multimodal fusion methods fail to effectively utilize the high temporal frequency characteristics and modal interaction relationships of video and audio, thus achieving more efficient emotion recognition results.
Patent Information
- Application Number
- CN202111368526.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-18
- Publication Date
- 2025-12-12
- Estimated Expiration
- 2041-11-18
AI Technical Summary
Existing multimodal fusion methods fail to effectively consider the high temporal frequency characteristics of video and audio behavior, resulting in insufficient fine-grained multimodal information interaction, which affects model performance. Furthermore, they ignore the interaction relationships between multiple modalities, which affects the performance of sentiment analysis tasks.
By segmenting aligned multimodal sequence information, a heterogeneous multimodal interaction graph is constructed. A multimodal attention mechanism is used to capture the interaction relationships between modalities, and residual connections are used to compensate for the continuity within the modalities to obtain a new multimodal representation, which is then fed into a pre-trained model for emotion recognition.
It improves the accuracy of sentiment recognition based on multimodal data, enhances the model's generalization ability and robustness, achieves fine-grained multimodal information fusion, and improves the accuracy of sentiment classification.
Smart Images
Figure CN114169408B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the field of multi-modal sentiment recognition in the cross field of natural language processing, video and audio, and specifically relates to a method for judging the emotional state of a subject by using an attention mechanism to fuse multi-modal information and integrate different single-modal information into a compact multi-modal representation. BACKGROUND
[0002] The field of sentiment analysis usually contains data such as text, video and audio. In previous work, it has been proven that these single-modal data usually contain recognition information related to emotional state. At the same time, through a large number of studies, it has been found that the consistency and complementarity existing between these single-modal data can effectively explain the internal correlation representation of multi-modal data, further enhance the generalization ability and robustness of the model, and improve the performance of the sentiment analysis task.
[0003] Existing fusion models based on attention mechanism can integrate information extracted from different single-modal data sources into a compact multi-modal representation. Therefore, it has attracted more and more attention from researchers. Specifically, based on the language modality, the attention coefficients between the information of the other two modalities (video and audio) and the information of the language modality are obtained by using the attention mechanism, and then the fusion is performed according to the attention coefficients. However, this only considers the interaction between the two modalities, ignores the interaction relationship between the multiple modal information, and greatly affects the performance of the model. In addition, because the time frequency of video and audio behavior is usually much higher than that of text, this leads to a series of video and audio "sub-word" units accompanying each word uttered. However, the existing multi-modal fusion method rarely takes this fact into account and fails to realize the interaction of fine-grained multi-modal information. Thus affecting the final task performance. SUMMARY
[0004] An object of the present application is to propose a sentiment classification method based on a multi-modal attention mechanism in view of the deficiencies of existing multi-modal fusion technologies. First, the aligned multi-modal sequence information is divided to obtain fine-grained multi-modal information. Then, the multi-modal sequence data after division is organized into a heterogeneous multi-modal interaction graph. In this interaction graph, the rich and complex interaction relationship between fine-grained multi-modal information is captured by means of the rich interaction between heterogeneous graph nodes, which constitutes the interaction between the three modalities of text, video and audio. Then, the compact multi-modal representation obtained by the multi-modal attention mechanism on the interaction graph is used to offset the position of the word information in the original semantic space. Since the division of the modalities will destroy the continuity within the modalities, the continuity within the modalities is reasonably compensated by a scaling factor in a residual manner to obtain new multi-modal representations. Finally, the new multi-modal representations are sent into a large pre-trained model to perform a sentiment recognition task.
[0005] The present application is a multi-modal attention mechanism-based fusion method, comprising the following steps:
[0006] A sentiment classification method based on a multi-modal attention mechanism, comprising the following specific steps:
[0007] Step 1: Obtain multi-modal feature information under different emotional categories.
[0008] Step 2: Preprocess the multi-modal feature information to extract the primary representation h v , h a , and h l of the audio modality, the video modality and the language modality.
[0009] Step 3: Construction of a heterogeneous multi-modal interaction graph:
[0010] 3-1. Perform a segmentation operation on the aligned multi-modal data. A triple (L k , V k , A k ) is used to represent the three modality feature information corresponding to the kth word. The language, video and audio modalities are divided into where N represents the number of divided nodes. L k , V k , and A k are connected in turn to form where M = 3N.
[0011] 3-2. Calculate the importance e ij of each language modality node relative to itself and all video and audio modality nodes as follows:
[0012]
[0013] where W is the weight matrix. a(·) represents the node performing attention mechanism operation; j∈{i}∪[N+1,M]; i=1,2,...,N.
[0014] 3-3. Importance e between all adjacent nodes ij Normalization processing is performed to obtain the corresponding correlation coefficient α ij .
[0015] 3-4. Update the language modal node using the correlation coefficient between different nodes. The update expression of the i-th language modal node is as follows:
[0016]
[0017] 3-5. Concatenate all the updated language modal nodes of a word to obtain the multimodal representation h′ k as follows:
[0018]
[0019] 3-6. Supplement the continuity within the video and audio modalities by the scaling factor β to obtain the updated multimodal representation s i as follows:
[0020]
[0021] where, represents element-wise addition; β is the scaling factor.
[0022] 3-7. Offset the position of the word in the semantic space in the language modal to obtain the offset language modal information L' i as follows:
[0023] L′ k = L k + s k
[0024] Step 4, after training the classification model using the language modal information L' i , the classification model is used to identify the multimodal information of the test subject to obtain the emotional category of the test subject when collecting the multimodal information.
[0025] As a preferred, in step 1, the multimodal data is recorded under the condition that the subject performs a specific emotional task, including language modal, video modal and audio modal.
[0026] As a preferred, in step 2, the primary representation hv , h a , the initial audio modal, video modal feature information is obtained by processing through a long short-term memory network; the primary representation h l , the language modal feature information is obtained by mapping operation.
[0027] As a preferred, in step 3-6, the scaling factor β is determined by cross-validation machine.
[0028] As a preferred, in step 3-3, the normalization is carried out by using a softmax function.
[0029] As a preferred, in step 4, the classification model adopts BERT algorithm; the language modal information L' i is input into the Transformers layer. Training is carried out.
[0030] On this basis, the application further provides an electronic device; it includes a central processor and a memory; the memory stores machine executable instructions capable of being executed by the processor, and the central processor executes the machine executable instructions to realize the aforementioned sentiment classification method.
[0031] The beneficial effects of the application are:
[0032] The application combines attention mechanism and large pre-training language model, and in the process of converging multiple modal information, rich and complex interactions between heterogeneous graph nodes are used to construct simultaneous interactions between multiple modalities. Relying on the attention mechanism, fine-grained multi-modal information fusion is realized, and the multi-modal representation information is used to offset the original position of the word in the semantic space in the language modal. After the offset operation, the language modal information is injected into the video and audio modal information. At the same time, the new language modal information obtained is sent into a large pre-training language model, so as to capture the context correlation between text information. This fusion framework is an effective supplement to the current multi-modal sentiment recognition field, and can further improve the sentiment recognition accuracy of multi-modal data. BRIEF DESCRIPTION OF DRAWINGS
[0033] Figure 1 is a flowchart of the application;
[0034] Figure 2 is a schematic diagram of multi-modal fusion in step 3 of the application;
[0035] Figure 3 is a whole schematic diagram of step 3 of the application;
[0036] Figure 4 is a schematic diagram of the fusion framework of the application. DETAILED DESCRIPTION
[0037] The application will be described in detail below with reference to the accompanying drawings.
[0038] As shown in Figure 1 and 4 , a sentiment classification method based on a multi-modal attention mechanism, the specific steps are as follows:
[0039] Step 1, obtaining multi-modal original low-level feature data
[0040] Under the condition that the subject performs a specific emotion task, record the multi-modal data emitted by the subject.
[0041] The multi-modal data includes but is not limited to: language modal, video modal and audio modal;
[0042] One of the purposes of multi-modal fusion is to combine two or more modal information for prediction, which is used to make up for the limitations of single modal information expression ability, so as to improve the robustness of multi-modal model;
[0043] Step 2, multi-modal information data preprocessing: extracting multi-modal information linear transformation primary representation
[0044] Multi-modal fusion is carried out at the feature level, and long short-term memory network is used to extract primary features of sequence video and audio modal data, and language modal is mapped and then primary features are extracted;
[0045] h v = LSTM(V)
[0046] h a = LSTM(A)
[0047] h l = Embedding(L) Formula (1)
[0048] Wherein, is the primary representation of the mth modal, m ∈ {v, a, l}; v, a, l are audio, video and language modal respectively; V, A, L are the original data of the three modalities; LSTM represents long short-term memory network; Embedding represents mapping operation; T m is the size of time domain dimension, d m is the length of feature vector at each time;
[0049] Step 3, as shown in Figure 2 and 3 , the construction of heterogeneous multi-modal interaction graph:
[0050] One of the core processing modules included in the framework of multi-modal fusion is the multi-modal interaction graph. In the process of multi-modal information fusion, there are several such interaction graphs for converging various different unimodal data into a compact multi-modal representation to offset the position of words in the original semantic space in the language modality. Based on the fact that the time frequency of video and audio behaviors is usually much higher than that of text, the core of the multi-modal interaction graph is to use fine-grained multi-modal information to construct the interaction between different modalities.
[0051] In order to obtain fine-grained multi-modal information, the aligned multi-modal sequence data is first subjected to a segmentation operation. The k-th word is represented by a triple (L k ,V k ,A k ), where L represents the language modality feature information, V represents the video modality feature information, and A represents the audio modality feature information. After the language modality feature information is divided into smaller granularity, it is represented as where N represents the number of nodes of division. Similarly, the division of the video and audio modalities can be represented as For convenience of representation, we use to replace the above equation, where M = 3*N. In this embodiment, N = 2; from 1 to N represents the language modality feature information, N+1 to 2N represents the video modality feature information, and 2N+1 to M (3N) represents the audio modality feature information. Each node representing the language modality and all nodes representing the audio and video will be organized into a multi-modal interaction graph. In this graph, text and video and text and audio can interact, for the next fusion operation. The multi-modal interaction graph refers to introducing multi-modal data into a graph structure, using the edges between different nodes to construct the interaction relationship between multi-modalities, taking the language modality as the core node, and constructing the interaction graph between audio, video and language.
[0052] In order to obtain sufficient expression capability to convert the input features into higher level features, at least one learnable linear transformation is required. For this purpose, first, a shared linear transformation parameterized by a weight matrix W ∈ R d×d is performed on each node. Then, an attention mechanism operation (a: R 2d → R, a shared attention mechanism) is performed on the nodes to calculate the attention coefficients between nodes. Assuming that e i,j represents the importance of node j to node i, the above process can be represented as:
[0053]
[0054] where The language modality node in the processed multi-modal interaction graph; j e {i} U [N+1, M], which realizes the operation of each language modality node with itself and other nodes. After obtaining the importance coefficients representing the language modality nodes and all nodes of the video and audio modalities, in order to facilitate the comparison of the attention coefficient size between different nodes, normalization operation is performed on them:
[0055]
[0056] Wherein, softmax() represents a normalization processing function. N i All nodes connected to node i in the graph (including itself) are represented as a set. Since the attention mechanism a is a single-layer feedforward neural network, we add a nonlinear activation function R(x) based on it:
[0057]
[0058] γ is the parameter of the nonlinear activation function, which is a constant between 0 and 1.
[0059] α ij Can be represented as:
[0060]
[0061] Wherein, R(·) is a nonlinear activation function, T represents transposition, || represents concatenation, and a is the transformed importance coefficient. Then the correlation coefficient between different nodes is used to update the node representing the language modality. The i-th language modality node can be represented as:
[0062]
[0063] Then, all nodes of the language modality of the k-th word are concatenated to obtain the multi-modal representation h' k :
[0064]
[0065] We reasonably supplement the continuity of the video and audio modalities through a scaling factor β (β < 1) without affecting the above operations, and obtain the updated multi-modal representation s i .
[0066]
[0067] Wherein, represents element-wise addition; β is a hyperparameter determined by cross-validation machine;
[0068] The multi-modal representation s kobtaining the language modal information L' after offsetting the position of the word in the semantic space in the language modal i As formula (9).
[0069] L' k = L k +s k Formula (9)
[0070] Step 4, continue pre-training:
[0071] According to step 3, for each triple (L k ,V k ,A k ) representing the multi-modal information, first, a segmentation operation is performed to change it into fine-grained multi-modal information, and the fine-grained multi-modal information is organized into several interactive graphs to obtain the output spliced together, and then a residual connection is used to supplement the continuity of the modal, forming a new multi-modal representation. The position of the word in the semantic space in the language modal is offset using the new multi-modal representation, and a new language modal representation is obtained. In the new language modal representation, video and audio modal information is added, greatly supplementing the limitations of the expression ability of single language modal information.
[0072] Suppose there are N triples representing multi-modal information in the aligned multi-modal sequence data, a special mark (CLS) is added in front of each triple as a label for multi-modal sentiment classification. Each triple will obtain a new language modal representation after the above operation, and {CLS, L'1, L'2,..., L'N} which has gathered multi-modal information is sent into the Transformers layer of BERT for further training. The sentiment classification label output by the model is compared with the original annotated sentiment classification label to update the parameters of the model, and the final classification result is obtained to complete the sentiment classification task. N
[0073] The present application and various existing multi-modal fusion methods are used to perform sentiment state discrimination tasks on two public multi-modal emotion databases CMU-MOSI and CMU-MOSEI, and the results are shown in Tables 1 and 2. The results in the table are the mean absolute error MAE, the correlation coefficient Corr, the precision Acc-2 corresponding to the sentiment binary classification task, the F1 score F1-Score, and the precision Acc-7 corresponding to the sentiment seven classification task. It can be seen that compared with the existing multi-modal fusion framework which shows excellent level, the five evaluation indexes of the present application are better than the existing fusion model, proving the effectiveness of the method proposed by the present application.
[0074] Table 1. Comparison of CMU-MOSI results
[0075]
[0076] Table 2. CMU-MOSEI results comparison table
[0077]
Claims
1. A method for sentiment classification based on multi-modal attention mechanism, characterized in that: Step 1, obtaining multi-modal feature information under different emotion categories; Step 2, pre-processing the multi-modal feature information, extracting the primary representation of the audio modality, the video modality and the language modality 、 、 ; the primary representation of the audio modality and the video modality 、 is obtained by processing the initial audio modality and video modality feature information through a long short-term memory network; the primary representation of the language modality is obtained by mapping the language modality feature information; Step 3, construction of heterogeneous multi-modal interaction graph: 3-1. Perform a segmentation operation on the aligned multi-modal data; use a triple to represent the three modal feature information corresponding to the first word The language, video and audio modalities are respectively divided into , , ; wherein N represents the number of divided nodes; the , , are sequentially connected to form ; wherein M=3N; 3-2. For each language modality node its importance is computed with respect to itself and all video and audio modality nodes, respectively as follows: ； wherein, is a weight matrix; a(·) denotes that the node performs an attention mechanism; ; i = 1, 2,..., N; 3-3. Importance between all adjacent nodes Normalization is performed to obtain the corresponding correlation coefficient ; 3-4. Update the language modality nodes with the correlation coefficients between different nodes; the update expression of the i-th language modality node is as follows: ； Wherein, R(·) is a nonlinear activation function. 3-5. Concatenate all language modal nodes of a word after updating to get multi-modal representation As follows: ； 3-6. By scaling factor Supplementing the continuity within the video and audio modalities, resulting in updated multi-modal representations As follows: ； wherein denotes an element-wise addition; is a scaling factor; said scaling factor is determined by a cross-validation machine; 3-7. Shift the position of the word in the semantic space in the language modality, obtain the language modality information after shifting As follows: ； Step 4, using language modal information After training the classification model, the multi-modal information of the testee is recognized using the classification model to obtain the emotional category of the testee when the multi-modal information is collected.
2. The method of claim 1, wherein the method is based on a multi-modal attention mechanism. In step 1, the multi-modal data is recorded under the execution of a specific emotion task by a subject, including a language mode, a video mode and an audio mode.
3. The method of claim 1, wherein the method is based on a multi-modal attention mechanism. In step 3-3, a softmax function is used for normalization.
4. The method of claim 1, wherein the method is based on a multi-modal attention mechanism. In step 4, the classification model employs a BERT algorithm; language modality information input transformers layer; training is performed.
5. An electronic device comprising a central processing unit and a memory; characterized by: The memory stores machine executable instructions capable of being executed by the processor, and the central processor executes the machine executable instructions to implement the method in any one of claims 1-4.