Multi-modal sentiment analysis model based on multi-head attention perception fusion
By using a multimodal sentiment analysis model based on multi-head attention perception fusion, the problem of insufficient accuracy in multi-domain sentiment analysis is solved, achieving higher analysis accuracy and recommendation efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HAINAN UNIV
- Filing Date
- 2024-11-08
- Publication Date
- 2026-05-08
AI Technical Summary
Existing multimodal sentiment analysis models have low accuracy and poor effectiveness in various applications.
We employ a multimodal sentiment analysis method based on multi-head attention perception fusion. By extracting features from different modalities and using a multi-head attention mechanism, we combine multimodal datasets to perform feature interaction and cross-forward fusion, thereby improving the accuracy of sentiment analysis.
The model significantly improves the accuracy and efficiency of recommendations in multi-domain sentiment analysis, and its superiority has been verified through multiple experiments.
Smart Images
Figure FT_1 
Figure FT_2 
Figure FT_3
Abstract
Description
Technical Field
[0001] This invention belongs to the field of sentiment analysis in natural language processing, and designs a sentiment analysis model based on multi-head attention perception fusion. Background Technology
[0002] In e-commerce, movie websites, and online advertising, to improve user experience, there's a desire to accurately recommend items that users might be interested in, including goods and movies, while browsing websites or using software. Users generate a large amount of data during their interactions; using recommendation models to calculate user preferences from this data not only significantly reduces manual intervention but also effectively improves the accuracy and speed of recommendations.
[0003] Current sentiment analysis models suffer from insufficient accuracy and limited application areas. In other fields, modifications are often required to achieve basic recommendation accuracy.
[0004] In summary, current multimodal sentiment models lack sufficient accuracy in multimodal datasets and exhibit poor effectiveness across multiple domains, resulting in room for improvement in overall analytical accuracy. Summary of the Invention
[0005] To address the shortcomings of existing technologies in multimodal research regarding analytical accuracy, this invention provides a multimodal sentiment analysis method based on multi-head attention perception fusion. This method solves the single-modality problem in sentiment analysis and improves analytical accuracy. The technical solution is as follows:
[0006] The main steps of a multimodal sentiment analysis model based on multi-head attention perception fusion are as follows:
[0007] (1) Considering the complexity of multiple domains and the diversity of input data, we propose modal analysis and different methods to extract features from multimodal mixed datasets;
[0008] (2) After extracting features for different categories from the multimodal dataset, we combine a multi-head attention mechanism to ensure that each modality can pay more attention to the sentiment label information and make more accurate judgments on the extracted feature information.
[0009] (3) After labeling the modal with key emotional information through the multi-head attention mechanism, multimodal fusion is started using the corresponding information;
[0010] (4) The fusion process adopts interactive forward fusion, which goes through a multi-head attention mechanism and two interactive processes of fusion, and extracts emotional information more accurately;
[0011] (5) Under the same environment, multiple comparative experiments with other baseline models were conducted, and the experimental results showed that the accuracy of the proposed model was improved. Attached Figure Description
[0012] To more clearly illustrate the specific technical solutions of the present invention, the accompanying drawings will be described below.
[0013] Figure 1 It is a diagram of a sentiment analysis model that combines data input, feature extraction, and multi-head attention fusion.
[0014] Figure 2 This is a diagram of the internal model of the multi-head attention mechanism;
[0015] Figure 3 This is a schematic diagram of the overall operation;
[0016] Figure 4 This is a comparative analysis chart of the experimental results. Detailed Implementation
[0017] The present invention will now be described with reference to the accompanying drawings.
[0018] This invention proposes a multimodal sentiment analysis model based on multi-head attention perception fusion. This model addresses the problem of improving the accuracy of sentiment analysis in multiple modalities. Through multiple experiments and data comparison with various baseline models, it has been proven that it can effectively improve the accuracy and efficiency of recommendations.
[0019] In the field of multimodal sentiment analysis, the model acquires relevant data from a multimodal dataset, extracts important modal feature information from the data, focuses on sentiment information through a multi-head attention mechanism, performs feature interactions between modalities, and finally performs cross-forward fusion to obtain the prediction result.
[0020] The specific operation process of this invention is as follows:
[0021] 1. Collect a multimodal dataset, which can be divided into three parts: training, validation, and testing. It also includes data from different modalities: text, acoustic, and visual, and performs feature extraction.
[0022] 2. In terms of text feature research, the BERT model is used to extract features from text modalities.
[0023] 3. In terms of acoustic feature research, the COVAREP correlation method is used to extract acoustic features.
[0024] 4. In the study of visual features, Facet is used for visual feature extraction.
[0025] 5. After feature extraction is complete, the mean of the features can be calculated using a formula and then divided by the total number of input data samples, N. This will give you the mean of the data set.
[0026] 6. After extracting features for different categories from the multimodal dataset, a multi-head attention mechanism is combined to ensure that each modality can pay more attention to sentiment label information, thereby improving the accuracy of subsequent multimodal sentiment analysis.
[0027] 7. Use three built-in weight matrices to perform a linear transformation on the input matrix X, representing the query, key, and value, respectively.
[0028] 8. After obtaining the three values, calculate the attention weight V, where Q, K, and V represent the linear transformations of query, key, and value, respectively, is the linear transformation matrix of each attention head i, and is the dimension of query or key.
[0029] 9. Finally, all attention points are concatenated, and in the formula, represents the output weight matrix (see...). Figure 2 ).
[0030] 10. After labeling the modalities with key sentiment information using a multi-head attention mechanism, multimodal fusion analysis is then performed.
[0031] 11. Using the extracted feature vectors as input, the linear layer matches the dimensions between modalities, and the linear layer projects the modalities onto the same space to form a new matrix.
[0032] 12. Next, fusion is performed, connecting the two different modalities A and B, and interspersing segments of a third modality to perform multimodal fusion (see...). Figure 1 ).
[0033] 13. The model was trained using the CMU-MOSI and CMU-MOSEI public datasets, and by comparing it with other baseline models, the proposed method was deemed the optimal choice (see [link to data]). Figure 4 ).
Claims
1. Collect a multimodal dataset, which can be divided into three parts: training, validation, and testing. It also includes data from different modalities: text, acoustic, and visual, and performs feature extraction.
2. In terms of text feature research, the BERT model is used to extract features from text modalities.
3. In terms of acoustic feature research, the COVAREP correlation method is used to extract acoustic features.
4. In the study of visual features, Facet is used for visual feature extraction.
5. After feature extraction is complete, the mean of the features can be calculated using a formula and then divided by the total number of input data samples, N. This will give you the mean of the data set.
6. After extracting features for different categories from the multimodal dataset, a multi-head attention mechanism is combined to ensure that each modality can pay more attention to sentiment label information, thereby improving the accuracy of subsequent multimodal sentiment analysis.
7. Use three built-in weight matrices to perform a linear transformation on the input matrix X, representing the query, key, and value, respectively.
8. After obtaining the three values, calculate the attention weight V, where Q, K, and V represent the linear transformations of query, key, and value, respectively, is the linear transformation matrix of each attention head i, and is the dimension of query or key.
9. Finally, all attention points are concatenated, and the formula represents the output weight matrix (see Figure 2).
10. After labeling the modalities with key sentiment information using a multi-head attention mechanism, multimodal fusion analysis is then performed.
11. Using the extracted feature vectors as input, the linear layer matches the dimensions between modalities, and the linear layer projects the modalities onto the same space to form a new matrix.
12. Next, the two different modes A and B are connected and a third mode segment is inserted to perform multimodal fusion (see Figure 1).
13. The model was trained using the CMU-MOSI and CMU-MOSEI public datasets, and the proposed method was found to be the optimal choice by comparing it with other baseline models (see Figure 4).