Multi-modal emotion data analysis method
By extracting and fusing features at different levels in multimodal sentiment analysis, and performing noise reduction and decoupling, the problems of information redundancy and missing information are solved, improving the accuracy and robustness of sentiment analysis, especially in complex emotional scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-31
- Publication Date
- 2026-04-10
AI Technical Summary
Existing multimodal sentiment analysis methods often neglect mid-level or even shallow-level feature information, resulting in both information redundancy and missing information. This affects the robustness and accuracy of sentiment analysis, especially in complex emotional expressions or weak emotional scenarios.
By extracting features from different levels of multimodal sentiment data, feature fusion and enhancement are performed. Noise reduction and compression are carried out using the multimodal information bottleneck module. The features are decoupled into shared features and multimodal private features, and multi-view feature aggregation is performed to finally output the sentiment intensity.
It significantly improves the accuracy and stability of multimodal sentiment analysis. By effectively utilizing shallow and medium-level semantic information, it enhances the robustness and accuracy of sentiment recognition, especially performing better in complex emotional scenarios.
Smart Images
Figure CN121834701A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of artificial intelligence, in particular to a multi-modal sentiment data analysis method. BACKGROUND
[0002] With the rapid development of Internet technology, social media platforms such as microblog, TikTok, and Xiaohongshu have deeply integrated into people's daily life. Users publish and interact information through various forms such as text, audio, image, and video, which has given birth to large-scale, multi-modal data resources. Multi-modal data can depict the emotional state and expression intention of users from different perspectives, and has gradually become an important carrier for people to express emotions and convey opinions, which also provides rich data basis and broad application scenarios for the development of multi-modal sentiment computing. Compared with single-modal sentiment analysis methods, multi-modal sentiment analysis can comprehensively utilize the semantic information of text, the prosodic features of audio, and the expression and action information of visual modal, thereby having obvious advantages in the robustness and accuracy of sentiment discrimination. Therefore, how to efficiently and fully mine and fuse multi-modal information to achieve more accurate sentiment prediction has become an important problem that needs in-depth research in the current sentiment computing field.
[0003] However, the existing multi-modal sentiment analysis method still has certain limitations. In recent years, most of the existing technologies rely on pre-training models to extract features of different modalities, and carry out cross-modal fusion on this basis, but there is a problem of coexistence of information redundancy and information loss in the feature utilization strategy. Specifically, existing technologies generally focus on using high-level semantic features of pre-training models, that is, directly using the last layer output for multi-modal fusion, and less attention is paid to the fine-grained semantic and structural information contained in the middle and even shallow features. These middle and shallow features still have an irreplaceable role in describing emotional clues, emotional intensity, and emotional changes, especially in complex emotional expression or weak emotional scenarios, and their complementary value is more prominent. Therefore, how to effectively model the complementary relationship between different levels of features in the multi-modal sentiment analysis framework, fully mine the middle and shallow semantic information, and realize the collaborative fusion of hierarchical features is one of the key research directions to improve the performance and generalization ability of multi-modal sentiment analysis. SUMMARY
[0004] The present application provides a multi-modal sentiment data analysis method, which can solve the problem of coexistence of information redundancy and information loss caused by ignoring middle and even shallow feature information.
[0005] In order to achieve the above-mentioned purpose, according to the first aspect of the present application, a multi-modal sentiment data analysis method is provided, comprising the following steps: S1, extract features of different levels in multi-modal sentiment data, perform feature fusion and feature enhancement on the features of different levels, and output fused and enhanced features; S2, denoise and compress the fused and enhanced features, and output semantic sentiment features; S3, the semantic sentiment features output shared features and multi-modal private features through multi-modal decoupling; S4, multi-view feature aggregation is performed on the shared features and multi-modal private features, and finally the sentiment intensity is output.
[0006] Optionally, the S1 further includes the following steps: inputting the multi-modal sentiment data into a pre-trained fine-tuning encoder to obtain encoder features , wherein ; fine-tuning the last layer of the encoder features :
[0007]
[0008] wherein, is a loss function of a single-modal encoder, is a predicted value output by the single-modal encoder, is a real sentiment label.
[0009] Further, the fine-tuning adopts an early stopping strategy; after the fine-tuning is completed, audio encoding features , text encoding features and video encoding features are extracted from the encoder features ; wherein, is the number of layers of the audio encoder, is the dimension of the audio encoding features, is the number of layers of the text encoder, is the dimension of the text encoding features, is the number of layers of the video encoder, is the dimension of the video encoding features.
[0010] Optionally, the S1 further includes the following steps: selecting initial semantic features from the encoder features and performing enhancement fusion, wherein , is the length of the modal sequence, is the modal feature dimension; transmitting the initial semantic features to a weight linear layer and combining with projection transformation features The summation of the dot products yields the adaptive weighted summation features. Initial semantic features The data is transmitted to a linear layer along the feature dimension to obtain the projection transformation features. Projection transformation characteristics With classification head After being concatenated, the data is transmitted to the Transformer layer to obtain the interaction features between the shallow and deep layers. After concatenating the deep and shallow layer interaction features and the adaptive weighted summation features, the fused enhanced features are output through an enhanced feature linear layer. .
[0011] Optionally, step S2 further includes the following steps: By using the MIB layer, i.e., the multimodal information bottleneck module, to enhance fused features. Optimize noise reduction:
[0012] in, For multimodal real labels, This is the hidden state of a single-modal system. For single-modal task loss coefficients, For mutual information calculation, Compressive strength; The MIB layer retains the output target information and compresses and fuses enhanced features. semantic sentiment features were obtained. .
[0013] Optionally, step S3 further includes the following steps: Semantic sentiment features Decoupled into shared features by the following formula and multimodal private features :
[0014]
[0015]
[0016]
[0017] in, and All are linear layers. Shared spatial mapping features, including shared spatial mapping text features. Shared spatial mapping audio features and shared spatial mapping video features , This involves concatenating the classification header with three single-modal structures. This is the classification header.
[0018] Optionally, step S3 further includes the following steps: Shared features Apply constraints:
[0019] in, Predicted labels for shared features; Shared features By extracting the encoder Extract shared features from the corresponding modalities and private features from the multimodalities. After splicing, the encoder is restored. Reconstruction And to restore it to its original semantic and emotional features as much as possible. :
[0020] in, To reconstruct the features of the corresponding mode, the encoder is recovered. and extract encoder All are composed of linear layers.
[0021] Furthermore, the multimodal private features Including text-private features Audio private features and video private features ; S4 also includes the following steps: Shared features Text private features Audio private features and video private features Combine:
[0022]
[0023] The combined data is processed through the same Transformer layer to obtain viewpoint information. :
[0024] in, Emotional intensity is calculated using an average pooling layer.
[0025]
[0026] in, Emotional intensity.
[0027] Optionally, S4 further includes the following steps: Optimize the network using a loss function:
[0028]
[0029]
[0030]
[0031]
[0032] in, For multimodal loss function, For sentiment loss function with shared features, For multimodal real labels, For orthogonal loss, For private features of a single modality, To rebuild the losses, This is the total loss function; Set up several training rounds. When all training rounds are completed, stop training the model and output the final sentiment intensity. .
[0033] To achieve the above objectives, according to a second aspect of this application, a multimodal sentiment data analysis system is also provided. The system executes the multimodal sentiment data analysis method described in the first aspect. The system includes a feature enhancement module, a multimodal information bottleneck module, a multimodal decoupling module, and a multi-perspective feature aggregation module. The feature enhancement module receives multimodal sentiment data and outputs fused enhanced features. The multimodal information bottleneck module receives the fused enhanced features and outputs semantic sentiment features. The multimodal decoupling module receives the semantic sentiment features and outputs shared features and multimodal private features. The multi-perspective feature aggregation module receives the shared features and multimodal private features and finally outputs the sentiment intensity.
[0034] In this application, by introducing feature representations at different depth levels and using a feature enhancement module for adaptive weighted fusion, the limitations of single-level features in semantic expression are effectively compensated, thereby significantly enriching the multimodal feature representation capabilities. This enables the subsequent multimodal fusion stage to acquire more comprehensive and discriminative emotional semantic information, thus improving the overall accuracy and stability of emotion recognition. Before multimodal feature decoupling, the multimodal features are first enhanced, and a multimodal information bottleneck mechanism is introduced to constrain, denoise, and compress the enhanced features. This allows the model to actively suppress noise information that is irrelevant to the emotion recognition task or redundant during information transmission, retaining only the core representations that are highly relevant to emotion discrimination. Through the above information bottleneck processing, not only is the negative impact of noise interference and redundant information between different modalities on the decoupling process reduced, but the features input to the decoupling module are also made more compact, pure, and have stronger task orientation, thereby significantly improving the effectiveness and robustness of multimodal feature decoupling and facilitating the acquisition of more accurate and separable multimodal decoupling output results. Attached Figure Description
[0035] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0036] Figure 1 This is a network structure diagram for sentiment data analysis provided according to embodiments of this application; Figure 2 This is a flowchart of a multimodal sentiment data analysis method provided according to an embodiment of this application; Figure 3 This is a structural diagram of a multimodal sentiment data analysis system provided according to an embodiment of this application. Detailed Implementation
[0037] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0038] Example 1 like Figure 1 As shown, this embodiment provides a multimodal sentiment data analysis method including the following steps, such as... Figure 2 : S1. Extract features at different levels from multimodal sentiment data, perform feature fusion and feature enhancement on features at different levels, and output fused and enhanced features; Multimodal sentiment data is input into a pre-trained, fine-tuned encoder to obtain encoder features. ,in ; for encoder features Make minor adjustments to the last layer:
[0039]
[0040] in, The loss function of a single-mode encoder. The predicted value output by the single-mode encoder. Labels representing genuine emotions; From encoder features Selecting initial semantic features And enhance integration, among which , The length of the modal sequence. For modal feature dimensions; Initial semantic features After being transmitted to the weighted linear layer, the features are transformed by projection. The summation of the dot products yields the adaptive weighted summation features. Initial semantic features The data is transmitted to a linear layer along the feature dimension to obtain the projection transformation features. Projection transformation characteristics With classification head After being concatenated, the data is transmitted to the Transformer layer to obtain the interaction features between the shallow and deep layers. After concatenating the deep and shallow layer interaction features and the adaptive weighted summation features, the fused enhanced features are output through an enhanced feature linear layer. .
[0041] S2. Denoise and compress the fused and enhanced features to output semantic sentiment features; By using the MIB layer, i.e., the multimodal information bottleneck module, to enhance fused features. Optimize noise reduction:
[0042] in, For multimodal real labels, This is the hidden state of a single-modal system. For single-modal task loss coefficients, For mutual information calculation, Compressive strength; The MIB layer retains the output target information and compresses and fuses enhanced features. semantic sentiment features were obtained. .
[0043] S3. Semantic sentiment features output shared features and multimodal private features through multimodal decoupling; Semantic sentiment features Decoupled into shared features by the following formula and multimodal private features :
[0044]
[0045]
[0046]
[0047] in, and All are linear layers. Shared spatial mapping features, including shared spatial mapping text features. Shared spatial mapping audio features and shared spatial mapping video features , This involves concatenating the classification header with three single-modal structures. For classification headers; Shared features Apply constraints:
[0048] in, Predicted labels for shared features; Shared features By extracting the encoder Extract shared features from the corresponding modalities and private features from the multimodalities. After splicing, the encoder is restored. Reconstruction And to restore it to its original semantic and emotional features as much as possible. :
[0049] in, To reconstruct the features of the corresponding mode, the encoder is recovered. and extract encoder All are composed of linear layers.
[0050] S4. Perform multi-perspective feature aggregation on shared features and multimodal private features, and finally output the sentiment intensity; Shared features Text private features Audio private features and video private features Combine:
[0051]
[0052] The combined data is processed through the same Transformer layer to obtain viewpoint information. :
[0053] in, Emotional intensity is calculated using an average pooling layer.
[0054]
[0055] in, Intensity of emotion; Optimize the network using a loss function:
[0056]
[0057]
[0058]
[0059]
[0060] in, For multimodal loss function, For sentiment loss function with shared features, For multimodal real labels, For orthogonal loss, For private features of a single modality, To rebuild the losses, This is the total loss function; Set up several training rounds. When all training rounds are completed, stop training the model and output the final sentiment intensity. .
[0061] Example 2 In this embodiment, based on Embodiment 1, the fine-tuning adopts an early-stop strategy; after the fine-tuning is completed, in the encoder features Extracting audio coding features Text encoding features and video coding features ;in, The number of layers in the audio encoder. Dimensions of audio coding features. The number of layers in the text encoder. For the dimensions of text encoding features, This refers to the number of layers in the video encoder. Dimensions of video coding features; The multimodal private features Including text-private features Audio private features and video private features .
[0062] Example 3 like Figure 3 As shown in the figure, this embodiment also provides a multimodal sentiment data analysis system, which executes the multimodal sentiment data analysis method described in Embodiment 1; the system includes a feature enhancement module, a multimodal information bottleneck module, a multimodal decoupling module, and a multi-view feature aggregation module; the feature enhancement module receives multimodal sentiment data and outputs fused enhanced features; the multimodal information bottleneck module receives fused enhanced features and outputs semantic sentiment features; the multimodal decoupling module receives semantic sentiment features and outputs shared features and multimodal private features; the multi-view feature aggregation module receives shared features and multimodal private features and finally outputs sentiment intensity.
[0063] Example 4 This embodiment uses PyTorch 2.1.1 on an NVIDIA-4090d GPU. As shown in Tables 1 and 2, Acc-2 represents the accuracy for binary classification, F1 is the harmonic mean of binary classification precision and recall, Acc-3 represents the accuracy for 3-class classification, Acc-5 represents the accuracy for 5-class classification, Acc-7 represents the accuracy for 7-class classification, Corr is the correlation coefficient, and the higher the value of the above indicators, the better the model. MAE is the mean absolute error, and the lower the value of MAE, the better the model.
[0064] Table 1 Performance comparison of this application and baseline models in the CMU-MOSI dataset.
[0065] Table 2 Performance comparison of this application and baseline model in the CH-SIMS dataset.
[0066] Table 3 Performance comparison of this application and baseline model in the CMU-MOSEI dataset.
[0067] This application, based on a pre-trained model, fine-tunes the pre-trained model so that different layers can directly perceive different emotional cues. By combining layers of varying depths, the model can learn richer semantic information. Simultaneously, an information bottleneck is added for noise reduction and compression to retain emotion-related semantic information and remove redundant noise. The predicted sentiment value is then obtained through a decoupling module and a multi-view feature aggregation module. The performance of this application's model is improved compared to existing models. This performance improvement stems from the multi-layer aggregation of emotional cues and the noise reduction, enabling it to capture richer semantic and emotional cues. Baseline models, on the other hand, rely on a single feature extraction method and lack noise reduction, potentially leading to suboptimal feature fusion results that negatively impact performance. Furthermore, this application's model, through decoupling and final multi-view feature aggregation, considers different feature combinations from multiple perspectives, resulting in a more robust performance improvement.
[0068] This application has the following advantages or beneficial effects: 1. By adding pre-trained model layers of varying depths and adaptively fusing them through a feature enhancement module, the multimodal feature representation is enriched, enabling subsequent fusion to obtain richer semantic information and further improving the accuracy of emotion recognition. 2. Before decoupling, the features were enhanced, and the bottleneck of multimodal information was used to reduce noise and compress sentiment information, so that the input decoupling information contains task-related information with less noise, which is more conducive to multimodal decoupling output.
[0069] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.
[0070] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can be physically included separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.
[0071] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute some steps of the transmission and reception methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0072] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. A method for multimodal sentiment data analysis, characterized in that, It includes the following steps: S1. Extract features at different levels from multimodal sentiment data, perform feature fusion and feature enhancement on features at different levels, and output fused and enhanced features; S2. Denoise and compress the fused and enhanced features to output semantic sentiment features; S3. Semantic sentiment features output shared features and multimodal private features through multimodal decoupling; S4. Perform multi-perspective feature aggregation on shared features and multimodal private features, and finally output the sentiment intensity.
2. The multimodal sentiment data analysis method according to claim 1, characterized in that, S1 further includes the following steps: Multimodal sentiment data is input into a pre-trained, fine-tuned encoder to obtain encoder features. ,in ; for encoder features Make minor adjustments to the last layer: in, The loss function of a single-mode encoder. The predicted value output by the single-mode encoder. This is a label representing genuine emotions.
3. The multimodal sentiment data analysis method according to claim 2, characterized in that, The fine-tuning employs an early-stop strategy; after the fine-tuning is completed, the encoder features... Extracting audio coding features Text encoding features and video coding features ;in, The number of layers in the audio encoder. Dimensions of audio coding features. The number of layers in the text encoder. For the dimensions of text encoding features, This refers to the number of layers in the video encoder. Dimensions of video coding features.
4. The multimodal sentiment data analysis method according to claim 3, characterized in that, S1 further includes the following steps: From encoder features Selecting initial semantic features And enhance integration, among which , The length of the modal sequence. For modal feature dimensions; Initial semantic features After being transmitted to the weighted linear layer, the features are transformed by projection. The summation of the dot products yields the adaptive weighted summation features. Initial semantic features The data is transmitted to a linear layer along the feature dimension to obtain the projection transformation features. Projection transformation characteristics With classification head After being concatenated, the data is transmitted to the Transformer layer to obtain the interaction features between the shallow and deep layers. After concatenating the deep and shallow layer interaction features and the adaptive weighted summation features, the fused enhanced features are output through an enhanced feature linear layer. .
5. The multimodal sentiment data analysis method according to claim 4, characterized in that, S2 also includes the following steps: By using the MIB layer, i.e., the multimodal information bottleneck module, to enhance fused features. Optimize noise reduction: in, For multimodal real labels, This is the hidden state of a single-modal system. For single-modal task loss coefficients, For mutual information calculation, Compressive strength; The MIB layer retains the output target information and compresses and fuses enhanced features. semantic sentiment features were obtained. .
6. The multimodal sentiment data analysis method according to claim 5, characterized in that, S3 also includes the following steps: Semantic sentiment features Decoupled into shared features by the following formula and multimodal private features : in, and All are linear layers. Shared spatial mapping features, including shared spatial mapping text features. Shared spatial mapping audio features and shared spatial mapping video features , This involves concatenating the classification header with three single-modal structures. This is the classification header.
7. The multimodal sentiment data analysis method according to claim 6, characterized in that, S3 also includes the following steps: Shared features Apply constraints: in, Predicted labels for shared features; Shared features By extracting the encoder Extract shared features from the corresponding modalities and private features from the multimodalities. After splicing, the encoder is restored. Reconstruction And to restore it to its original semantic and emotional features as much as possible. : in, To reconstruct the features of the corresponding mode, the encoder is recovered. and extract encoder All are composed of linear layers.
8. The multimodal sentiment data analysis method according to claim 7, characterized in that, The multimodal private features Including text-private features Audio private features and video private features ; S4 also includes the following steps: Shared features Text private features Audio private features and video private features Combine: The combined data is processed through the same Transformer layer to obtain viewpoint information. : in, Emotional intensity is calculated using an average pooling layer. in, Emotional intensity.
9. The multimodal sentiment data analysis method according to claim 8, characterized in that, S4 also includes the following steps: Optimize the network using a loss function: in, For multimodal loss function, For sentiment loss function with shared features, For multimodal real labels, For orthogonal loss, For private features of a single modality, To rebuild the losses, This is the total loss function; Set up several training rounds. When all training rounds are completed, stop training the model and output the final sentiment intensity. .
10. A multimodal sentiment data analysis system, characterized in that, The system executes the multimodal sentiment data analysis method according to any one of claims 1-9; the system includes a feature enhancement module, a multimodal information bottleneck module, a multimodal decoupling module, and a multi-view feature aggregation module; The feature enhancement module receives multimodal sentiment data and outputs fused enhanced features; the multimodal information bottleneck module receives fused enhanced features and outputs semantic sentiment features; the multimodal decoupling module receives semantic sentiment features and outputs shared features and multimodal private features. The multi-view feature aggregation module receives shared features and multimodal private features, and finally outputs the sentiment intensity.