Multi-modal sentiment analysis method based on task association perception learning

Abstract

The invention relates to the field of multi-modal sentiment analysis, in particular to a multi-modal sentiment analysis method based on task association perception learning. According to the scheme, the method comprises the steps that multi-modal data including text, audio and visual modal data are obtained, feature extraction is conducted on the multi-modal data, a double-branch comparison module is constructed for each kind of modal data, each double-branch comparison module comprises a fusion branch and a comparison branch, and the fusion branch is connected with the comparison branch; input of a fusion branch and a comparison branch of each double-branch comparison module is an extracted text modal feature, an extracted audio modal feature and an extracted visual modal feature; and finally, feature fusion and prediction output are carried out, so that the accuracy and robustness of multi-modal sentiment analysis are improved. The method is suitable for multi-modal sentiment analysis.

Description

Technical Field

[0001] This invention relates to the field of multimodal sentiment analysis, specifically to a multimodal sentiment analysis method based on task-related perceptual learning. Background Technology

[0002] Traditional multimodal sentiment analysis methods typically employ uniform sentiment labeling, which involves applying a single sentiment label to the entire multimodal dataset. However, this uniform labeling approach fails to reflect the differences in sentiment expression between different modalities, limiting the model's ability to effectively mine and integrate modality-specific sentiment information and further improve its fusion performance.

[0003] To address this issue, researchers both domestically and internationally have recently proposed multimodal sentiment analysis based on modality sub-labels, which involves supervising the learning of modal features separately through refined unimodal sentiment annotation. This approach aims to better capture modality-specific emotional expressions and improve the performance and interpretability of multimodal sentiment analysis. For this task, existing technical solutions typically employ a multimodal multi-task learning framework. This framework guides unimodal learning through modality sub-labels and jointly optimizes unimodal and multimodal sentiment analysis tasks, enhancing the model's understanding and utilization of each modality's features.

[0004] However, this direct joint training approach has certain limitations when dealing with complex emotional expressions. Specifically, because different modalities may independently express signals different from the overall emotion, there may be inconsistencies between unimodal and multimodal emotion labels, thus introducing task conflicts between unimodal and multimodal tasks. For example, in real-world contexts, text may convey positive signals, but when combined with audio and visual information, the overall emotion may appear negative. This conflict causes the model to receive inconsistent supervision signals on the same sample, interfering with the learning process of unimodal representations and further affecting the robustness and accuracy of the final multimodal emotion prediction. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a multimodal sentiment analysis method based on task-related perception learning, which improves the accuracy and robustness of multimodal sentiment analysis.

[0006] This invention achieves the above objectives by adopting the following technical solution: This invention provides a multimodal sentiment analysis method based on task-related perceptual learning, comprising:

[0007] S1. Acquire multimodal data, including text, audio, and visual modal data;

[0008] S2. Feature extraction from multimodal data;

[0009] S3. Construct a dual-branch comparison module;

[0010] For each type of modality data, a dual-branch comparison module is constructed. Each dual-branch comparison module contains a fusion branch and a comparison branch. The inputs of the fusion branch and the comparison branch of each dual-branch comparison module are the extracted text modality features, audio modality features, and visual modality features, respectively.

[0011] Each dual-branch comparison module outputs a fusion branch and a comparison branch single-modal representation through their respective encoders. JS divergence is used to measure the similarity between the output probability distributions of the fusion branch and the comparison branch. For each training batch sample set, the samples are sorted according to the JS divergence score, and the top few samples with the smallest JS divergence are selected to form a task collaboration subset, while the remaining samples form a task conflict subset.

[0012] S4. Execute the correlation adaptive training strategy;

[0013] S401, Apply single-modal supervision;

[0014] Single-modal supervision is applied to the contrastive branch, and its contrastive single-modal representation is processed by a multilayer perceptron to obtain the prediction result. Optimization is achieved through mean squared error loss:

[0015] ;

[0016] in, This represents the total number of training samples. This indicates the corresponding single-modal sentiment tag. This represents the single-mode mean square error loss;

[0017] S402, Design of Differentiated Loss Function;

[0018] Task correction loss:

[0019] For samples in the task conflict subset, the forced fusion branch reduces the dependence on single-modal labels and instead learns multimodal targets to prevent conflict signals from interfering with the model.

[0020] ;

[0021] in, This indicates the prediction results of the fusion branch. Represents multimodal sentiment tags. Indicates task correction loss, Represents a subset of task collaborations. Indicates a subset of task conflicts;

[0022] Comparative loss:

[0023] For task collaboration subsets, the ability of the model to extract complementary features is enhanced by constraining the distribution of the two branches to tend to be consistent.

[0024] ;

[0025] in, This represents the JS divergence value corresponding to the sample. Indicates comparative loss;

[0026] S403. Joint optimization is performed based on single-modal mean square error loss, task correction loss, and contrast loss.

[0027] The overall optimization objective of the association-adaptive training strategy based on the dual-branch comparison module is uniformly expressed as:

[0028] ;

[0029] in, This represents the loss of the two-branch comparison module, where Indicates the modal type, , Represents text modality, Indicates audio modality, Represents visual modality;

[0030] The bi-branch comparison module for each modality is passed through Joint optimization is used to obtain a single-modal representation of task-related perception from the fusion branch;

[0031] S5. Finally, feature fusion and prediction output are performed.

[0032] Furthermore, step S2 specifically includes:

[0033] For text modal data, word vector features are extracted using a pre-trained BERT model. The specific process is as follows:

[0034] Using the bert-base-chinese version, the pre-trained BERT model has a 12-layer Transformer encoder, with each layer having an output dimension of 768. For each input text, the BERT model's built-in Chinese word segmenter is used for encoding, and special markers are added before the sequence to represent sentence-level semantics. Then, the token sequence length is uniformly truncated or padded to 50 markers. Tokens longer than 50 are truncated, and those shorter than 50 are padded with zeros to obtain a fixed-length context embedding vector sequence. Finally, the output of each sample text modality is a semantic feature sequence with a length of 50 and a dimension of 768 at each time step.

[0035] Furthermore, step S2 specifically includes:

[0036] For audio modal data, acoustic features are extracted using OpenSMILE. The specific process is as follows:

[0037] The OpenSMILE tool was used to extract acoustic features, and its eGeMAPS was selected to extract low-level acoustic descriptor features. The audio data was uniformly sampled at 16kHz. The extracted features for each frame were 25-dimensional vectors. The audio feature sequence was padded or truncated to 925 frames. If there were fewer than 925 frames, zeros were added to the end. If there were more than 925 frames, the extra frames were truncated. Finally, each audio segment was represented as a 925×25 matrix.

[0038] Furthermore, step S2 specifically includes:

[0039] For visual modality data, FFmpeg and OpenFace are used to extract visual features. The specific process is as follows:

[0040] First, the multimedia processing tool FFmpeg is used to extract images at 25 frames per second. After detecting the presence of a speaker's face in the image using the TalkNet method, OpenFace is used to extract facial features, including 68 facial points, 17 facial action units, head pose, head orientation, and eye gaze direction. Finally, a 177-dimensional visual feature is obtained, and its sequence length is uniformly 232 by padding or truncation.

[0041] Furthermore, step S5 specifically includes:

[0042] The modal fusion features optimized by the correlation adaptive training strategy are concatenated and then passed through multiple linear layers to achieve cross-modal space mapping.

[0043] ;

[0044] in, This represents a multimodal fusion representation. This represents the text modality fusion features optimized by an association-adaptive training strategy. This represents the audio modality fusion features optimized by the correlation adaptive training strategy. This represents the visual modality fusion features optimized by an associative adaptive training strategy;

[0045] Multimodal fusion representation The result is mapped to the final multimodal sentiment prediction value using a multilayer perceptron. ;

[0046] Multimodal sentiment prediction Mean squared error is used as the supervision signal for the regression task;

[0047] ;

[0048] in, This represents the multimodal mean square error loss;

[0049] The final total loss function is obtained based on the multimodal mean square error loss and the loss of the bi-branch comparison module, and joint optimization is performed based on the total loss function.

[0050] The beneficial effects of this invention are as follows:

[0051] This invention uses an associative adaptive training strategy and a dual-branch contrast module to perceive and regulate task relevance. The dual-branch contrast module sets a fusion branch and a contrast branch within each modality. The two branches have the same structure but different supervision signals. The degree of conflict is characterized by the difference measure (using JS divergence) of the output of the two branches, thereby dynamically dividing the training samples into task-cooperative subsets and task-conflicting subsets. In the associative adaptive training strategy, contrast loss is applied to the cooperative subset to enhance the learning of complementary information between modalities, and task correction loss is introduced to the conflict subset to suppress irrelevant or interfering information, thereby improving the accuracy and robustness of multimodal sentiment analysis. Attached Figure Description

[0052] Figure 1 This is a flowchart of a multimodal sentiment analysis method based on task-related perceptual learning provided by the present invention;

[0053] Figure 2 This is a schematic diagram of the multimodal sentiment analysis method framework for task-related perceptual learning provided by the present invention;

[0054] Figure 3 This is a schematic diagram of the correlation adaptive training strategy provided by the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0056] This invention provides a multimodal sentiment analysis method based on task-related perceptual learning, such as... Figures 1 to 3 As shown, it specifically includes:

[0057] S1. Acquire multimodal data, including text, audio, and visual modal data;

[0058] S2. Feature extraction from multimodal data;

[0059] For text modal data, word vector features are extracted using a pre-trained BERT model;

[0060] Using the bert-base-chinese version, the pre-trained BERT model has a 12-layer Transformer encoder, with each layer having an output dimension of 768. For each input text, the BERT model's built-in Chinese word segmenter is used for encoding, and special markers are added before the sequence to represent sentence-level semantics. Then, the token sequence length is uniformly truncated or padded to 50 markers. Tokens longer than 50 are truncated, and those shorter than 50 are padded with zeros to obtain a fixed-length context embedding vector sequence. Finally, the output of each sample text modality is a semantic feature sequence with a length of 50 and a dimension of 768 at each time step.

[0061] For audio modal data, acoustic features are extracted using OpenSMILE;

[0062] The OpenSMILE tool was used to extract acoustic features, and its eGeMAPS was selected to extract low-level acoustic descriptor features. The audio data was uniformly sampled at 16kHz. The extracted features for each frame were 25-dimensional vectors. The audio feature sequence was padded or truncated to 925 frames. If there were fewer than 925 frames, zeros were added to the end. If there were more than 925 frames, the extra frames were truncated. Finally, each audio segment was represented as a 925×25 matrix.

[0063] For visual modality data, FFmpeg and OpenFace are used to extract visual features;

[0064] First, the multimedia processing tool FFmpeg is used to extract images at 25 frames per second. After detecting the presence of a speaker's face in the image using the TalkNet method, OpenFace is used to extract facial features, including 68 facial points, 17 facial action units, head pose, head orientation, and eye gaze direction. Finally, a 177-dimensional visual feature is obtained, and its sequence length is uniformly 232 by padding or truncation.

[0065] S3. Construct a dual-branch comparison module;

[0066] For each modal data, a two-branch comparison module is constructed. Each two-branch comparison module includes a fusion branch and a comparison branch. The inputs to the fusion branch and the comparison branch are... ,in Indicates the modal type, , Represents text modality, Indicates audio modality, Representing visual modality, Represents the sequence length. Represents the feature dimension. That is... for hour, This represents the extracted text modal features; for hour, This represents the extracted audio modal features; for hour, This represents the extracted visual modal features.

[0067] The two branches output the fused branch single-modal representation through their respective encoders. Compared with the single-mode characterization of the contrast branch :

[0068] ;

[0069] ;

[0070] in, and The parameters for the merging branch and the comparison branch;

[0071] JS divergence is used to measure the similarity between the output probability distributions of the fusion branch and the contrast branch:

[0072] ;

[0073] in Let KL divergence be the KL divergence. and They represent respectively by , The obtained probability distribution , The larger the value, the more severe the task conflict of the sample.

[0074] In each size training batch sample set ,according to The samples are sorted by score, and the top (those with the smallest JS divergence) are selected. Each sample constitutes a task collaboration subset. The remaining samples constitute a subset of task conflicts. :

[0075] ;

[0076] Where |⋅| represents the number of samples in the set, and the hyperparameter is... Used to flexibly control the ratio of task collaboration subsets to task conflict subsets;

[0077] S4. Execute the correlation adaptive training strategy;

[0078] S401. Apply single-modal supervision to the comparison branch;

[0079] To ensure that the comparison branch obtains a pure and task-unaffected single-modal representation, this invention continuously applies single-modal supervision to the comparison branch, and its comparison single-modal representation... The prediction result is obtained after processing by a multilayer perceptron. Optimization is achieved through mean squared error loss:

[0080] ;

[0081] in, This represents the total number of training samples. This indicates the corresponding single-modal sentiment tag. This represents the single-mode mean square error loss;

[0082] S402, Design of Differentiated Loss Function;

[0083] Task correction loss:

[0084] For samples in the task conflict subset, the forced fusion branch reduces the dependence on single-modal labels and instead learns multimodal targets to prevent conflict signals from interfering with the model.

[0085] ;

[0086] in, This indicates the prediction results of the fusion branch. Represents multimodal sentiment tags. Indicates task correction loss;

[0087] Comparative loss:

[0088] For task collaboration subsets, the ability of the model to extract complementary features is enhanced by constraining the distribution of the two branches to tend to be consistent.

[0089] ;

[0090] in, This represents the JS divergence value corresponding to the sample. Indicates comparative loss;

[0091] S403. Joint optimization is performed based on mean squared error loss, task correction loss, and contrast loss.

[0092] The overall optimization objective of the association-adaptive training strategy based on the dual-branch comparison module is uniformly expressed as:

[0093] ;

[0094] in, This represents the loss of the two-branch comparison module;

[0095] The bi-branch comparison module for each modality is passed through Joint optimization is used to obtain a single-modal representation of task-related perception from the fusion branch.

[0096] S5. Feature fusion and prediction output;

[0097] The modal fusion features optimized by the correlation adaptive training strategy are concatenated and then passed through multiple linear layers to achieve cross-modal space mapping.

[0098] ;

[0099] in, This represents a multimodal fusion representation. This represents the text modality fusion features optimized by an association-adaptive training strategy. This represents the audio modality fusion features optimized by the correlation adaptive training strategy. This represents the visual modality fusion features optimized by an associative adaptive training strategy;

[0100] Multimodal fusion representation The result is mapped to the final multimodal sentiment prediction value using a multilayer perceptron. ;

[0101] Multimodal sentiment prediction Mean squared error is used as the supervision signal for the regression task;

[0102] ;

[0103] in, This represents the multimodal mean square error loss;

[0104] The final total loss function is obtained based on the multimodal mean square error loss and the loss of the bi-branch comparison module, and joint optimization is performed based on the total loss function.

[0105] ,in This represents the total loss function.

[0106] In summary, existing multimodal multitask learning methods typically incorporate single-modal subtasks directly into joint optimization, assuming that they all have a positive effect on multimodal tasks. This invention constructs a "fusion branch - comparison branch" through a dual-branch comparison module and uses JS divergence to quantify the conflict intensity, so that task conflicts can be detected and measured, thereby avoiding mutual interference between conflict supervision during training.

[0107] Existing methods lack targeted processing for samples with "contradictory single-modal labels and multimodal labels", which can easily lead to misleading single-modal representations and affect fusion prediction. The association adaptive training strategy of this invention dynamically divides beneficial and harmful subsets within a mini-batch and uses task correction supervision for harmful subsets to prevent conflicting samples from updating the fusion branch in the wrong direction.

[0108] This invention not only suppresses the adverse effects of conflicting samples, but also applies contrast constraints to beneficial subsets to enhance the learning of complementary information from collaborative samples, so that single-modal sub-labels "generate gains when available and are suppressed when conflicting", thereby making more efficient use of differentiated labeling information and improving the overall effect of multimodal emotion recognition.

[0109] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A multimodal sentiment analysis method based on task-related perceptual learning, characterized in that, include: S1. Acquire multimodal data, including text, audio, and visual modal data; S2. Feature extraction from multimodal data; S3. Construct a dual-branch comparison module; For each type of modality data, a dual-branch comparison module is constructed. Each dual-branch comparison module contains a fusion branch and a comparison branch. The inputs of the fusion branch and the comparison branch of each dual-branch comparison module are the extracted text modality features, audio modality features, and visual modality features, respectively. Each dual-branch comparison module outputs a single-modal representation of the fusion branch and a single-modal representation of the comparison branch through their respective encoders. JS divergence is used to measure the similarity between the output probability distributions of the fusion branch and the comparison branch. For each training batch sample set, the samples are sorted according to the JS divergence score, and the top few samples with the smallest JS divergence are selected to form a task collaboration subset, while the remaining samples form a task conflict subset. S4. Execute the correlation adaptive training strategy; S401, Apply single-modal supervision; Single-modal supervision is applied to the contrastive branch, and its contrastive single-modal representation is processed by a multilayer perceptron to obtain the prediction result. Optimization is achieved through mean squared error loss: ； in, This represents the total number of training samples. This indicates the corresponding single-modal sentiment tag. This represents the single-mode mean square error loss; S402, Design of Differentiated Loss Function; Task correction loss: For samples in the task conflict subset, the forced fusion branch reduces the dependence on single-modal labels and instead learns multimodal targets to prevent conflict signals from interfering with the model. ； in, This indicates the prediction results of the fusion branch. Represents multimodal sentiment tags. Indicates task correction loss, Represents a subset of task collaborations. Indicates a subset of task conflicts; Comparative loss: For task collaboration subsets, the ability of the model to extract complementary features is enhanced by constraining the distribution of the two branches to tend to be consistent. ； in, This represents the JS divergence value corresponding to the sample. Indicates comparative loss; S403. Joint optimization is performed based on single-modal mean square error loss, task correction loss, and contrast loss. The overall optimization objective of the association-adaptive training strategy based on the dual-branch comparison module is uniformly expressed as: ； in, This represents the loss of the two-branch comparison module, where Indicates the modal type, , Represents text modality, Indicates audio modality, Represents visual modality; The bi-branch comparison module for each modality is passed through Joint optimization is used to obtain a single-modal representation of task-related perception from the fusion branch; S5. Finally, feature fusion and prediction output are performed.

2. The multimodal sentiment analysis method based on task-related perceptual learning according to claim 1, characterized in that, Step S2 specifically includes: For text modal data, word vector features are extracted using a pre-trained BERT model. The specific process is as follows: Using the bert-base-chinese version, the pre-trained BERT model has a 12-layer Transformer encoder, with each layer having an output dimension of 768. For each input text, the BERT model's built-in Chinese word segmenter is used for encoding, and special markers are added before the sequence to represent sentence-level semantics. Then, the token sequence length is uniformly truncated or padded to 50 markers. Tokens longer than 50 are truncated, and those shorter than 50 are padded with zeros to obtain a fixed-length context embedding vector sequence. Finally, the output of each sample text modality is a semantic feature sequence with a length of 50 and a dimension of 768 at each time step.

3. The multimodal sentiment analysis method based on task-related perceptual learning according to claim 1, characterized in that, Step S2 specifically includes: For audio modal data, acoustic features are extracted using OpenSMILE. The specific process is as follows: The OpenSMILE tool was used to extract acoustic features, and its eGeMAPS was selected to extract low-level acoustic descriptor features. The audio data was uniformly sampled at 16kHz. The extracted features for each frame were 25-dimensional vectors. The audio feature sequence was padded or truncated to 925 frames. If there were fewer than 925 frames, zeros were added to the end. If there were more than 925 frames, the extra frames were truncated. Finally, each audio segment was represented as a 925×25 matrix.

4. The multimodal sentiment analysis method based on task-related perceptual learning according to claim 1, characterized in that, Step S2 specifically includes: For visual modality data, FFmpeg and OpenFace are used to extract visual features. The specific process is as follows: First, the multimedia processing tool FFmpeg is used to extract images at 25 frames per second. After detecting the presence of a speaker's face in the image using the TalkNet method, OpenFace is used to extract facial features, including 68 facial points, 17 facial action units, head pose, head orientation, and eye gaze direction. Finally, a 177-dimensional visual feature is obtained, and its sequence length is uniformly 232 by padding or truncation.

5. The multimodal sentiment analysis method based on task-related perceptual learning according to claim 1, characterized in that, Step S5 specifically includes: The modal fusion features optimized by the correlation adaptive training strategy are concatenated and then passed through multiple linear layers to achieve cross-modal space mapping. ； in, This represents a multimodal fusion representation. This represents the text modality fusion features optimized by an association-adaptive training strategy. This represents the audio modality fusion features optimized by the correlation adaptive training strategy. This represents the visual modality fusion features optimized by an associative adaptive training strategy; Multimodal fusion representation The result is mapped to the final multimodal sentiment prediction value using a multilayer perceptron. ; Multimodal sentiment prediction value Mean squared error is used as the supervision signal for the regression task; ； in, This represents the multimodal mean square error loss; The final total loss function is obtained based on the multimodal mean square error loss and the loss of the bi-branch comparison module, and joint optimization is performed based on the total loss function.

Images