A multi-modal sentiment analysis method, a training method of a multi-modal sentiment analysis model, and a computer program product
By aligning visual and text feature sequences through spatial and temporal attention mechanisms and utilizing dynamic temporal weighting, the modality inconsistency problem in multimodal sentiment analysis is solved, thereby improving the accuracy of sentiment analysis.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA MOBILE INT LTD
- Filing Date
- 2026-03-26
- Publication Date
- 2026-06-05
AI Technical Summary
In existing multimodal sentiment analysis methods, the temporal and spatial inconsistencies among audio, visual, and text modalities lead to inaccurate sentiment analysis results.
We employ spatial attention and temporal attention mechanisms to align visual and text feature sequences, and combine dynamic temporal weights to weight the visual and text feature sequences. We then determine the sentiment category through spatiotemporally aligned multimodal features.
It achieves precise semantic alignment in multimodal sentiment analysis, improving the accuracy of sentiment analysis.
Smart Images
Figure CN122153798A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to convolutional neural network models, and more particularly to a multimodal sentiment analysis method, a training method for a multimodal sentiment analysis model, and a computer program product. Background Technology
[0002] Multimodal sentiment analysis technology integrates data from different modalities such as text, audio, images, and videos to comprehensively analyze human emotional states. It is currently widely used in fields such as intelligent human-computer interaction, mental health assessment, and personalized recommendation systems. One of the mainstream multimodal sentiment analysis methods is a cross-modal attention mechanism based on mid-term fusion. However, due to significant temporal and spatial inconsistencies among the audio, visual, and text modalities in multimodal data, precise semantic alignment is difficult to achieve, leading to inaccurate sentiment analysis results. Summary of the Invention
[0003] The technical problem to be solved by this invention is how to achieve accurate semantic alignment of multimodal data and improve the accuracy of multimodal sentiment analysis.
[0004] To address the aforementioned technical problems, this invention provides a multimodal sentiment analysis method, comprising the following steps: S1. Receive a video stream and decode it to obtain the audio feature sequence a t Visual feature sequence v t and text feature sequence t t ; S2. Using spatial attention mechanisms to make the visual feature sequence v t Align audio feature sequences a in spatial dimensions t Obtain the aligned visual feature sequence ; S3. Use a time attention mechanism to make the text feature sequence t t Align audio feature sequence a in the time dimension t Obtain the aligned text feature sequence ; S4. Visual feature sequences and text feature sequences The spatiotemporally aligned multimodal features are obtained by weighting them according to the following formula. :
[0005] in, It is based on the audio feature sequence a t The resulting dynamic time weights; S5. Based on spatiotemporal alignment of multimodal features Determine the emotion category corresponding to the video stream.
[0006] Furthermore, step S2 specifically: S21. Using convolutional networks For audio feature sequence a t Convolution to generate spatial offset fields The formula is as follows:
[0007] in, These are learnable parameters; S22. Using the above spatial offset field The visual feature sequence v is processed according to the following formula. t Perform deformable convolution operations to obtain an audio feature sequence a aligned in the spatial dimension. t visual features :
[0008] in, These are learnable parameters.
[0009] Furthermore, step S3 specifically includes: S31. From audio feature sequence a t Extracting low-level prosodic feature sequences ; S32. Based on the audio feature sequence a t Calculate the dynamic time weights The calculation formula is as follows:
[0010] in, For the Sigmoid function, Audio feature sequence a t Harmony and prosodic feature sequence The components are concatenated, where W and b are learnable parameters. S33. Use a lightweight fully connected network based on dynamic time weights. Calculate the text feature sequence t t Relative to audio feature sequence a t time offset The calculation formula is as follows:
[0011] in, For lightweight, fully connected networks, For learnable parameters, This is the time offset; S34. Based on time offset Let the text feature sequence t t Align audio feature sequence a in the time dimension t Obtain the aligned text feature sequence .
[0012] Furthermore, step S5 specifically: S51. Using state-space models for multimodal features The mask reconstruction and future prediction tasks are performed in parallel to construct multimodal causal relationships, thereby outputting an audio feature sequence containing these multimodal causal relationships. Visual feature sequences and text feature sequences ; S52. A gating attention mechanism is used to process audio feature sequences containing multimodal causal relationships. Visual feature sequences and text feature sequences Perform deep fusion to obtain fusion features as follows:
[0013] Among them, g t For adaptive learning, the dynamic gating coefficients are used. S53. Using a classifier to fuse features The system performs reasoning and classification to determine the emotion category corresponding to the video stream and outputs it to the user.
[0014] Furthermore, in step S53, the classifier is specifically a fully connected layer classifier.
[0015] This invention also provides a training method for a multimodal sentiment analysis model, comprising the following steps: A set of training samples is formed by taking a video stream as input and the sentiment category of the video stream as output. The multimodal sentiment analysis model is trained using multiple sets of such training samples until it performs the multimodal sentiment analysis method described above to determine the sentiment category of the input video stream.
[0016] The present invention also provides a computer program product, including a computer program that, when executed, can implement the multimodal sentiment analysis method as described above.
[0017] The multimodal sentiment analysis method presented in this invention uses spatial attention and temporal attention mechanisms to make the visual feature sequence v t Text feature sequence t t Align audio feature sequence a t The aligned visual feature sequence is obtained. Text feature sequences Then S4 is based on dynamic time weights visual feature sequences and text feature sequences Weighted multimodal features are obtained by spatiotemporal alignment. This means dynamically adjusting the contribution weight of each modality in the time dimension based on audio prosody, ensuring that the model gives higher attention to the corresponding modality at moments of high emotional intensity, thus achieving accurate semantic alignment of multimodalities. S5 utilizes spatiotemporal alignment multimodal features. The identified emotion categories have a high accuracy rate. Attached Figure Description
[0018] Figure 1 This is a flowchart illustrating the multimodal sentiment analysis method presented in this invention. Detailed Implementation
[0019] The present invention will be further described in detail below with reference to specific embodiments.
[0020] This embodiment describes a computer program product obtained by technicians using computer program code, which is referred to as a multimodal sentiment analysis model. The multimodal sentiment analysis model includes, in sequence, a multimodal data preprocessing and feature extraction module, a spatiotemporal attention module (including a spatial dynamic attention unit (SDA), a temporal dynamic attention unit (TDA), and a spatiotemporally aligned multimodal feature output unit), an autoregressive state causality modeling module (ASM), a feature fusion module, and a sentiment classification module.
[0021] This embodiment uses a video stream as input and the sentiment category of the video stream as output to form a set of training samples. Multiple sets of such training samples are used to train the multimodal sentiment analysis model until the model performs as described above. Figure 1 The multimodal sentiment analysis method shown determines the sentiment category of the input video stream. The multimodal sentiment analysis model includes a computer program installed in a computer; the computer processor executes the model to achieve... Figure 1 The multimodal sentiment analysis method is shown below. The steps of this method are described in detail below.
[0022] S1. Receive a video stream and decode it to obtain the audio feature sequence a t Visual feature sequence v t and text feature sequence t t .
[0023] The multimodal data preprocessing and feature extraction module of the multimodal sentiment analysis model receives Urdu video streams and then decodes them. Specifically: (1) Extract the Mel spectrum of the video stream and input the extracted Mel spectrum into the VGGish network to obtain the audio feature sequence A.
[0024] (2) Decode the video stream into image frames, perform face detection and cropping on the image frames, and input the cropped face image frames into the ResNet network to obtain the visual feature sequence V.
[0025] (3) Extract subtitles from the video stream to obtain text or perform speech recognition on the video stream to obtain text. BPE segments the text and inputs the segmented text into the BERT-Urdu model to obtain the text feature sequence TT.
[0026] S2. Using spatial attention mechanisms to make the visual feature sequence v t Align audio feature sequences a in spatial dimensions t Obtain the aligned visual feature sequence .
[0027] The spatiotemporal attention module of the multimodal sentiment analysis model uses a spatial attention mechanism through the Spatial Dynamic Attention Unit (SDA) to focus the visual feature sequence v t Align audio feature sequences a in spatial dimensions t Obtain the aligned visual feature sequence The specific process is as follows: S21. Using convolutional networks For audio feature sequence a t Convolution to generate spatial offset fields The formula is as follows:
[0028] in, These are learnable parameters; S22. Using the above spatial offset field The visual feature sequence v is processed according to the following formula. t Performing deformable convolution operations (which enable the model to go beyond a fixed face mesh and focus on subtle muscle movements that resonate with the emotions in the voice) yields a spatially aligned audio feature sequence a. t visual features :
[0029] in, These are learnable parameters.
[0030] The Spatial Dynamic Attention Unit is designed to enable the model to adaptively focus on more emotionally expressive facial regions (such as the corners of the mouth and eyebrows) in the visual modality, based on the audio content.
[0031] S3. Use a time attention mechanism to make the text feature sequence t t Align audio feature sequence a in the time dimension t Obtain the aligned text feature sequence .
[0032] The spatiotemporal attention module of the multimodal sentiment analysis model uses a temporal attention mechanism through a temporal dynamic attention unit (TDA) to focus the text feature sequence t. t Align audio feature sequence a in the time dimension t Obtain the aligned text feature sequence The specific process is as follows: S31. From audio feature sequence a t Extracting low-level prosodic feature sequences This includes prosodic features such as fundamental frequency (F0) and energy. S32. Based on the audio feature sequence a t The dynamic time weights are calculated using a fully connected layer and a Sigmoid activation function. The calculation formula is as follows:
[0033] in, For the Sigmoid function, Audio feature sequence a t Harmony and prosodic feature sequence The components are concatenated, where W and b are learnable parameters. S33. Use a lightweight fully connected network based on dynamic time weights. Calculate the text feature sequence t t Relative to audio feature sequence a t time offset The calculation formula is as follows:
[0034] in, For lightweight, fully connected networks, For learnable parameters, This is the time offset; S34. Based on time offset Let the text feature sequence t t Align audio feature sequence a in the time dimension t Obtain the aligned text feature sequence .For example, =0.5 indicates that the text features need to be calibrated with a 0.5-second delay.
[0035] S4. Visual feature sequences and text feature sequences The spatiotemporally aligned multimodal features are obtained by weighting them according to the following formula. :
[0036] in, It is based on the audio feature sequence a t The resulting dynamic time weights.
[0037] The spatiotemporal attention module of the multimodal sentiment analysis model outputs spatiotemporally aligned multimodal features through a spatiotemporally aligned multimodal feature output unit. The specific process is as follows: The spatiotemporally aligned multimodal feature output unit calculated above utilizes... Aligned visual feature sequences and text feature sequences Weighted summaries are applied to output spatiotemporally aligned multimodal features. Utilizing dynamic time weights This ensures that the model gives higher attention to the corresponding modality during moments of high emotional intensity.
[0038]
[0039] in, It is based on the audio feature sequence a t The resulting dynamic time weights.
[0040] It should be noted that, as mentioned above, the video feature sequence v t and text feature sequence t t Based on audio feature sequence a t The alignment is guided by the alignment of visual feature sequences. and text feature sequences Contains audio feature sequence a t Information, therefore multimodal features It also includes audio feature sequences. 、 Visual feature sequence and text feature sequences .
[0041] S5. Based on spatiotemporal alignment of multimodal features The specific process for determining the sentiment category corresponding to the video stream is as follows: S51. Using state-space models for multimodal features The mask reconstruction and future prediction tasks are performed in parallel to construct multimodal causal relationships, thereby outputting an audio feature sequence containing these multimodal causal relationships. Visual feature sequences and text feature sequences .
[0042] The Autoregressive State Causality Modeling (ASM) module of the multimodal sentiment analysis model employs a state-space model (SSM). This model incorporates a mask reconstruction unit and a future prediction unit, achieving multimodal causal relationship modeling through the dual tasks of mask reconstruction and future prediction. The spatiotemporal attention module outputs multimodal features. The Autoregressive State Causality Modeling Module (ASM) incorporates multimodal features. In the input state-space model, the state-space model generates hidden states. .
[0043] (1) Mask reconstruction task Random mask: for the input multimodal feature sequence { ,..., Randomly mask 15%-40% of feature units.
[0044] Context Reconstruction: Reconstructing masked features using unmasked contextual information through a Transformer-based decoder.
[0045] in These are the hidden states generated by the State-Space Model (SSM). This task forces the model to learn complementary contextual information between modes, effectively mitigating overfitting.
[0046] (2) Future prediction task Sequence prediction: based on historical hidden states Predicting features for the next K time steps ,,..., For sequence prediction, the prediction loss is:
[0047] Mask reconstruction mines instantaneous causal relationships between modes in the same spatiotemporal space through cross-modal complementary dependencies to form static causal relationships. Future prediction captures modal causal transmission across time steps through temporal evolution dependencies to construct dynamic causal chains. Together, they encode intermodal causal relationships in ASM hidden states and gated scalars, realizing the quantitative storage and interpretable output of causal information. The final output of the state-space model contains an audio feature sequence containing multimodal causal relationships. Visual feature sequences and text feature sequences For example, in an Urdu complaining video, the stress feature of the audio at t = 2 seconds → the negative keyword that drives the text "The express delivery is too slow" appears synchronously with the visual action of the downturned corners of the mouth, and the three form an immediate causal association of "audio stress → text semantics → visual expression".
[0048] S52. Adopt a gated attention mechanism for the audio feature sequence containing multimodal causal relationships , visual feature sequence and text feature sequence to perform deep fusion and obtain the fused feature as follows:
[0049] where g t is the dynamically gated coefficient of adaptive learning.
[0050] The feature fusion module of the multimodal sentiment analysis model adopts a gated attention mechanism to perform deep fusion on the multimodal causal enhanced features, and adaptively allocates the contributions of each modality through a dynamic gating scalar. The fusion formula is as follows:
[0051] where is the fused feature, g t is the dynamically gated coefficient of adaptive learning, is the visual feature sequence and text feature sequence are concatenated.
[0052] S53. Use a classifier to perform inference classification on the fused feature to obtain the sentiment category corresponding to the video stream and output it to the user.
[0053] The sentiment classification module of the multimodal sentiment analysis model adopts a fully connected layer classifier to perform inference classification on the fused feature to obtain the probability distribution of the sentiment category of the video stream, such as the probability distribution of the three sentiment categories of positive, neutral, and negative, and take the one with the largest probability as the sentiment category of the video stream and output it to the user.
[0054] The multimodal sentiment analysis method given by the present invention respectively uses a spatial attention mechanism and a temporal attention mechanism to align the visual feature sequence v t , text feature sequence t t with the audio feature sequence a t to obtain the aligned visual feature sequence , text feature sequence , and then S4 uses the dynamic time weight for the visual feature sequence and text feature sequences Weighted multimodal features are obtained by spatiotemporal alignment. This means dynamically adjusting the contribution weight of each modality in the time dimension based on audio prosody, ensuring that the model gives higher attention to the corresponding modality at moments of high emotional intensity, thus achieving accurate semantic alignment of multimodalities. S5 utilizes spatiotemporal alignment multimodal features. The identified emotion categories have a high accuracy rate.
[0055] The above description is merely an embodiment of the present invention and does not limit the scope of patent protection. Any non-substantial changes or substitutions made by those skilled in the art based on the present invention will still fall within the scope of patent protection.
Claims
1. A multimodal sentiment analysis method, characterized in that, Includes the following steps: S1. Receive a video stream and decode it to obtain the audio feature sequence a t Visual feature sequence v t and text feature sequence t t ; S2. Using spatial attention mechanisms to make the visual feature sequence v t Align audio feature sequences a in spatial dimensions t Obtain the aligned visual feature sequence ; S3. Use a time attention mechanism to make the text feature sequence t t Align audio feature sequences a in the time dimension t Obtain the aligned text feature sequence ; S4. Visual feature sequences and text feature sequences The spatiotemporally aligned multimodal features are obtained by weighting them according to the following formula. : in, It is based on the audio feature sequence a t The resulting dynamic time weights; S5. Based on spatiotemporal alignment of multimodal features Determine the emotion category corresponding to the video stream.
2. The multimodal sentiment analysis method as described in claim 1, characterized in that, Step S2 specifically: S21. Using convolutional networks For audio feature sequence a t Convolution to generate spatial offset fields The formula is as follows: in, These are learnable parameters; S22. Using the above spatial offset field The visual feature sequence v is processed according to the following formula. t Perform deformable convolution operations to obtain an audio feature sequence a aligned in the spatial dimension. t visual features : in, These are learnable parameters.
3. The multimodal sentiment analysis method as described in claim 1, characterized in that, Step S3 specifically: S31. From audio feature sequence a t Extracting low-level prosodic feature sequences ; S32. Based on the audio feature sequence a t Calculate the dynamic time weights The calculation formula is as follows: in, For the Sigmoid function, Audio feature sequence a t Harmony and prosodic feature sequence The components are concatenated, where W and b are learnable parameters. S33. Use a lightweight fully connected network based on dynamic time weights. Calculate the text feature sequence t t Relative to audio feature sequence a t time offset The calculation formula is as follows: in, For lightweight, fully connected networks, For learnable parameters, This is the time offset; S34. Based on time offset Let the text feature sequence t t Align audio feature sequences a in the time dimension t Obtain the aligned text feature sequence .
4. The multimodal sentiment analysis method as described in claim 1, characterized in that, Step S5 specifically: S51. Using state-space models for multimodal features The mask reconstruction and future prediction tasks are performed in parallel to construct multimodal causal relationships, thereby outputting an audio feature sequence containing these multimodal causal relationships. Visual feature sequences and text feature sequences ; S52. A gating attention mechanism is used to process audio feature sequences containing multimodal causal relationships. Visual feature sequences and text feature sequences Perform deep fusion to obtain fusion features as follows: Among them, g t For adaptive learning, the dynamic gating coefficients are used. S53. Using a classifier to fuse features The system performs reasoning and classification to determine the emotion category corresponding to the video stream and outputs it to the user.
5. The multimodal sentiment analysis method as described in claim 4, characterized in that, In step S53, the classifier is specifically a fully connected layer classifier.
6. A training method for a multimodal sentiment analysis model, characterized in that, Includes the following steps: A training sample is formed by taking a video stream as input and the sentiment category of the video stream as output. The multimodal sentiment analysis model is trained using multiple sets of such training samples until it performs the multimodal sentiment analysis method as described in any one of claims 1 to 5 to determine the sentiment category of the input video stream.
7. A computer program product comprising a computer program, characterized in that, When executed, the computer program is capable of implementing the multimodal sentiment analysis method as described in any one of claims 1 to 5.