Multi-modal sentiment analysis method and system based on principal modal two-stage guidance

Abstract

The application provides a multi-modal sentiment analysis method and system based on a main mode two-stage guide, and relates to the technical field of sentiment analysis. Multi-modal data is input into a multi-modal sentiment analysis model, wherein language, visual and acoustic features are extracted from the multi-modal data by a feature extraction module; multi-modal feature semantics are decoupled into mode-invariant features and mode-unique features by a feature space distribution alignment module, and a language mode is used as a guide to realize feature distribution alignment by alignment reconstruction constraints; self-attention modeling is performed on the mode-invariant features by an attention enhancement module to obtain first enhanced features, and the language-unique features are used as a guide to adaptively enhance the visual and acoustic unique features by a cross-modal attention mechanism to obtain second enhanced features; and the first enhanced features and the second enhanced features are fused by a sentiment prediction module, and a sentiment intensity prediction result is obtained by regression prediction, so that the accuracy and robustness of sentiment analysis in a complex scene are improved.

Description

Technical Field

[0001] This invention relates to the field of sentiment analysis technology, and in particular to a multimodal sentiment analysis method and system based on a two-stage guidance of the dominant modality. Background Technology

[0002] With the rapid proliferation of smart mobile devices, multimodal data is experiencing explosive growth in multimedia applications such as social media and e-commerce. To effectively integrate heterogeneous data from different modalities (such as language, acoustics, and vision) and achieve a more comprehensive and accurate identification and understanding of human emotional states, multimodal sentiment analysis is attracting increasing attention from researchers. In multimodal sentiment analysis tasks, different modalities from the same video segment encode complementary emotional cues. One research challenge lies in effectively capturing the underlying emotional signals in these modalities and integrating their representations to accurately infer the underlying sentiment of the original data.

[0003] Despite recent advancements in multimodal sentiment analysis, the heterogeneity between different modalities hinders the development of robust and unified sentiment representations. To address the challenges posed by modal heterogeneity, many approaches attempt to leverage representation decoupling techniques to enhance the robustness and discriminative power of multimodal features.

[0004] However, existing methods often overlook the differences in dominance among modalities. Recent research has shown that linguistic, visual, and acoustic modalities contribute asymmetrically to the final prediction performance because they convey varying degrees of emotion-related information. Previous studies have indicated that the linguistic modality typically plays a dominant role in emotion reasoning, while visual and acoustic modalities serve more as supplementary sources. Leveraging this insight, some works have strategically enhanced the dominance of the linguistic modality in the fusion space by channeling auxiliary information into the fusion space. However, this enhancement is limited to the fusion space and lacks a unified modeling strategy that coordinates the feature space and the fusion space. Furthermore, this single-layer approach restricts the model's ability to construct stable untangled feature representations and effective emotion-aware decision features, thus limiting the exploitation of modality-specific advantages during multimodal representation learning. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes a multimodal sentiment analysis method and system based on a two-stage master modality guidance. By decoupling multimodal feature semantics, language-driven distribution alignment, and hierarchical attention enhancement, it achieves accurate prediction of multimodal sentiment intensity and improves the accuracy and robustness of sentiment analysis in complex scenarios.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a multimodal sentiment analysis method based on a two-stage guided master modality, comprising:

[0008] Acquire multimodal data to be predicted, including language, visual, and acoustic modalities;

[0009] The multimodal data is input into a trained multimodal sentiment analysis model to obtain sentiment intensity prediction results. Specifically, a feature extraction module extracts linguistic, visual, and acoustic features from the multimodal data; a feature space distribution alignment module decouples the multimodal features into modality-invariant and modality-specific features, and achieves feature distribution alignment dominated by the linguistic modality through alignment reconstruction constraints; an attention enhancement module performs self-attention modeling on the modality-invariant features to obtain a first enhanced feature, and adaptively enhances the visual and acoustic unique features through a cross-modal attention mechanism, dominated by the linguistic unique features, to obtain a second enhanced feature; a sentiment prediction module fuses the first and second enhanced features, and obtains the sentiment intensity prediction result through regression prediction.

[0010] Secondly, this invention provides a multimodal sentiment analysis system based on a two-stage guidance of the dominant modality, comprising:

[0011] A data acquisition unit is used to acquire multimodal data to be predicted, wherein the multimodal data includes language, visual and acoustic modalities;

[0012] The sentiment analysis unit is used to input the multimodal data into a trained multimodal sentiment analysis model to obtain sentiment intensity prediction results. Specifically, a feature extraction module extracts linguistic, visual, and acoustic features from the multimodal data; a feature space distribution alignment module decouples the multimodal features into modality-invariant features and modality-specific features, and achieves feature distribution alignment dominated by the linguistic modality through alignment reconstruction constraints; an attention enhancement module performs self-attention modeling on the modality-invariant features to obtain a first enhanced feature, and adaptively enhances the visual and acoustic-specific features through a cross-modal attention mechanism, dominated by the linguistic-specific features, to obtain a second enhanced feature; and a sentiment prediction module fuses the first and second enhanced features, and obtains the sentiment intensity prediction result through regression prediction.

[0013] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multimodal sentiment analysis method based on a two-stage master modality guidance described in the first aspect.

[0014] Fourthly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the multimodal sentiment analysis method based on a two-stage master modality as described in the first aspect.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0016] (1) This invention decouples multimodal features into modality-invariant features and modality-specific features through a feature space distribution alignment module, avoiding information redundancy and cross-modal leakage, and laying a precise feature foundation for sentiment analysis. Secondly, the alignment and reconstruction constraint dominated by the language modality ensures that visual and acoustic features are highly unified with the emotional semantics of language in the shared semantic space, while ensuring the reversibility and fidelity of feature structure and semantics. Furthermore, the attention enhancement module strengthens the core emotional semantics of modality-invariant features through self-attention, and guides cross-modal attention with language-specific features to adaptively enhance unique details in vision and acoustics that are strongly related to emotion, thereby achieving hierarchical and precise enhancement of multimodal features. Finally, the sentiment prediction module integrates the dual-enhanced features to accurately capture the collaborative emotional information of language, vision, and acoustics, greatly improving the accuracy and robustness of sentiment intensity prediction, and effectively solving problems such as feature mixing, cross-modal semantic inconsistency, and insufficient capture of emotional details in multimodal sentiment analysis.

[0017] (2) This invention proposes a two-stage guidance concept. The first stage focuses on the alignment and decoupling of the feature space, while the second stage strengthens the guidance role of language modalities during the fusion stage. Specifically, the first stage optimizes feature quality through decoupling and alignment to avoid fusion interference caused by the mixing of original features, providing a clean and unified feature input for subsequent processing. The second stage specifically strengthens key sentiment information, allowing the core semantics of modality-invariant features and the detailed information of modality-unique features to be accurately mined. This phased design makes the objectives of each stage of the model clear and each performs its own function, effectively avoiding the problems of incomplete feature processing and incomplete capture of sentiment information in single-layer fusion, and effectively improving the accuracy and robustness of sentiment analysis.

[0018] (3) This invention takes language modality as the main focus and uses distribution alignment constraints to make visual and acoustic features align with language semantics, which can solve the problem of cross-modal semantic inconsistency and ensure that multimodal features convey consistent emotional information in a unified semantic space. At the same time, by using language-specific features to guide cross-modal attention, the model can prioritize focusing on the emotional key areas indicated by language, reduce the interference of irrelevant information in auxiliary modalities, make multimodal fusion more targeted, and further improve the accuracy of emotional intensity prediction.

[0019] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0020] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute a limitation thereof.

[0021] Figure 1 This is a flowchart illustrating the main process of a multimodal sentiment analysis method based on a two-stage guidance of the dominant modality, as provided in an embodiment of the present invention.

[0022] Figure 2 A detailed flowchart of a multimodal sentiment analysis method based on a two-stage guidance of the dominant modality is provided for an embodiment of the present invention;

[0023] Figure 3 This is a schematic diagram of the feature space distribution alignment module provided in an embodiment of the present invention;

[0024] Figure 4 This is a schematic diagram of the decoupling regularization submodule provided in an embodiment of the present invention;

[0025] Figure 5 This is a schematic diagram of the alignment and reconstruction submodule provided in an embodiment of the present invention. Detailed Implementation

[0026] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0027] Example 1

[0028] like Figure 1 As shown, this embodiment discloses a multimodal sentiment analysis method based on a two-stage guidance of the dominant modality, including the following steps:

[0029] S1: Obtain the multimodal data to be predicted, including language, visual and acoustic modalities;

[0030] S2: Input the multimodal data into the trained multimodal sentiment analysis model to obtain the sentiment intensity prediction result; wherein, the feature extraction module extracts linguistic, visual, and acoustic features from the multimodal data; the feature space distribution alignment module decouples the multimodal features into modality-invariant features and modality-specific features, and achieves feature distribution alignment dominated by linguistic modality through alignment reconstruction constraints; the attention enhancement module performs self-attention modeling on the modality-invariant features to obtain the first enhanced feature, and, dominated by linguistic-specific features, adaptively enhances the visual and acoustic-specific features through a cross-modal attention mechanism to obtain the second enhanced feature; the sentiment prediction module fuses the first and second enhanced features, and obtains the sentiment intensity prediction result through regression prediction.

[0031] Next, combined Figure 2 This embodiment provides a detailed description of a multimodal sentiment analysis method based on a two-stage guidance of the dominant modality.

[0032] First, acquire the current multimodal data.

[0033] In this embodiment, a training dataset for a multimodal sentiment analysis model is constructed: First, short videos containing film reviews are obtained from publicly available videos as data sources. Through video segmentation, speech transcription, feature extraction, and sentiment annotation, a multimodal sentiment dataset integrating language, visual, and acoustic modalities is finally formed. Each multimodal data entry contains speech, video, and corresponding text, along with manually annotated sentiment polarity intensity (i.e., predicted sentiment intensity value) labels for use in training the multimodal sentiment analysis model.

[0034] 1. Collect original film review videos: First, obtain video clips containing film reviews from social media platforms, and then select film review materials with complete language expression and including facial expressions and voice signals.

[0035] For example, in the CMU-MOSI dataset, researchers extracted about 2,199 segments from film review videos by 93 different speakers. Each video segment had an average length of about 5 seconds, ensuring semantic integrity and containing identifiable emotional expressions.

[0036] 2. Speech Transcription and Text Construction: Speech recognition and text transcription are performed on the film review videos acquired by S101 to generate text information synchronized with the speech.

[0037] For example, in the CMU-MOSI dataset, an automatic speech recognition (ASR) system is used to extract speech content, which is then manually proofread to ensure that the text content is consistent with the audio semantics; finally, each video segment corresponds to a complete text sentence, which serves as the language modality input.

[0038] 3. Multimodal feature extraction and time alignment: Using timestamp information, the extracted raw data of the three modalities, namely video frames, audio signals and corresponding text transcription content, are aligned at the frame level to ensure the consistency of text, audio and visual frames in the time dimension.

[0039] After processing, each sample contains synchronized trimodal raw data (language, vision, and acoustics), providing a foundation for subsequent feature extraction and modeling.

[0040] For example, in the CMU-MOSI dataset, a speaker in a movie review video clip expresses the comment, "I really enjoyed this movie." First, all video frames within the corresponding time range (e.g., from 12.5 seconds to 17.2 seconds) are extracted from the video clip. Simultaneously, the audio signal for that time period is separated, and the start and end times of each word are precisely located based on the manually transcribed file. After timestamp alignment, the corresponding trimodal raw data for this sentence can be obtained.

[0041] 4. Artificial sentiment annotation and dataset construction: Assign sentiment polarity intensity to each time-aligned multimodal data point, and perform statistical analysis on all sentiment polarity intensities in the dataset.

[0042] For example, the MOSI dataset contains 2,199 subjective opinion videos, each rated by 5 annotators on sentiment intensity within the range of [-3, +3], where -3 represents extremely negative and +3 represents extremely positive. The average of the 5 annotators' ratings is used as the sentiment polarity label for that sample and is then used for training set label statistics. Subsequently, the sentiment polarity intensity corresponding to each multimodal data point in the multimodal sentiment analysis dataset is summarized to form the training set labels.

[0043] Furthermore, the multimodal data is input into a pre-trained multimodal sentiment analysis model for prediction to obtain the corresponding sentiment polarity intensity.

[0044] A multimodal sentiment analysis model is constructed, comprising a feature extraction module, a feature space distribution alignment module, an attention enhancement module, and a sentiment prediction module, connected sequentially. The attention enhancement module includes parallel intramodal self-attention enhancement modules and a language-guided cross-modal attention module.

[0045] (a) Feature Extraction Module

[0046] This module aims to extract semantically consistent and temporally aligned feature representations from three modalities: language, vision, and acoustic, and generate corresponding global CLS vectors to provide high-level input for subsequent multimodal fusion.

[0047] (1) Language modality feature extraction: For language modality, 300-dimensional GloVe static word vectors and 768-dimensional BERT-base-uncased context semantic vectors are first extracted from the input text to form a token-level representation sequence. Then, the [CLS] marker is introduced at the beginning of the sequence, and the corresponding CLS vector is extracted as language features through the BERT output layer. .

[0048] (2) Visual modality feature extraction: For the visual modality, 35-dimensional facial action unit (Action Unit) features were extracted from video frames using the Facet toolkit, and frame-level representations were obtained at a sampling rate of 30 FPS. After temporal alignment, the features were input into a lightweight Transformer encoder, and a [CLS] marker was added before the sequence. The output was then extracted as the visual features. .

[0049] (3) Acoustic modal feature extraction: For acoustic modalities, the COVAREP toolkit was used to extract 74-dimensional audio descriptors (window length 25 ms, step size 10 ms), including tract features, energy, formants, and vocal cord parameters. After alignment with the text timeline, the frame-level acoustic feature sequence was input into the Transformer encoder, and a [CLS] marker was added at the beginning. The output CLS vector was extracted as the acoustic feature. .

[0050] (ii) Feature Space Distribution Alignment Module

[0051] like Figure 3 As shown, this module aims to alleviate the coupling interference of heterogeneous modalities in the feature space through joint modeling of modality-invariant feature encoders and modality feature encoders, thereby achieving semantic decoupling and distribution alignment of multimodal features. This module consists of two sub-modules: a decoupling regularization sub-module and an alignment reconstruction sub-module, responsible for structural constraints in the feature space and cross-modal alignment, respectively.

[0052] (1) Decoupling regularization submodules: such as Figure 4 As shown, this submodule aims to achieve explicit decoupling between modality-invariant features and modality-specific features, thereby avoiding information redundancy and cross-modal leakage.

[0053] It should be understood that modality-invariant features are core information shared by data from different modalities, while modality-specific features are details exclusive to a single modality. Explicit decoupling involves using technical means to clearly separate these two types of features, allowing them to exist independently and without mixing. This separation avoids information redundancy—that is, the same core information is repeatedly stored in different modalities—and also prevents cross-modal leakage, such as errors in modality-specific details interfering with the processing of another modality, thereby ensuring the accuracy and efficiency of multimodal data processing.

[0054] Specifically, it consists of three types of regularization constraints, mainly comprising four parts: feature decoupling structure, cycle consistency loss, boundary loss, and orthogonality loss. Details are as follows:

[0055] 1. Construct a feature decoupling structure: To separate mode-invariant and mode-specific features from multimodal inputs, it receives the extracted input features of each mode. Using a shared mode-invariant encoder Obtain modality invariant features At the same time, an independent modality-specific encoder is designed for each modality. Obtain modal-specific features The specific formula is as follows:

[0056] (1)

[0057] (2)

[0058] 2. Obtaining Cycle Consistency Loss: To maintain the consistency of unique features during the recoding process, cycle consistency loss is introduced. First, the modality invariance and unique features are concatenated and then input into the modality decoder. Reconstruction is performed to obtain reconstructed features. The specific formula is as follows;

[0059] (3)

[0060] Subsequently, the features will be reconstructed. Enter the corresponding unique encoder again Obtain re-encoded features The specific formula is as follows:

[0061] (4)

[0062] Finally, unique features are constrained. With recoding features To achieve consistency and realize the goal of cycle reconstruction, thereby obtaining the cycle consistency loss. The specific formula is as follows:

[0063] (5)

[0064] In this embodiment, the purpose of calculating the cycle consistency loss is to maintain the consistency of modality-specific features during the encoding-reconstruction-reencoding process. By concatenating the modality-invariant features with the unique features, reconstructing them using a decoder, and then inputting the reconstructed features into a unique encoder to obtain re-encoded features, the consistency between the unique features and the re-encoded features is constrained, thereby achieving cyclic reconstruction. This ensures that the unique features remain stable in multimodal data processing, avoiding feature distortion caused by the encoding or reconstruction process.

[0065] 3. Obtaining Boundary Loss: To enhance the discriminative constraint of sentiment consistency across modalities, a boundary loss based on cosine similarity cross-modal triples is introduced. First, using mode-invariant features... As input, construct a set of triples based on the sentiment intensity labels. ,in Indicates anchor sample, This represents similar sentiment samples across modalities. This represents heterogeneous sentiment samples within the same modality. Secondly, it calculates the cosine similarity difference between the anchor sample and positive and negative samples by setting interval boundaries. The constraint is that the similarity between samples of the same emotional polarity intensity across modalities is higher than that between samples of different emotional polarities, thus obtaining the boundary loss. This is to achieve semantic aggregation of the modality-invariant feature space. The specific formula is as follows:

[0066] (6)

[0067] Where T represents the size of the cross-modal triplet set.

[0068] In this embodiment, modality-invariant features are used as input to construct a cross-modal triplet set. The cosine similarity difference between the anchor sample and the positive and negative samples is calculated. By setting an interval boundary, the similarity of cross-modal sentiment samples of the same type is constrained to be higher than that of dissimilar samples. This enables semantic aggregation of the modality-invariant feature space, making samples with the same emotional polarity intensity in different modalities more closely clustered in the feature space, and samples of different emotions more separated. This improves the ability to discriminate cross-modal features in tasks such as multimodal sentiment analysis and semantic understanding, and enhances the model's grasp of the cross-modal consistency of semantic information such as emotions.

[0069] 4. Obtaining Orthogonality Loss: To ensure information complementarity and semantic independence between modality-invariant features and modality-private features, an orthogonality loss is introduced. First, for mode-invariant features under the same mode... Modal unique features Vector normalization is performed to ensure scale consistency across different modalities. The specific formula is as follows:

[0070] (7)

[0071] (8)

[0072] Subsequently, through calculation and By using the cosine similarity between them to constrain their near-orthogonal relationship, we obtain the orthogonality loss. To reduce redundancy and information overlap, the specific formula is as follows:

[0073] (9)

[0074] in, Indicates the number of modes.

[0075] In this embodiment, after vector normalization of two types of features under the same modality, their cosine similarity is calculated and constrained to be close to orthogonal. This can reduce redundancy and information overlap between modality-invariant and unique features, allowing the two types of features to focus on capturing cross-modal general information and single-modal specific details respectively, achieving efficient division of information. This ensures effective sharing of cross-modal information in multimodal tasks while preserving the unique details of single modality, thereby improving the richness and discriminability of multimodal feature representation.

[0076] This embodiment introduces cycle consistency loss, boundary loss, and orthogonality loss, which work synergistically to form a complete constraint system from three dimensions: feature consistency, semantic discriminability, and information independence. This accurately separates the general core information from the single-modality specific details in multimodal data, effectively avoiding information redundancy and cross-modal leakage. It provides comprehensive and reliable technical support for the explicit decoupling of modality-invariant features and modality-specific features, effectively improving the quality of multimodal feature representation and the performance of multimodal tasks.

[0077] (2) Align and reconstruct sub-modules: such as Figure 5 As shown, this submodule aims to achieve a unified distribution and structure preservation of multimodal features in a shared semantic space, primarily driven by language modality. It specifically comprises two parts: distribution alignment loss and reconstruction loss. Details are as follows:

[0078] 1. Obtaining Distribution Alignment Loss: To construct a unified feature space centered on language modalities, a distribution alignment loss guided by the main modality is introduced. Reconstruction features of language modalities Based on this, the reconstructed features of auxiliary modalities (visual modality, acoustic modality) are analyzed. By applying the Euclidean distance minimization constraint, the distribution alignment loss is obtained. This ensures semantic consistency with the language modality. The specific formula is as follows:

[0079] (10)

[0080] In this embodiment, the reconstruction features of the text modality are used as a benchmark, and the Euclidean distance minimization constraint is applied to the reconstruction features of auxiliary modalities such as vision and speech. This enables the auxiliary modal features to maintain consistency with the language modality at the semantic level, realizes the distribution alignment of multimodal features in the shared semantic space, and allows features of different modalities to be compared and interact in the same space, thereby improving the semantic consistency of the multimodal system.

[0081] 2. Obtaining the reconstruction loss: To ensure the structural and semantic reversibility of the decoupled features, a reconstruction loss is introduced. By calculating reconstructed features Input features The Euclidean distance is used to obtain the reconstruction loss. To constrain the semantic fidelity and structural consistency of features, the specific formula is as follows:

[0082] (11)

[0083] In this embodiment, by calculating the Euclidean distance between the reconstructed features and the input features, the semantic fidelity and structural consistency of the features are constrained. This ensures that after alignment and other operations, the multimodal features can still retain the core semantic and structural information of the original input, avoiding semantic loss or structural distortion of features during processing, and providing quality assurance for the effective use of multimodal features.

[0084] This embodiment introduces distribution alignment loss and reconstruction loss to achieve unified distribution and structure preservation of multimodal features in a shared semantic space. This provides feature representations that are both consistent and faithful for tasks such as multimodal semantic understanding and cross-modal interaction, thereby improving the performance and reliability of multimodal systems.

[0085] (III) Attention Enhancement Module

[0086] (1) Intramodal self-attention enhancement module

[0087] This module aims to capture global semantic dependencies and shared structural consistency in modality-invariant features, thereby improving cross-modal fusion robustness.

[0088] First, take each mode-invariant feature Given the input, calculate its query respectively. ,key Sum The matrix is ​​used to obtain enhanced mode-invariant features through scaling dot product attention. This is to achieve semantic modeling and adaptive reinforcement of modality-invariant features. The specific formula is as follows:

[0089] (12)

[0090] in, Representing language, visual, and acoustic modalities; , , It is a learnable linear transformation matrix; The dimension of the key vector is used to scale the dot product result to stabilize the gradient.

[0091] (2) Language-guided cross-modal attention module

[0092] This module aims to further solidify the dominant position of language modalities in the multimodal fusion space, compensating for the shortcomings of the feature space distribution alignment stage which relies solely on distribution alignment loss for implicit constraints. This is achieved by leveraging language-specific features... As an anchor point, it guides the unique features of auxiliary modalities (visual, acoustic). In the fusion space, we adaptively focus on language-related emotional cues to achieve explicit dominance of the dominant modality and dynamic fusion of cross-modal information, as detailed below:

[0093] 1. Obtain the unique visual features guided by language: using language-specific features As a query With unique visual characteristics Generate keys respectively AND value The unique visual features guided by language are obtained through a cross-modal attention mechanism. The specific formula is as follows:

[0094] (13)

[0095] in, , , It is a learnable linear projection matrix.

[0096] 2. Obtaining the unique acoustic characteristics of language guidance: using the unique characteristics of language For query With unique acoustic characteristics For key Sum Acoustic features unique to language guidance are obtained through cross-modal attention mechanisms. The specific formula is as follows:

[0097] (14)

[0098] in, , , It is a learnable linear projection matrix.

[0099] 3. Obtain enhanced language-specific features: using language-specific features Calculate the query based on the input. ,key Sum And obtain enhanced language-specific features through self-attention mechanisms. The specific formula is as follows:

[0100] (15)

[0101] in, , , Learnable linear projection matrix.

[0102] This embodiment proposes a two-stage guidance concept based on a feature space distribution alignment module and an attention enhancement module. In the first stage, the feature space distribution alignment module separates modality-invariant and unique features through a decoupling regularization submodule, and then achieves language-led feature distribution unification through an alignment and reconstruction submodule, laying a semantically consistent feature foundation for subsequent fusion. In the second stage, the attention enhancement module takes over, with intra-modal self-attention strengthening the core semantics of modality-invariant features and language-guided cross-modal attention enhancing modality-unique details. The two stages are progressively advanced, solving the feature mixing problem and strengthening the capture of sentiment-related information, significantly improving the model's sentiment analysis capabilities for complex multimodal data.

[0103] (iv) Sentiment Prediction Module

[0104] This module aims to receive enhanced features processed by the attention enhancement module, perform unified fusion and sentiment regression prediction to estimate the sentiment intensity of the input segment. The processed enhanced features are then concatenated and fused to obtain the final joint feature representation. The specific formula is as follows:

[0105] (16)

[0106] in, , , These are the enhanced modality-invariant features for language, visual, and acoustic modalities, respectively.

[0107] To further enhance the expressive power and stability of fusion features, Input to a two-layer feedforward network In this process, the original information is preserved through a residual connection structure, achieving nonlinear mapping of features and optimized sentiment prediction output. Finally, a linear regression layer is used to obtain continuous sentiment intensity prediction values. The specific formula is as follows:

[0108] (17)

[0109] To optimize the model's sentiment prediction performance, the sentiment prediction loss is obtained by minimizing the absolute difference between the predicted value and the true sentiment label. This guides the model to learn a more accurate mapping relationship of sentiment intensity, and the specific formula is as follows:

[0110] (18)

[0111] in, For the sample size, and The first The actual sentiment labels of each sample and the model prediction results.

[0112] As one implementation method, the model proposed in this embodiment achieves better results than other models on the CMU-MOSI dataset. The comparison of experimental results is shown in Table 1 below.

[0113] Table 1. Comparison of experimental results on the CMU-MOSI dataset;

[0114]

[0115] The model proposed in this embodiment also achieved better results than other models on the CMU-MOSEI dataset. The comparison of experimental results is shown in Table 2 below.

[0116] Table 2 Comparison of experimental results on the CMU-MOSEI dataset;

[0117]

[0118] This specific embodiment utilizes three loss functions—orthogonality, cycle consistency, and boundary loss—to precisely decouple modality-invariant features from modality-specific features. This avoids information redundancy and prevents cross-modal leakage, providing a precise feature foundation for sentiment analysis. Secondly, distribution alignment and reconstruction constraints, primarily driven by the language modality, ensure high consistency between visual and acoustic features and linguistic sentiment semantics within a shared semantic space, while maintaining the reversible fidelity of feature structure and semantics, thus enhancing the semantic unity of multimodal features. Furthermore, the attention enhancement module strengthens the core sentiment semantics of modality-invariant features through self-attention and guides cross-modal attention with linguistic-specific features, adaptively enhancing unique details in visual and acoustic representations that are strongly correlated with sentiment, achieving hierarchical enhancement of multimodal features. Finally, the sentiment prediction module, integrating dual-enhanced features, accurately captures collaborative sentiment information from language, vision, and acoustics, significantly improving the accuracy and robustness of sentiment intensity prediction. This effectively addresses pain points in multimodal sentiment analysis such as feature mixing, cross-modal semantic inconsistency, and insufficient capture of sentiment details, providing an efficient and reliable technical solution for multimodal sentiment understanding in complex scenarios.

[0119] Example 2

[0120] This embodiment provides a multimodal sentiment analysis system based on a two-stage guidance of the dominant modality, including:

[0121] A data acquisition unit is used to acquire multimodal data to be predicted, wherein the multimodal data includes language, visual and acoustic modalities;

[0122] The sentiment analysis unit is used to input the multimodal data into a trained multimodal sentiment analysis model to obtain sentiment intensity prediction results. Specifically, a feature extraction module extracts linguistic, visual, and acoustic features from the multimodal data; a feature space distribution alignment module decouples the multimodal features into modality-invariant features and modality-specific features, and achieves feature distribution alignment dominated by the linguistic modality through alignment reconstruction constraints; an attention enhancement module performs self-attention modeling on the modality-invariant features to obtain a first enhanced feature, and adaptively enhances the visual and acoustic-specific features through a cross-modal attention mechanism, dominated by the linguistic-specific features, to obtain a second enhanced feature; and a sentiment prediction module fuses the first and second enhanced features, and obtains the sentiment intensity prediction result through regression prediction.

[0123] Example 3

[0124] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a multimodal sentiment analysis method based on a two-stage master modality guidance as described in Embodiment 1 above.

[0125] Example 4

[0126] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the multimodal sentiment analysis method based on a two-stage master modality as described in Embodiment 1 above.

[0127] The steps or modules involved in Embodiments 2 to 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0128] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for multi-modal sentiment analysis based on dominant modality two-stage guidance, characterized in that, include: Acquire multimodal data to be predicted, including language, visual, and acoustic modalities; The multimodal data is input into a trained multimodal sentiment analysis model to obtain sentiment intensity prediction results. Specifically, a feature extraction module extracts linguistic, visual, and acoustic features from the multimodal data; a feature space distribution alignment module decouples the multimodal feature semantics into modality-invariant features and modality-specific features; and alignment reconstruction constraints are used to achieve feature distribution alignment dominated by the linguistic modality. The decoupled modal features are input into the reconstruction network to generate reconstructed features. The difference between the reconstructed features and the original features is calculated by the reconstruction loss to ensure the semantic and structural fidelity of the features. Based on the reconstructed features of the language modality, the distance between the visual and acoustic modality reconstructed features and the language reconstructed features is minimized by the distribution alignment loss, which forces the auxiliary modality to align with the language modality in the shared semantic space, forming a unified feature distribution dominated by language. The attention enhancement module performs self-attention modeling on the modality-invariant features to obtain the first enhanced feature. Then, taking language-specific features as the main component, the visual and acoustic-specific features are adaptively enhanced through a cross-modal attention mechanism to obtain the second enhanced feature. The sentiment prediction module fuses the first and second enhanced features and obtains the sentiment intensity prediction result through regression prediction.

2. The method of claim 1, wherein the method is based on a two-stage guidance of the dominant modality, and The feature extraction module includes three sub-modules, which are used to extract language features, visual features, and acoustic features based on the input multimodal data.

3. The method of claim 1, wherein the method is based on a two-stage guidance of dominant modality, and The decoupling of multimodal feature semantics into modality-invariant features and modality-unique features specifically includes: Each modal feature is input into the modality-invariant encoder and the modality-specific encoder respectively, and modality-invariant features and modality-specific features are extracted. The decoupling regularization loss function is used for constraint. Among them, the cycle consistency loss is used to ensure that modality-unique features maintain feature consistency after encoding, reconstruction, and re-encoding; the boundary loss constrains the aggregation of modality-invariant features of the same type and the separation of different types by constructing cross-modality triplet samples; and the orthogonality loss is used to constrain modality-invariant features and modality-unique features to be semantically independent.

4. The method of claim 1, wherein the method is based on a two-stage guidance of dominant modality, and The attention enhancement module includes a parallel intramodal self-attention enhancement module and a language-guided cross-modal attention module; The intramodal self-attention enhancement module is used to model modality-invariant features. By calculating the correlation weights of each element within the feature, it strengthens key emotional semantic information, weakens irrelevant interference, and generates a first enhanced feature that focuses on the core semantics. The language-guided cross-modal attention module is used to calculate the semantic similarity between language-specific features and visual and acoustic-specific features, and assign weights accordingly. It dynamically enhances the key unique information of the auxiliary modality with language semantics and outputs the second enhanced feature.

5. The method of claim 1, wherein the method is based on a two-stage guidance of dominant modality, and The process of performing self-attention modeling on the modality-invariant features to obtain the first enhanced feature specifically includes: The modality-invariant features are transformed into a feature matrix, and then three types of vectors—query, key, and value—are generated through linear transformation. The similarity between the query and the key is calculated to obtain the attention weight, and the weight ratio of sentiment-related features is strengthened. Weighted aggregation of value vectors is performed based on weights, and the aggregation result is fused with the original modality-invariant features to obtain the first enhanced feature that strengthens the key semantics.

6. The method of claim 1, wherein the method is based on a two-stage guidance of dominant modality, and The second enhanced feature, which is primarily based on language-specific features and adaptively enhances visual and acoustic features through a cross-modal attention mechanism, specifically includes: Using language-specific features as guiding vectors, semantic matching is performed with visual and acoustic-specific features respectively; The similarity between the language guidance vector and the elements of the auxiliary modality-specific features is calculated to generate dynamic attention weights; By weighting and fusing the unique features of the auxiliary modality, a second enhanced feature that incorporates language semantic guidance is obtained.

7. A multi-modal sentiment analysis system based on dominant modality two-stage guidance, characterized in that, include: A data acquisition unit is used to acquire multimodal data to be predicted, wherein the multimodality includes language, visual and acoustic modalities; The sentiment analysis unit is used to input the multimodal data into a trained multimodal sentiment analysis model to obtain sentiment intensity prediction results. Specifically, it extracts linguistic, visual, and acoustic features from the multimodal data through a feature extraction module; and decouples the multimodal feature semantics into modality-invariant and modality-specific features through a feature space distribution alignment module, and achieves feature distribution alignment dominated by the linguistic modality through alignment reconstruction constraints. The decoupled modal features are input into the reconstruction network to generate reconstructed features. The difference between the reconstructed features and the original features is calculated by the reconstruction loss to ensure the semantic and structural fidelity of the features. Based on the reconstructed features of the language modality, the distance between the visual and acoustic modality reconstructed features and the language reconstructed features is minimized by the distribution alignment loss, which forces the auxiliary modality to align with the language modality in the shared semantic space, forming a unified feature distribution dominated by language. The attention enhancement module performs self-attention modeling on the modality-invariant features to obtain the first enhanced feature. Then, taking language-specific features as the main component, the visual and acoustic-specific features are adaptively enhanced through a cross-modal attention mechanism to obtain the second enhanced feature. The sentiment prediction module fuses the first and second enhanced features and obtains the sentiment intensity prediction result through regression prediction.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the program is executed by the processor, it implements the steps in the multimodal sentiment analysis method based on a two-stage master modality as described in any one of claims 1-6.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the multimodal sentiment analysis method based on a two-stage master modality as described in any one of claims 1-6.

Images