Mongolian multi-modal sentiment analysis method based on cross-modal information enhancement and fusion

Abstract

A Mongolian multi-modal sentiment analysis method based on cross-modal information enhancement and fusion. To solve the problems of scattered sentiment expression caused by the agglutinative characteristics of Mongolian and the difficulty of capturing implicit clues in multi-modal sentiment information, a three-level innovative solution is proposed: (1) Cross-modal explicit enhancement layer: Use multi-modal large language models to convert implicit sentiment clues such as prosody and expression in audio and video into text descriptions of sentiment, enriching the information of each modality through the "implicit→explicit→semantic fusion" path; (2) Semantic alignment layer based on Gram matrix: Extract the second-order statistical structure of the text modality as the semantic reference, and map heterogeneous audio and video features to a unified semantic space to solve the cross-modal alignment difficulty caused by the morphological changes of Mongolian; (3) Gating displacement fusion layer: Use an adaptive displacement mechanism to fine-tune the semantic position of text features to achieve precise injection of non-text information, avoiding the modality submersion problem in long morpheme sequences of Mongolian.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to sentiment analysis, and particularly to a method for multimodal sentiment analysis of Mongolian based on cross-modal information enhancement and fusion. This invention focuses on the sentiment analysis problem of low-resource languages, and, considering the unique characteristics of Mongolian as a typical agglutinative language in sentiment expression and multimodal fusion, proposes a cross-modal information enhancement and fusion technology solution adapted to the characteristics of agglutinative languages. Background Technology

[0002] Multimodal sentiment analysis aims to understand human emotions through information from multiple modalities, including text (speech content), visual information (facial expressions, gestures, etc.), and speech information (stress, intonation, etc.). With the help of multimodal sentiment analysis technology, more natural and intelligent human-computer interaction can be achieved.

[0003] While multimodal information input expands the machine's perception range, compared to sentiment analysis research in Chinese and English, Mongolian, as a minority language, has unique structural and expressive characteristics that lead to the asynchronicity, heterogeneity, and combinatoriality of multimodal sentiment information, posing significant challenges to multimodal sentiment analysis research in Mongolian.

[0004] I. Linguistic Features of Mongolian and Their Challenges to Sentiment Analysis

[0005] Mongolian belongs to the Mongolian branch of the Altaic language family. It is a typical agglutinative language and has the following notable characteristics:

[0006] 1. Morphological complexity

[0007] Mongolian expresses grammatical relationships and semantic information by adding a large number of affixes after word roots. A Mongolian word may contain: word root + case marker + possessive marker + number marker + tense marker, etc. This morphological complexity may lead to the following problems: (1) Scattered sentiment morphemes: sentiment information may be scattered in multiple components such as word roots, affixes, and modal particles, and simple word-level feature extraction is difficult to fully capture sentiment semantics; (2) Lexical sparsity: morphological changes generate a large number of low-frequency words, which exacerbates the data sparsity problem of low-resource languages; (3) Long-distance dependence: SOV word order (subject-object-predicate) makes the core sentiment word (usually the predicate) located at the end of the sentence, which is far away from the subject and is not easy for the model to learn.

[0008] 2. Vowel Harmony

[0009] The vowel harmony of Mongolian requires that vowels within a word must belong to the same category (masculine / feminine / neuter), which affects the choice of affixes and phonetic rhythm, and thus the acoustic expression of emotions.

[0010] 3. Multimodal dependence of emotion expression

[0011] Because emotional markers in Mongolian texts are relatively subtle, their emotional expression relies more on prosodic features (i.e., acoustic cues such as tone, speech rate, and pauses) and non-linguistic features (i.e., visual cues such as facial expressions and gestures). Therefore, sentiment analysis in Mongolian relies more on the fusion of multimodal information than in Indo-European languages.

[0012] II. Limitations of Existing Multimodal Sentiment Analysis Techniques

[0013] Existing multimodal sentiment analysis methods mainly suffer from the following problems:

[0014] The first problem is the difficulty in mining implicit emotional information. While the text modality is dominant in existing multimodal sentiment analysis, it fails to capture the numerous non-linguistic cues (such as instantaneous facial micro-movements, acoustic prosody changes, rhythmic pauses, and energy shifts) in audio and video that carry significant emotional discriminative power. These implicit emotional cues often remain in an "implicit vector" form when directly spliced ​​or simply integrated with text, making it difficult to fully utilize the model's internal linguistic priors. This results in existing models struggling to capture emotional cues hidden in non-textual modalities, further reducing their ability to discriminate complex emotions. This is particularly true for languages ​​like Mongolian, where emotional expression relies heavily on prosody and facial expressions; implicit features cannot fully leverage the linguistic priors of the text modality. For example, the implicit feature vector [0.23, -0.45, ...] of "high pitch + fast speech rate" in audio is difficult to explicitly associate with emotional words expressing "excitement" or "excitement" in text.

[0015] The second problem is the difficulty of aligning multimodal sentiment semantics. Due to the asynchronicity and heterogeneity of sentiment information from different modalities, models struggle to accurately model the semantic relationships between them, and direct fusion can lead to a "semantic gap." Existing alignment methods (such as simple linear projection) only consider first-order statistical information and ignore the relational structure between features. For agglutinative languages ​​like Mongolian, where intra-word components are tightly linked, this can easily disrupt the integrity of the semantics.

[0016] The third problem is the difficulty of multimodal sentiment semantic fusion. Due to the combinatorial nature of sentiment information from different modalities, simple sentiment information can express complex emotions through different combinations. Existing models suffer from "modal bias" in the text modality—that is, multimodal models often over-rely on the text modality while ignoring the unique sentiment information in the weaker modality, i.e., the audiovisual modality. This prevents the model from fully learning semantic knowledge from non-textual modalities, significantly weakening its ability to discern complex emotions, which poses a great challenge to existing fusion methods. For Mongolian, because sentiment markers in the text are implicit, this bias can cause the model to miss key sentiment cues in audio and video.

[0017] In addition, traditional splicing or weighted fusion methods are prone to problems such as dimensionality explosion (linear growth of feature dimensions and high computational cost), modality submersion (information of weak modalities is submerged by strong modalities), and semantic perturbation (for long morpheme sequences in Mongolian, simple splicing will destroy the original semantic structure).

[0018] The fourth problem is the lack of research on low-resource languages. Existing multimodal sentiment analysis research mainly focuses on high-resource languages ​​such as English and Chinese, and the relevant datasets, pre-trained models, and technical solutions are difficult to directly transfer and apply to low-resource minority languages ​​such as Mongolian.

[0019] III. Technical Requirements

[0020] Therefore, there is an urgent need for a multimodal sentiment analysis method targeting the stickiness characteristics of the Mongolian language. This method should:

[0021] (1) It can make the implicit emotional cues in audio and video explicit, which is convenient for integration with text semantics;

[0022] (2) It can achieve cross-modal alignment while maintaining the integrity of the Mongolian morphological structure;

[0023] (3) It can avoid text bias and make full use of the emotional information in audio and video;

[0024] (4) It can adapt to low-resource scenarios and has strong generalization ability. Summary of the Invention

[0025] To overcome the shortcomings of the prior art, the present invention aims to provide a Mongolian multimodal sentiment analysis method based on cross-modal information enhancement and fusion, in order to solve the problems of insufficient existing Mongolian sentiment analysis research results and existing technologies such as insufficient extraction of implicit sentiment cues, difficulty in cross-modal semantic alignment, imbalance in modal fusion, and insufficient adaptation to the characteristics of agglutinative languages, thereby achieving good Mongolian multimodal sentiment analysis results and efficiency.

[0026] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0027] A Mongolian multimodal sentiment analysis method based on cross-modal information enhancement and fusion includes the following steps:

[0028] Step 1: Preprocess the raw data of text, audio, and video modalities to extract initial features for each modality, and then use a Transformer encoder to obtain shallow feature sequences for each modality. Simultaneously, through the speech description generation part based on a large language model, obtain shallow feature sequence one regarding emotional cues in the audio modality from the preprocessed audio data, and concatenate it with the audio shallow feature sequence to obtain audio enhancement features. Through the video description generation part based on a large language model, obtain shallow feature sequence two regarding emotional cues in the video modality from the preprocessed video data, and concatenate it with the video shallow feature sequence to obtain video enhancement features. Finally, concatenate the text shallow feature sequence with shallow feature sequence one and shallow feature sequence two to obtain text enhancement features.

[0029] Step 2: A modular interactive attention mechanism is used to achieve deep semantic alignment between various enhanced feature modalities. Self-modal attention is used to refine the feature associations within a modality, and cross-modal attention is used to ensure deep fusion between different modalities, ultimately obtaining the deep features of each modality.

[0030] Step 3: First, use a multi-head self-attention mechanism to enable the deep features of audio and video to interact fully and obtain non-text features. Then, inject the non-text features into the deep text features through a weighted shift operation to form a joint multimodal sentiment representation. Finally, use the joint multimodal sentiment representation to achieve sentiment classification.

[0031] Compared with the prior art, the beneficial effects of the present invention are:

[0032] (1) By utilizing a cross-modal information enhancement method based on a large language model, high-quality semantic descriptions of audiovisual modalities are generated and fused with the original features to form enhanced text, audio, and video modalities with richer information and more comprehensive semantics, thereby improving the model's ability to extract implicit sentiment cues in the shallow space. For Mongolian, this enhancement is particularly important because sentiment markers in the text are relatively implicit, and it can effectively make up for the shortcomings of relying solely on the text.

[0033] (2) A feature alignment and preliminary fusion method based on intermodal interactive attention mechanism is used to achieve deep alignment and dynamic fusion between features of different modalities through intermodal interactive attention and modal masking strategy. Among them, the Gram matrix-based alignment method is more suitable for the characteristics of agglutinative languages. For long morpheme sequences in Mongolian, the Gram matrix can capture the association patterns between morphemes and avoid simple linear projection from destroying semantic integrity.

[0034] (3) Using the attention-guided fusion method based on displacement, the visual and auditory modal features are guided by the gating displacement mechanism to enhance the interaction between modalities and realize the deep fusion and full interaction of multimodal information, thereby improving the performance and generalization of Mongolian sentiment analysis. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the overall process of the present invention.

[0036] Figure 2 This is a schematic diagram of the overall architecture of the model of this invention.

[0037] Figure 3 This is a flowchart of the audio description generation process of the present invention.

[0038] Figure 4 This is a flowchart of the video description generation process of the present invention.

[0039] Figure 5 This is a schematic diagram of the overall architecture of the MIA of this invention.

[0040] Figure 6 This is a schematic diagram of the Aligner architecture of the present invention.

[0041] Figure 7 This is a schematic diagram of the modular attention mask of the present invention.

[0042] Figure 8 This is a schematic diagram of the attention-guided fusion strategy architecture based on the displacement concept of the present invention. Detailed Implementation

[0043] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings and examples.

[0044] To address the problem of scattered emotional expression caused by the agglutinative nature of Mongolian and the technical challenge of capturing implicit cues in multimodal emotional information, this invention provides a Mongolian multimodal sentiment analysis method based on cross-modal information enhancement and fusion. It innovates in three levels, improving the effectiveness and efficiency of Mongolian multimodal sentiment analysis from three aspects: data enhancement, multimodal sentiment semantic alignment, and multimodal sentiment semantic fusion. (1) Cross-modal explicit enhancement layer: using a multimodal large language model to transform implicit emotional cues such as rhythm and expression in audio and video into Mongolian emotional descriptions. The text enriches text modality and audiovisual modality information through the path of "implicit → explicit → semantic fusion"; (2) Semantic space alignment layer based on Gram matrix: extract the second-order statistical structure of text modality as semantic reference space, map heterogeneous audio and video features to a unified semantic space, and solve the cross-modal alignment difficulties caused by morphological changes in Mongolian; (3) Gated displacement fusion layer: adopts an adaptive displacement mechanism instead of simple splicing, and achieves accurate injection of non-text information by fine-tuning the semantic position of text features, avoiding the modality submersion problem in long morpheme sequences of Mongolian. Under the premise of maintaining the integrity of Mongolian language structure, this invention realizes the deep fusion of multimodal emotional information, and provides a new technical path for multimodal sentiment analysis of low resource cohesion languages.

[0045] like Figure 1 and Figure 2 As shown, the main steps of this invention are as follows:

[0046] Step 1: Perform preliminary preprocessing on the raw data of text, audio, and video modalities.

[0047] Step 2: Based on the preprocessed data, initial features are extracted for text, audio, and video modalities respectively. Then, a Transformer encoder is used to obtain shallow feature sequences for each modality. Furthermore, through the speech description generation part based on a large language model, shallow feature sequence one regarding emotional cues in the audio modality is obtained from the preprocessed audio data; similarly, through the video description generation part based on a large language model, shallow feature sequence two regarding emotional cues in the video modality is obtained from the preprocessed video data.

[0048] This invention can extract initial text features using RoBERTa, initial audio features using LibROSA, and initial video features using OpenFace. Then, the initial text features are processed by a Transformer encoder to obtain a shallow feature sequence for the text modality, the initial audio features by a Transformer encoder to obtain a shallow feature sequence for the audio modality, and the initial video features by a Transformer encoder to obtain a shallow feature sequence for the video modality. The specific processing method is as follows:

[0049] a. Text processing

[0050] This invention uses RoBERTa to process text. RoBERTa is a multilingual pre-trained model that systematically re-evaluates and improves the original BERT pre-training strategy, and generally outperforms the original BERT at the same scale.

[0051] When processing Mongolian text using RoBERTa, the text is segmented and encoded by loading a multilingual RoBERTa model. The processed input is then fed into the model to extract embedded features, which effectively express the semantic information of the text.

[0052] b. Audio processing

[0053] This invention uses LibROSA to process speech features. LibROSA is a Python library specifically developed for audio signal processing and music data parsing, providing various tools and corresponding function interfaces for audio feature extraction. Users only need to provide the audio signal and related parameters to obtain a normalized feature matrix output for feature extraction and analysis. LibROSA's feature package provides a variety of feature extraction functions, including chromatograms, Mel spectra, Mel frequency cepstral coefficients (MFCCs), and various other spectral and volume features, significantly improving the convenience and visualization of feature extraction.

[0054] First, the LibROSA library is used to read the audio data, obtaining the sampling rate and audio signal. Then, the audio signal is normalized to ensure it falls within the range [-1, 1], and audio features such as MFCC and spectral characteristics are extracted. Finally, Short-Time Fourier Transform (STFT) is used to perform frequency domain analysis on the audio signal to examine its spectral characteristics.

[0055] c. Video processing

[0056] This invention uses OpenFace to extract features from videos. OpenFace is an open-source facial behavior analysis framework jointly developed by Carnegie Mellon University and the University of Cambridge, supporting tasks such as facial landmark detection, expression recognition, identity verification, and head pose estimation. During feature extraction, OpenFace first uses pre-trained models from dlib or OpenCV to detect face regions and then aligns the faces using affine transformations to reduce the influence of pose and lighting. Subsequently, a deep neural network maps facial features into 128-dimensional vectors, capturing subtle facial differences to facilitate subsequent classification and regression tasks.

[0057] In this invention, OpenCV is first used to read video frames, and certain keyframes are selected as images for feature extraction. Then, OpenFace is used for face detection and feature extraction. Once a human face is detected, OpenFace extracts a series of facial features, including facial landmarks and facial expression features.

[0058] The raw data underwent data preprocessing to obtain the initial features of each mode. ,in .

[0059] While multimodal sentiment analysis theoretically aims for a synergistic and relatively balanced decision-making role across text, video, and audio modalities, numerous empirical studies show that the contributions of each modality to model performance are inconsistent. Text modality, due to its concentrated semantic and emotional information and relatively low annotation noise, dominates and has a significantly greater impact on classification results than video and audio signals. However, when humans express emotions implicitly (e.g., using superficially neutral or positive language to express dissatisfaction), negative emotions are often conveyed through subtle facial movements, body posture, and vocal rhythm. Text often lacks these implicit cues, making it difficult for models to capture this cross-modal inconsistency. This leads to the "modal gap" problem, severely limiting the model's ability to correctly identify implicit emotions (variants of explicit emotions such as sarcasm, repression, and restraint).

[0060] The aforementioned problems are even more pronounced in Mongolian language scenarios. As a typical low-resource language, Mongolian multimodal sentiment corpora are limited in size and have coarse-grained labels. Traditional models often overfit rapidly under extremely sparse supervision signals and tend to rely on the relatively easier-to-model text modality. Furthermore, because Mongolian texts lack explicit encoding of subtle facial expressions and acoustic prosody, many cues needed for sentiment discrimination are missing from the training signals. This limits the model's ability to recognize implicit emotional phenomena such as micro-expressions, sarcasm, and tone of voice, further reducing the model's ability to analyze real emotions.

[0061] To address the issue of data sparsity and the lack of implicit semantics in the text modality, which leads to the absence of implicit emotional cues in the audiovisual modality, due to the agglutinative nature of Mongolian, this invention employs a cross-modal information enhancement module based on a large language model to enrich the semantic features of each original modality, especially the text modality, which is the dominant modality. Leveraging the powerful contextual understanding and structured description generation capabilities of the large language model, implicit emotional information in the non-textual modality is "explicitly" and "textualized," and then fused with the initial text features. This directly enriches the dominant text modality in the fusion process, effectively bridging the information gap between different modalities. Simultaneously, by fusing the semantic descriptions of the audiovisual modality generated by the large model with the original modality, enhancement of the non-textual modality is achieved.

[0062] This invention, through a speech description generation component based on a large language model, obtains a shallow feature sequence of audio modality sentiment cues from preprocessed audio data, such as... Figure 3 As shown, the implementation method is as follows:

[0063] 1) Input the preprocessed audio data and corresponding prompts into the large model Qwen-Audio. In practical applications, the prompts, system commands, and audio paths can be input into Qwen-Audio separately.

[0064] 2) Qwen-Audio encodes the prompt words through an embedding layer to obtain encoded features. The audio data is resampled and subjected to short-time Fourier transform to obtain the Log-Mel spectrum. And concatenate it with the encoded features to obtain For example, using components such as the Log-Mel filter bank, the original audio is first resampled to 16kHz, and an 80-dimensional Log-Mel spectrum is extracted through short-time Fourier transform. Each frame (or a time-sequential frame segment) is treated as a sequence of audio tokens and fed into a Transformer-based audio encoder for contextual feature modeling.

[0065] 3) Generate high-quality structured text descriptions of audio by utilizing the concatenated features from the Qwen large model within Qwen-Audio. This is to fill the information gap in emotional expression in audio modalities.

[0066] 4) Describe the audio using structured text. The token sequence is processed using a BERT encoder. This refers to the first shallow feature sequence regarding audio modality emotional cues. The above process can be described by the following formula:

[0067]

[0068]

[0069]

[0070]

[0071]

[0072]

[0073] This invention, through a video description generation component based on a large language model, obtains a shallow feature sequence two regarding video modality sentiment cues from preprocessed video data, such as... Figure 4 As shown, the implementation method is as follows:

[0074] 1) Input the preprocessed video data and corresponding prompts into the large model Qwen-2.5-VL. In practical applications, the prompts, system commands, and video paths can be input into Qwen-2.5-VL separately.

[0075] 2) Qwen-2.5-VL encodes the prompt words to obtain encoded features. ViT is used to extract image features from the images obtained by extracting frames from video data. and with encoding features By splicing Specifically, the video is first framed at a fixed frequency, and then the images obtained from the frame extraction are used for image feature extraction using ViT. ViT divides the image into 14×14 image blocks, treats each image block as a token, and passes it into the Transformer structure for feature extraction.

[0076] 3) Utilize the features obtained by stitching together the large Qwen model within Qwen-2.5-VL. Generate high-quality structured text descriptions for videos To fill the information gaps in emotional expression in video modalities;

[0077] 4) Structural text description of the video The token sequence is processed using a BERT encoder. This refers to the second shallow feature sequence regarding video modality sentiment cues. The above process can be described by the following formula:

[0078]

[0079]

[0080]

[0081]

[0082]

[0083]

[0084] This invention utilizes the speech description generation component of a large language model. Leveraging the model's superior understanding capabilities, it analyzes and generates structured semantic descriptions of body postures and facial micro-expressions in videos, as well as emotional cues such as intonation, speed, and tone in audio. These descriptions are then encoded using BERT to obtain corresponding shallow feature sequences. These sequences are then deeply fused with the corresponding original audiovisual features to enhance their semantic expressiveness. Furthermore, they are linked across modalities with textual modal features, resulting in a semantically enhanced multimodal feature representation.

[0085] Step 3: Concatenate the first shallow feature sequence with the audio shallow feature sequence to obtain the audio enhancement feature, and concatenate the second shallow feature sequence with the video shallow feature sequence to obtain the video enhancement feature.

[0086] Step 4: Concatenate the text shallow feature sequence with shallow feature sequence one and shallow feature sequence two to obtain the text enhancement features.

[0087] Steps 3 and 4, where the structured semantic descriptions are encoded using BERT, are concatenated with the encoded sequences of the original data through fully connected layers to enhance features. Specifically:

[0088] The token sequence obtained in step 2, which describes the structured textual descriptions of implicit emotional cues in audio and video. and That is, shallow feature sequence one and shallow feature sequence two, and the text shallow feature sequence generated during the data preprocessing stage. The concatenated features are then joined to obtain a cross-modal joint feature representation. Subsequently, a fully connected layer maps the concatenated features along the dimensional lines to form text-enhanced features. At the same time, and The audio shallow feature sequences generated after preprocessing the original modality are compared with those of the original modality. With shallow feature sequences of video The connection is established, and a linear transformation is performed through a fully connected layer to output audio enhancement features. Video enhancement features .

[0089] This invention, based on cross-modal information enhancement using a large language model, innovatively utilizes the powerful contextual understanding and structured description generation capabilities of the large language model as a "perception generator." It deeply analyzes key emotional cues such as body posture and facial expressions in the video, and tone, intonation, and energy in the audio, generating corresponding textual descriptions. Subsequently, these descriptive texts, imbued with non-verbal emotional information, are deeply integrated with the original Mongolian text and corresponding original modal features to form a more information-rich and semantically comprehensive enhanced text and audiovisual modality.

[0090] Therefore, by utilizing a large model to explicitly and textualize the implicit emotional information in audiovisual modalities, this invention enriches and enhances the data at the semantic level. This allows the original data, especially the relatively simple text modal, to acquire supplementary information from other modalities, making it more "three-dimensional" and "full-bodied." This method can fill the semantic gaps between modalities by generating descriptions when information is missing or modal expression is insufficient, laying a unified semantic foundation for subsequent feature fusion.

[0091] Step 5: Utilize a modular interactive attention mechanism to achieve deep semantic alignment among various enhanced feature modalities.

[0092] In multimodal sentiment analysis, a key challenge stems from the inherent limitations of the global unified attention mechanism. Different modalities reflect different emotional tendencies, and these implicit multimodal data may contain independent or consistent information, thus contributing differently to the final sentiment prediction. When models face these modalities, especially when text, as a human language expression rich in high-level semantic information, coexists with low-level features such as speech and visual modalities, the global attention mechanism is prone to attention bias: computational resources and weights are excessively concentrated on the information-dense dominant modality, i.e., text; while fine-grained emotional cues contained in micro-expressions, intonation changes, etc., are regarded as "weak modalities" due to their relatively sparse signals, and their features are severely suppressed or even submerged during the fusion process.

[0093] Furthermore, the heterogeneity of multimodal data—specifically, the inherent semantic misalignment of feature spaces among different modalities—further exacerbates this problem, making it difficult to effectively understand and utilize the features of weaker modalities within a unified framework. Simply fusing features from different modalities fails to adequately consider the differences between modalities and the weighting issues, thus failing to achieve the desired results. More importantly, directly removing noisy modalities may result in the loss of information unique to each modality.

[0094] To address the semantic heterogeneity of multimodal features and the modality bias caused by global attention, this invention proposes a modality alignment mechanism to alleviate the semantic gap. Based on the Gram matrix, semantic space basis vectors of text modalities are extracted, and these are used as a reference to transfer feature representations of non-text modalities to a unified semantic space. This fundamentally bridges the gap between modalities, laying the foundation for effective subsequent interaction and fusion.

[0095] This invention proposes a Modular Interactive Attention (MIA) mechanism to effectively address the heterogeneity of multimodal data and balance attention allocation. It divides traditional integrated attention computation into two parallel processes: independent self-modal and cross-modal. Through the collaborative design of two core components—alignment and masking—and by designing independent attention masking strategies for the two interaction modes, this mechanism ensures the full extraction of intra-modal features and the effective exchange of cross-modal information. This significantly enhances the contribution weight and presence of weaker modal information in the final decision, systematically optimizing the interaction and fusion process of multimodal information.

[0096] First, this mechanism proposes a Modular Interactive Alignment module to construct a cross-fusion mechanism dominated by text features. This module utilizes A-Aligner and V-Aligner to map tokens from audio and video modalities to the semantic space of text modalities. The core idea is to compute the Gram matrix representing the tokens of the text modalities, extract core basis vectors that represent the structure of this semantic space, and use these as reference benchmarks for cross-modal mapping, achieving efficient conversion of tokens from different modalities within a unified semantic space. This method can achieve deeper semantic alignment while preserving the original semantics of each modality and helps reduce low-order redundant information from non-textual modalities.

[0097] After achieving semantic alignment, a modular attention masking module is added. This module regulates the attention flow to avoid the dominant modality suppressing the weaker modalities. Attention computation is decomposed into two independent streams: self-modal attention and cross-modal attention. Self-modal attention focuses on processing and refining feature associations within a modality, ensuring the integrity and effective transmission of information within each modality. Cross-modal attention is specifically responsible for handling feature interactions between different modalities, using textual information as the core and fusing it with features from other non-textual modalities (speech, video) to aid in sentiment inference.

[0098] By designing independent, learnable masking strategies for these two attention streams, MIA can leverage complementary features from non-textual modalities to enhance intermodal correlation, ultimately ensuring that audio, video, and text information are used in a balanced and sufficient manner in emotion recognition tasks. This significantly improves the robustness and interpretability of the model while strengthening implicit textual feature representations.

[0099] Because the feature spaces of visual, speech, and text information differ significantly, directly calculating attention makes it difficult to capture the true intermodal correspondences. Furthermore, traditional global attention mechanisms easily lead to an imbalance in intermodal attention, causing the model to overemphasize one modality while neglecting others. This invention utilizes a modular interactive attention mechanism and applies it to a hierarchical alignment architecture. This mechanism performs step-by-step alignment and fine-grained interaction on the local and global correlations between different modalities, mitigating semantic bias between modalities and ultimately achieving deep semantic alignment between the enhanced feature modalities. The implementation method is as follows:

[0100] 1) Construct a hierarchical alignment architecture to achieve deep alignment between non-linguistic modalities and text semantic space. Specifically, the architecture includes two parallel and interconnected processing paths: one is a deep semantic encoding path for text, and the other is an iterative cross-modal alignment path based on an interactive attention mechanism, which stacks three layers of interactive attention mechanism modules.

[0101] 2) The deep semantic encoding path employs a two-layer Transformer encoder to continuously deepen the text enhancement features, with each Transformer layer aiming to extract deeper levels of text semantic information.

[0102] 3) The iterative cross-modal alignment path executes an iterative alignment strategy through the three-layer interactive attention mechanism module: the first-layer interactive attention mechanism module uses the initial text enhancement features as a semantic benchmark to perform initial alignment of the audio enhancement features and video enhancement features; the second-layer interactive attention mechanism module uses the richer text enhancement features encoded by the first-layer Transformer as a new semantic benchmark to perform secondary correction and deepening of the initially aligned audio enhancement features and video enhancement features; the third-layer interactive attention mechanism module uses the deepest text enhancement features fully encoded by two layers of Transformer as a semantic benchmark to perform final fine alignment of the secondary corrected and deepened audio enhancement features and video enhancement features.

[0103] Through this layer-by-layer approach, audio and video features are progressively and stably mapped into a semantic space unified with the text while retaining their uniqueness, ultimately outputting deeply coupled and aligned text, audio, and video features.

[0104] As the core of this alignment architecture, the internal structure of the MIA module is as follows: Figure 5 As shown, MIA first addresses the feature space difference problem through modal interactive attention alignment, mapping video and audio tokens to the text modal space. Then, it solves the attention imbalance problem through a modular attention masking mechanism, designing self-modal and cross-modal attention strategies specifically optimized for feature fusion of video, audio, and text in emotion recognition tasks.

[0105] To reduce the semantic gap between video, audio, and text modalities, this invention proposes mapping video and audio tokens to the text modal space based on the embedding space basis vectors of the Gram matrix. Currently, the feature spaces of visual and audio information differ significantly from those of text information; directly utilizing this information for attention computation cannot capture the true correspondence between visual, audio, and text. Therefore, this method extracts basis vectors from the text modal space based on the token's Gram matrix, thereby compressing the semantics of the text modality and using it as a conversion medium for other modalities.

[0106] Specifically, the interactive attention mechanism module of this invention includes V-Aligner and A-Aligner, and its architecture is as follows: Figure 6 As shown, they are responsible for aligning the video and audio modalities to the text space, respectively. The implementation method is as follows:

[0107] Extracting the spatial basis of text modalities using the standardized Gram matrix The formula is:

[0108]

[0109] in, It is in the text token and The inner product between them is the element of the Gram matrix. This is the key state of the text modal token; the number of text tokens is... By including spatial basis vectors defined by the Gram matrix, the relationships between tokens within the text modality can be effectively captured, providing a unified semantic reference space for video and audio modalities.

[0110] Using the obtained standardized text Gram matrix as the spatial basis and as the cross-modal token transformation function, the calculated text tokens are first normalized and used as the basis for cross-modal mapping. Then, the video and audio modal tokens are transformed into the text space and used as the core mapping function. The aligned tokens are calculated as follows:

[0111]

[0112]

[0113] in, and These represent the key representations for the video and audio modalities, respectively. The mapped tokens are further fused with the original text tokens to enhance token similarity across the three modalities, achieving effective cross-modal feature alignment. Notably, the computation in the alignment stage maintains a linear complexity with the number of tokens.

[0114] Step 6: refine the feature associations within a modality through self-modal attention, and ensure deep fusion between different modalities through cross-modal attention, ultimately obtaining the deep features of each modality.

[0115] To enable the model to extract unique features from video, text, or audio, rather than focusing solely on the most important parts of the text, this step employs both self-modal attention and cross-modal attention mechanisms. The attention mask controls the flow of tokens between Transformer layers and introduces a positional bias into the multimodal sentiment analysis model. Its core function is to prevent any single modality (especially text) from excessively dominating attention allocation, ensuring that emotional cues such as facial expressions in videos and intonation variations in audio receive sufficient attention. To better accommodate the needs of token sequences from different modalities, this step assigns a modulated self-modal attention mask to each modality. and cross-modal attention mask Modality Self-attention is represented as:

[0116]

[0117] Modal Cross-modal attention is represented as:

[0118]

[0119] in, , representing text, audio, and video modalities respectively. Indicates and Different modes; For modal The query representation obtained by linear projection of the input token. For modal The key representation obtained by linear projection of the input token. For modal The value obtained by linear projection of the input Token is represented as follows: The attention scoring scaling term controls the scale of the score before Softmax, preventing numerical instability or excessive sharpness. It is composed of modes (i.e. with) Different modes, such as but The key representation obtained by linear projection of the input Token; It is composed of modes The value of the input Token is represented by a linear projection.

[0120] To effectively address the intermodal attention imbalance problem, this invention sets different masking strategies for different modal attention mechanisms (i.e., self-modal attention and cross-modal attention): a modular attention mask (i.e., a self-modal attention mask) is introduced for each. and cross-modal attention mask The core approach of this mask is to store unnecessary attention locations as pseudo-attention scores instead of directly setting them to negative infinity. The key role of this pseudo-attention score mechanism is to prevent the attention matrix from degenerating, avoid excessive smoothing of emotional features in deep networks, improve numerical stability, and alleviate attention collapse, thereby maintaining the uniqueness of emotional information of different modalities.

[0121] Each row in the attention score matrix represents the first... The attention score of each token, where the length of the sequence that the token can focus on is fixed. Therefore, each row contains A pseudo-attention score, These pseudo-attention scores are assigned to redundant values ​​based on the sequence length that the token can focus on. The attention score matrix uses a decay rate. The masking strategy, in which the real scores are retained for the positions of interest. Enter pseudo-attention scores in the unattentional positions. The formal expression is as follows:

[0122]

[0123]

[0124] in, This refers to the attention score matrix corrected using a "window + decay pseudo-score" strategy. Representing the Line number The pseudo-attention score of the column is used for placeholder and suppression of positions that are not payable or outside the window. The decay rate is used to control how quickly the pseudo-attention score decreases with the column index, for example, When greater than 0, Will follow The value decreases as it increases, thus more strongly suppressing the distal or out-of-window position.

[0125] In addition to absolute positional prior information, modal position can be further introduced to force the model to correct the token stream. This positional information is particularly important—for example, emotional climaxes in videos often occur within specific time periods, and emotional turning points in audio also have temporal characteristics; masks capture these temporal emotional patterns by introducing modality-specific positional encoding. Specifically, such as... Figure 7 As shown, Video Coord and Audio Coord indicators are constructed simultaneously during the key sequence concatenation of cross-modal attention. Learnable positive biases and mask corrections (including fixed window and pseudo-score decay) are applied to the scoring matrix before softmax through the indicators, thereby explicitly incorporating the modal position prior into the attention calculation. Subsequently, if necessary, indicator-driven bucketing normalization is performed on the weights after softmax to further stabilize the attention to video and audio positions.

[0126] Specifically, in cross-modal attention, the target key sequence is concatenated along the column dimension according to modality, and Video Coord and Audio Coord are used as position indicators to label the columns where the video and audio keys are located. These two indicators are aligned one-to-one with the columns of the attention score matrix to inject modal position priors into the model.

[0127] In conclusion, by Self-modal attention focuses on refining relationships within the same modality, ensuring that relevant information from video, audio, or text is effectively propagated between layers. Conversely, by... The cross-modal attention representation facilitates information exchange between different modalities, enabling the model to perform comprehensive sentiment analysis using visual cues from videos, acoustic features from audio, and semantic information from text, thus achieving more accurate multimodal sentiment recognition.

[0128] Through this modular design, the method can effectively handle the complementarity and consistency of emotional expression in video, audio and text, providing a more accurate and robust feature fusion mechanism for multimodal sentiment analysis.

[0129] Step 7: Use a multi-head self-attention mechanism to allow the deep audio features and deep video features to interact fully, obtaining non-text features. Then, inject the non-text features into the deep text features through a weighted shift operation to form a joint multimodal sentiment representation. Finally, use the joint multimodal sentiment representation to achieve sentiment classification.

[0130] After processing with multimodal large model semantic enhancement and modular interactive attention mechanism (MIA), the three modalities of text, video and audio have achieved good semantic alignment and attention balance. Among them, video and audio provide information that may not be captured from text.

[0131] However, a core issue remains in the final sentiment prediction stage: how to fully leverage the complementary role of the enhanced nonverbal modality to effectively support and correct the sentiment judgment of the text modality. Current multimodal sentiment analysis often employs a Transformer-based cross-modal fusion structure, placing greater emphasis on similarity matching between different modalities, often neglecting the unique emotional details contained in the nonverbal modality. This results in the model failing to capture the nuanced information within the nonverbal modality.

[0132] Therefore, to address the imbalance in contributions from strong and weak modalities in multimodal emotion fusion, this invention proposes an attention-based modality shifting fusion mechanism. Specifically, this method first concatenates deep audio and video features, then performs deep fusion using a multi-head self-attention mechanism to extract key non-verbal emotional information, resulting in the non-textual features. Subsequently, a gated shift vector generation mechanism is constructed, which adaptively filters and modulates relevant emotional details in the non-verbal modality based on the textual semantic context, generating targeted shift adjustment signals. Finally, a weighted shift operation is used to integrate the non-textual features into the textual emotion representation, and an adaptive scaling mechanism ensures the stability of the fusion process.

[0133] This method transforms MIA-enhanced nonverbal modal features into displacement vectors. Through a gated displacement mechanism, these MIA-enhanced features are converted into precise adaptive adjustment signals. This "displacement supplementation" rather than "feature replacement" approach finely refines and progressively optimizes the text sentiment representation. This design preserves the original semantics of the text modality while fully leveraging the supplementary information from the enhanced nonverbal modality, achieving synergistic enhancement between strong and weak modalities. This refined fusion significantly improves the model's ability to capture emotional details, thereby enhancing the accuracy and robustness of Mongolian multimodal sentiment analysis.

[0134] Specifically, such as Figure 8 As shown, the attention fusion of non-textual features is first implemented as follows:

[0135] Video features after MIA processing and audio features It contains rich non-verbal emotional information. To effectively extract and integrate this information, the features from the two modalities are first concatenated, and then deep fusion is performed using a multi-head self-attention mechanism.

[0136]

[0137] in, and These represent the third aligned video and audio features, respectively. For feature dimension, The fused nonverbal sentiment representation vector contains integrated nonverbal sentiment information from both video and audio modalities. The multi-head self-attention mechanism captures the intrinsic connections between the video and audio modalities, highlighting the non-textual feature combinations most crucial for sentiment recognition.

[0138] Secondly, to achieve accurate modulation of text representation by non-textual features, this invention designs a gated displacement vector generation mechanism. This mechanism can adaptively filter and modulate relevant emotional details in non-linguistic modalities based on the semantic context of the text. The adaptive gated displacement vector generation is implemented as follows:

[0139] By splicing deep text features Non-textual features Generate gate vector :

[0140]

[0141] in, This represents a vector concatenation operation. and These are the weight matrix and the bias vector, respectively. This is the activation function.

[0142] The purpose of gating vectors is to highlight relevant information in non-linguistic vectors based on the text representation, achieving selective attention. A displacement vector is generated using the gating vector. :

[0143]

[0144] in, This indicates element-wise multiplication. and These are learnable parameters. The displacement vector. Encodes key non-linguistic sentiment information filtered based on textual semantic context.

[0145] Finally, adaptive displacement fusion is performed, and the method is as follows:

[0146] Non-text features are incorporated into deep text features through weighted shift operations. In this process, a joint multimodal sentiment representation is obtained. :

[0147]

[0148] in, This is the adaptive scaling factor, used to ensure the stability of the fusion process. The formula is:

[0149]

[0150] in, express Norm, This is the threshold hyperparameter. This scaling mechanism ensures that the magnitude of the translation operation is moderate, avoiding excessive interference of non-textual features with text semantics, while also guaranteeing the numerical stability of the fusion process.

[0151] Through this refined displacement fusion strategy, the model can fully utilize non-linguistic modal information enhanced by MIA while maintaining the original semantic integrity of the text modality, thereby achieving synergistic enhancement between strong and weak modalities.

[0152] This step first fuses the enhanced audio and video features into a unified audiovisual representation, and then performs shift fusion with the text features to explicitly model the potential misalignment between the text and audiovisual modalities in terms of time and semantic focus, achieving more robust cross-modal collaboration. Finally, the fused multimodal comprehensive representation is input into a fully connected layer to complete the sentiment category determination.

Claims

1. A Mongolian multimodal sentiment analysis method based on cross-modal information enhancement and fusion, characterized in that, Includes the following steps: Step 1: Preprocess the raw data of text, audio, and video modalities to extract initial features for each modality, and then use a Transformer encoder to obtain shallow feature sequences for each modality. Simultaneously, through the speech description generation part based on a large language model, obtain shallow feature sequence one regarding emotional cues in the audio modality from the preprocessed audio data, and concatenate it with the audio shallow feature sequence to obtain audio enhancement features. Through the video description generation part based on a large language model, obtain shallow feature sequence two regarding emotional cues in the video modality from the preprocessed video data, and concatenate it with the video shallow feature sequence to obtain video enhancement features. Finally, concatenate the text shallow feature sequence with shallow feature sequence one and shallow feature sequence two to obtain text enhancement features. Step 2: A modular interactive attention mechanism is used to achieve deep semantic alignment between various enhanced feature modalities. Self-modal attention is used to refine the feature associations within a modality, and cross-modal attention is used to ensure deep fusion between different modalities, ultimately obtaining the deep features of each modality. Step 3: First, use a multi-head self-attention mechanism to enable the deep features of audio and video to interact fully to obtain non-text features. Then, inject the non-text features into the deep text features through a weighted shift operation to form a joint multimodal sentiment representation. Finally, use the joint multimodal sentiment representation to achieve sentiment classification. The method for achieving deep semantic alignment between various enhanced feature modalities using a modular interactive attention mechanism is as follows: A hierarchical alignment architecture is constructed, which contains two parallel and interconnected processing paths: one is a deep semantic encoding path for text, and the other is an iterative cross-modal alignment path based on an interactive attention mechanism, which stacks three layers of interactive attention mechanism modules. The deep semantic encoding path employs a two-layer Transformer encoder to continuously deepen the text enhancement features; The iterative cross-modal alignment path executes an iterative alignment strategy through the three-layer interactive attention mechanism module: the first-layer interactive attention mechanism module uses the initial text enhancement features as a semantic benchmark to perform initial alignment of the audio enhancement features and video enhancement features; the second-layer interactive attention mechanism module uses the text enhancement features encoded by the first layer of Transformer as a semantic benchmark to perform secondary correction and refinement of the initially aligned audio enhancement features and video enhancement features; the third-layer interactive attention mechanism module uses the text enhancement features encoded by two layers of Transformer as a semantic benchmark to perform final refined alignment of the secondary corrected and refined audio enhancement features and video enhancement features.

2. The Mongolian multimodal sentiment analysis method based on cross-modal information enhancement and fusion according to claim 1, characterized in that, In step 1, initial text features are extracted using RoBERTa, initial audio features are extracted using LibROSA, initial video features are extracted using OpenFace, and then the corresponding shallow feature sequences are obtained using a Transformer encoder.

3. The Mongolian multimodal sentiment analysis method based on cross-modal information enhancement and fusion according to claim 1, characterized in that, Step 1, through the speech description generation part based on a large language model, obtains a shallow feature sequence of audio modality sentiment cues from the preprocessed audio data. The implementation method is as follows: The preprocessed audio data and corresponding prompts are then fed into Qwen-Audio. Qwen-Audio encodes the cue words, obtains the Log-Mel spectrum from the audio data through resampling and short-time Fourier transform, and concatenates it with the encoded features; The Qwen large model inside Qwen-Audio uses the spliced ​​features to generate a structural text description of the audio. The structural textual description of the audio is processed by the BERT encoder to output a token sequence, which is the first shallow feature sequence of the emotional cues of the audio modality.

4. The Mongolian multimodal sentiment analysis method based on cross-modal information enhancement and fusion according to claim 1, characterized in that, Step 1, through the video description generation part based on a large language model, obtains a shallow feature sequence 2 regarding video modality sentiment cues from the preprocessed video data. The implementation method is as follows: The preprocessed video data and corresponding prompts are then fed into Qwen-2.5-VL. Qwen-2.5-VL encodes prompts, extracts image features from images obtained by extracting frames from video data using ViT, and concatenates them with the encoded features; The Qwen large model inside Qwen-2.5-VL is used to generate structural text descriptions of the video by using the spliced ​​features. The structural text description of the video is processed by the BERT encoder to output a token sequence, which is the second shallow feature sequence about the emotional cues of the video modality.

5. The Mongolian multimodal sentiment analysis method based on cross-modal information enhancement and fusion according to claim 1, characterized in that, The interactive attention mechanism module includes V-Aligner and A-Aligner, which are responsible for aligning video and audio modalities to the text space, respectively. The implementation method is as follows: Extracting the spatial basis of text modalities using the standardized Gram matrix The formula is: in, It is in the text token and The inner product between them is the element of the Gram matrix. This is the key state of the text modal token; the number of text tokens is... ; Using the obtained standardized text Gram matrix as the spatial basis, and using it as the cross-modal token transformation function, the calculated text tokens are first normalized and used as the basis for cross-modal mapping. Then, the video and audio modal tokens are transformed into the text space. The aligned tokens are calculated as follows: in, and The key represents the video modality and the audio modality, respectively.

6. The Mongolian multimodal sentiment analysis method based on cross-modal information enhancement and fusion according to claim 1, characterized in that, Step 2 involves refining the feature associations within a modality through self-modal attention and ensuring deep fusion between different modalities through cross-modal attention, ultimately obtaining the deep features of each modality. The implementation method is as follows: Assign a modulated self-modal attention mask to each mode. and cross-modal attention mask Modality Self-attention is represented as: Modal Cross-modal attention is represented as: in, , representing text, audio, and video modalities respectively. Indicates and Different modes; For modal The query representation obtained by linear projection of the input token. For modal The key representation obtained by linear projection of the input token. For modal The value obtained by linear projection of the input Token is represented as follows: The attention scoring scaling term controls the scale of the score before Softmax, preventing numerical instability or excessive sharpness. It is composed of modes The key representation obtained by linear projection of the input Token; It is composed of modes The value of the input Token is represented by a linear projection.

7. The Mongolian multimodal sentiment analysis method based on cross-modal information enhancement and fusion according to claim 6, characterized in that, Introduce self-modal attention masks for self-modal attention and cross-modal attention respectively. and cross-modal attention mask Each row in the attention score matrix represents the first row. Attention scores for each token, with each row containing... A pseudo-attention score, The pseudo-attention score is assigned to redundant values ​​based on the sequence length that the token can focus on; the attention score matrix uses a decay rate. The masking strategy, in which the real scores are retained for the positions of interest. Enter pseudo-attention scores in the unattentional positions. The formal expression is as follows: in, Refers to the corrected attention score matrix. Representing the Line number The pseudo-attention score of the column is used for placeholder and suppression of positions that are not payable or outside the window. The decay rate is used to control how quickly the pseudo-attention score decreases with the column index. Video Coord and Audio Coord indicators are constructed simultaneously during the key sequence concatenation of cross-modal attention. Learnable positive biases and mask corrections are applied to the scoring matrix before softmax using the indicators, thereby explicitly incorporating the modal position prior into the attention calculation. Subsequently, indicator-driven bucketing normalization is performed on the weights after softmax to further stabilize the attention to video and audio positions.

8. The Mongolian multimodal sentiment analysis method based on cross-modal information enhancement and fusion according to claim 1, characterized in that, Step 3 uses a multi-head self-attention mechanism to enable full interaction between deep audio features and deep video features to obtain non-text features. The implementation method is as follows: The deep features of the audio and video are concatenated and then deeply fused using a multi-head self-attention mechanism.

9. The Mongolian multimodal sentiment analysis method based on cross-modal information enhancement and fusion according to claim 1, characterized in that, Step 3 involves injecting non-textual features into deep textual features through a weighted shift operation to form a joint multimodal sentiment representation. The implementation method is as follows: By splicing deep text features Non-textual features Generate gate vector : in, This represents a vector concatenation operation. and These are the weight matrix and the bias vector, respectively. For activation functions; A displacement vector is generated using the gate vector. : in, This indicates element-wise multiplication. and These are learnable parameters; Non-text features are incorporated into deep text features through weighted shift operations. In this process, a joint multimodal sentiment representation is obtained. : in, The adaptive scaling factor is given by the following formula: in, express Norm, This is the threshold hyperparameter.

Images