Mongolian multi-modal sentiment analysis method based on dual-state space and multi-path transpose attention

Abstract

A Mongolian multi-modal sentiment analysis method based on a dual-state space and multipath transpose attention comprises the following steps: respectively extracting initial text features, initial audio features and initial video features from an original multi-modal data set, and coding through a shared encoder to obtain similar subspace modal uncertain features; encoding through a corresponding modal specific encoder to obtain specific subspace modal features; according to the method, advanced three-mode fusion indefinite features are obtained through Mama feature extraction and MPTT local transposition attention filtering; the method comprises the following steps of: fusing the similar subspace modal uncertain features with specific subspace modal features, obtaining advanced bimodal fusion specific features through local transposition attention filtering of MPTT, splicing the advanced bimodal fusion specific features with advanced three-modal fusion uncertain features to obtain a combined multi-modal representation, and realizing multi-modal sentiment classification through a full connection layer. According to the method, the problems of obvious heterogeneous distribution between modals and insufficient semantic mining are effectively relieved, and the performance and generalization ability of Mongolian sentiment analysis are improved.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology and relates to multimodal sentiment analysis combining text, audio and video, and in particular to a Mongolian multimodal sentiment analysis method based on dual-state space and multi-path transposed attention. Background Technology

[0002] Currently, multimodal sentiment analysis faces many challenges. Compared with English and Chinese, the techniques for sentiment research in Mongolian are still lacking. This is mainly due to two reasons: firstly, Mongolian has a unique and complex word formation system with diverse word forms, which brings natural difficulties to language processing; secondly, the construction of relevant corpora is relatively lagging behind, and the scarcity of data resources limits the in-depth development of research. Summary of the Invention

[0003] In order to overcome the shortcomings of the prior art, the present invention aims to provide a Mongolian multimodal sentiment analysis method based on dual-state space and multi-path transposed attention, which mainly solves the problem of limited prediction effect caused by information loss and insufficient fusion of different modal features.

[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A Mongolian multimodal sentiment analysis method based on dual-state space and multi-path transposed attention includes the following steps: Step 1: Extract initial text features, initial audio features, and initial video features from the original multimodal dataset. Step 2: Encode the initial text features, initial audio features, and initial video features using a shared encoder to obtain similar subspace modal indeterminate features, and then encode them using a corresponding modality-specific encoder to obtain specific subspace modal features; Step 3: The modal indeterminate features of the similar subspace are extracted using Mamba features and filtered by local transpose attention of the optimized Transformer encoder based on multipath transpose attention to obtain advanced trimodal fusion indeterminate features; Step 4: Complementarily fuse the modal indeterminate features of the similar subspace with the modal features of the specific subspace, and then obtain the advanced dual-modal fusion specific features by filtering the local transpose attention of the optimized Transformer encoder based on multi-path transpose attention. Step 5: The advanced trimodal fusion indeterminate features and the advanced bimodal fusion specific features are concatenated to obtain a joint multimodal representation, and finally multimodal sentiment classification is achieved through a fully connected layer.

[0005] In one embodiment, step 1 involves extracting text features using BERT and then eliminating bias factors using a text debiasing module to obtain the initial text features. The text debiasing module is implemented based on a global dictionary and a two-branch Transformer network. In the two-branch Transformer network, one branch performs self-attention calculation on the input text features to obtain an estimate of the knowledge within the sample; the other branch uses the input text features as a query vector and performs cross-attention calculation using the global dictionary as a key and value vector to obtain an estimate of the knowledge across samples.

[0006] In one embodiment, a deep adaptive clustering algorithm is used across the entire training set. The global dictionary is initialized on the text modality as follows: in, Indicates the global dictionary. This represents a deep adaptive clustering algorithm. This represents the average pooling operation. This represents the Transformer operation. This represents the number of clusters in the text feature clustering.

[0007] In one embodiment, the dual-branch Transformer network consists of L stacked Transformer blocks. Each Transformer block contains two branches. One branch initializes the input text features through embedding, captures semantic dependencies through self-attention calculation, and then uses a feedforward neural network to perform nonlinear transformation and dimension mapping to obtain transformed features. The other branch constructs the association link between the current sample and the global semantic pattern through cross-attention calculation, and then uses a feedforward neural network to perform nonlinear transformation and dimension mapping to obtain transformed features. The transformed features obtained from the two branches are merged and used as the dual-branch input of the next Transformer block. The output of the last Transformer block is the initial text features.

[0008] In one embodiment, step 1 involves extracting audio features using Librosa and then obtaining the initial audio features using a temporal LSTM encoder; and extracting video features using OpenFace and then obtaining the initial video features using a temporal LSTM encoder.

[0009] In one embodiment, step 2, which encodes the initial text features, initial audio features, and initial video features using a shared encoder to obtain similar subspace modality-indeterminate features, is implemented as follows: Using the initial text features, initial audio features, and initial video features as input, a lightweight structure employing fully connected layers (FCLayer) and non-linear activations (such as ReLU and Sigmoid) is adopted, combined with a self-attention mechanism to enhance the global correlation of features. Contrastive learning loss is used for training: to maximize the distance between semantically related cross-modal pairs (such as audio and text of the same event) in the shared space, and to maximize the distance between semantically unrelated cross-modal pairs. The final output is a feature vector mapped to the shared space, i.e., the modality-indeterminate features of the similar subspace. Simultaneously, classification loss, reconstruction loss, etc., can be used as auxiliary methods to enhance the discriminativeness and robustness of the features.

[0010] In one embodiment, the optimized Transformer encoder based on multi-path transposed attention embeds a one-dimensional convolution operation into the multi-head attention mechanism of the traditional Transformer encoder to adjust the attention calculation. The local transposed attention filtering is implemented as follows: First, the data is used as the input of the encoder through position encoding and is divided into three branches: query, key and value. Each branch first goes through a linear transformation layer to perform preliminary dimension adaptation. Then, the linear transformation results of the query branch and the key branch are respectively entered into a one-dimensional convolutional layer to perform preliminary capture and dimension adjustment of local features. Subsequently, the transposed attention layer is entered. Using the local attention mechanism, the similarity between features at different positions and different modalities is calculated based on the vector processed by the one-dimensional convolutional layer. The transposed attention weights are obtained by processing with the softmax function. Then, the transposed attention weights and the linear transformation result of the value branch are subjected to a dot product operation to obtain the weighted features. Finally, the weighted features are added to the original values ​​to obtain the encoder output.

[0011] In one embodiment, step 4 involves fusing the modal features of similar subspaces with those of specific subspaces using a modal feature complementarity fusion module. This module concatenates the similarity and specificity representations of each modality along the feature dimensions, and then uses a feedforward neural layer to achieve the fusion of complementary information for a single modality, as shown below: in For a unified representation that includes modality-specific information and invariant information, This represents the activation function. This indicates a splicing operation. This is a specific representation of the modality. For modal similarity representation, These refer to text, audio, and video modalities, respectively.

[0012] In one embodiment, the high-level bimodal fusion specific features obtained by local transpose attention filtering of the optimized Transformer encoder based on multipath transpose attention are expressed as follows: in, For specific features of text-audio bimodal fusion, For text-video bimodal fusion specific features, This represents a cross-modal interaction function, used to achieve deep-level exchange and fusion of information between different modalities. , , These are unified representations that contain modal-specific information and invariant information, respectively, for audio, text, and video modes. These are learnable parameters in the audio-text interaction process. These are learnable parameters in the visual-text interaction process; for ,by For query, Calculate the similarity matrix between the text and audio modal for the key. : for ,by For query, Calculate the similarity matrix between the text and video modalities for the key. : in and It is a learnable attention matrix. It is the sum of the dimensions of each attention head.

[0013] In one embodiment, step 5 involves concatenating the initial text features, initial audio features, initial video features, and advanced trimodal fusion indeterminate features and advanced bimodal fusion specific features together to obtain the joint multimodal representation through a cross-modal Transformer and a cross-modal attention mechanism.

[0014] Compared with existing technologies, the Mongolian multimodal sentiment analysis method based on dual-state space and multi-path transposed attention has the following beneficial effects: (1) By constructing an optimized Transformer encoder based on multi-path transposed attention, the deep interaction between different modal features is enhanced, which helps the model to better identify the features of different modalities; (2) By constructing a modal feature complementary fusion module based on dual-state space, the problem of significant heterogeneous distribution between modalities and insufficient semantic mining is effectively alleviated, thereby improving the performance and generalization ability of Mongolian sentiment analysis.

[0015] Furthermore, in the optimization method of this invention, a text debiasing module based on a two-branch Transformer is constructed to reduce spurious correlations of confounding variables, which effectively alleviates spurious correlations of text modalities in multimodal sentiment analysis and improves the accuracy of model prediction. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the overall method flow of the present invention.

[0017] Figure 2 This is a schematic diagram of the dual-branch Transformer structure of the present invention.

[0018] Figure 3 This is a schematic diagram of the similar subspace feature fusion of the present invention.

[0019] Figure 4 This is a schematic diagram of the dual-state space of the present invention.

[0020] Figure 5 This is a schematic diagram of the optimized Transformer model structure of the present invention.

[0021] Figure 6 This is a schematic diagram of the attention required for this invention.

[0022] Figure 7 This is a schematic diagram of the complementary fusion module (MFRF) of the present invention.

[0023] Figure 8 This is a schematic diagram of the feature fusion of the present invention.

[0024] Figure 9 This is a schematic diagram of the ablation experiment results of the present invention. Detailed Implementation

[0025] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings and examples.

[0026] Multimodal sentiment analysis overcomes the limitations of single-modal expression by integrating multimodal information such as speech, text, images, and videos. It leverages deep learning technology to capture emotional connections between modalities and accurately identify emotional states. However, compared to some mainstream languages ​​such as Chinese and English, Mongolian, as a less commonly spoken language, still requires further research progress.

[0027] Meanwhile, in multimodal sentiment analysis, modal heterogeneity stems from the fundamental differences between different modal data: text carries semantics through discrete symbol sequences, images rely on continuous pixel matrices to convey visual information, and audio expresses emotion through waveform signals. This significant heterogeneity in distribution makes direct fusion prone to feature conflicts and information loss. The shortcomings of speech mining are manifested in the lack of exploration of deep cross-modal associations: existing methods often stop at surface feature matching, and this shallow matching is insufficient to capture implicit semantic connections between modalities, severely limiting the understanding and reasoning capabilities of multimodal models.

[0028] Therefore, this invention aims to construct a Mongolian multimodal sentiment analysis method based on dual-state space and multi-path transposed attention. By introducing a dual-state space mechanism, dynamic interaction and complementarity of modal features in the hidden layer are achieved, thereby enhancing the model's ability to capture cross-modal semantic associations. In the specific implementation, the Mamba structure is used to extract sequence dependencies within a modality, while the optimized Transformer is responsible for the efficient alignment and fusion of cross-modal information. Combining the advantages of common and specific modal features, a joint representation with strong expressive power is formed.

[0029] Furthermore, in multimodal models, insufficient feature interaction manifests as superficial splicing of features from different modalities, lacking a deep fusion mechanism. Some implicit complementary information fails to be mined, making it difficult for the model to capture cross-modal semantic resonance. False textual relevance stems from statistical bias in text modality; the high-frequency co-occurrence of certain words and sentiment tags is misjudged by the model as a necessary rule, ignoring irony or scene limitations in specific contexts. This over-reliance on superficial associations, coupled with insufficient intermodal interaction, ultimately leads to frequent predictions that deviate from the true intent in complex scenarios. This invention constructs an optimized Transformer encoder based on multi-path transposed attention and a text debiasing module based on a two-branch Transformer to reduce spurious relevance of confounding variables. The MPTT model achieves dynamic alignment and deep interaction of multimodal features at different semantic levels through fine-tuning of local attention mechanisms. It breaks down the semantic gap between heterogeneous modalities by leveraging multi-path transposed optimization strategies, enabling the fusion of cross-modal information to leap from superficial splicing to organic symbiosis of intrinsic semantics. The MTDM model, relying on the collaborative modeling capabilities of the two-branch Transformer architecture, accurately removes confounding variables from text modalities that are irrelevant to the essence of emotion. Through targeted elimination of spurious relevance and reinforcement learning of the semantic kernel, it allows the emotional expression of text features to return to its true logic. The two form a closed-loop collaborative mechanism of "deep fusion - precise debiasing".

[0030] refer to Figure 1 As shown, the main steps of this invention are as follows: Step 1: Obtain the original multimodal dataset and extract the features of each modality.

[0031] In this invention, the modalities in the multimodal dataset include text, audio, and video. For ease of formula description, this invention adopts... Unified representation of modality, and with These refer to text, audio, and video modalities, respectively.

[0032] Specifically, this invention extracts text features using BERT, audio features using Librosa, and video features using OpenFace, with video features primarily consisting of facial features. The extracted features are low-level features.

[0033] BERT is a pre-trained model for processing spoken text, including Mongolian language recognition. When processing Mongolian text, BERT first breaks down Mongolian sentences into appropriate words or segments, adding specific start and delimiter symbols. Next, it generates word vectors unique to Mongolian, and then combines these with positional information to form an input format that the model can understand. The feature vectors generated after processing by the BERT model effectively express the meaning of the text.

[0034] For audio file input, the Librosa library first reads the Mongolian audio file, converts it into a digital signal, and acquires basic information such as the sampling rate. Next, the waveform data is preprocessed. Then, core feature extraction begins: first, a spectrogram is obtained through Short-Time Fourier Transform (STFT), then a Mel spectrogram is derived, and Mel frequency cepstral coefficients (MFCCs) are extracted from the spectrum. Following this, various features such as zero-crossing rate, spectral centroid, and tempo are calculated. Finally, the Mongolian feature vectors are organized into a numerical matrix for use in training subsequent tasks.

[0035] For video file input, OpenCV is first used to read the frames in the video, and keyframes are selected as image materials for feature extraction. Then, the OpenFace tool is called to perform face detection and feature extraction: when the system detects a face region, it automatically extracts multiple facial feature information, including the coordinate data of key facial points such as the eyes and nose, as well as facial expression features that can reflect emotional state.

[0036] Step 2: Further extract the initial features of each modality.

[0037] Since the features obtained in step 1 have different dimensions, a unidirectional long short-term memory network (LSTM) and a fully connected layer (FC) can be used to enable each token in the sequence to capture useful contextual information and map it to a fixed input. That is, the audio features and video features obtained in step 1 are respectively passed through a temporal LSTM encoder to obtain the initial audio features and initial video features.

[0038] In multimodal sentiment analysis, textual modalities are prone to spurious correlations. Some words frequently co-occur with positive sentiment, but may actually express negative sentiment in a specific context. If the model only learns the surface co-occurrence patterns without understanding the semantics, it will misjudge the original meaning. This error stems from textual data bias or labeling errors, causing the model to ignore the true sentiment logic and reduce prediction accuracy. Therefore, spurious correlations in textual modalities can lead to inaccurate predictions. Traditional text models typically predict by learning the statistical correlation between textual features and labels, but this correlation may contain "spurious associations." The source of these spurious associations is data bias in the training data. For example, the word "movie" might appear frequently in negative samples in the training set, causing the model to incorrectly associate these words with "negative sentiment," even though these words themselves have no sentimental meaning. The model will rely on these spurious associations for prediction, resulting in poor generalization ability on the test set. When "movie" appears in a positive context (such as "good movie"), traditional models may still misjudge it as negative due to the spurious association of "movie."

[0039] Therefore, this invention further constructs a Mongolian Text De-biasing Model (MTDM) based on a two-branch Transformer to reduce spurious correlations caused by confounding variables. This text de-biasing module is implemented based on a global dictionary and a two-branch Transformer network. Its function is to eliminate bias factors from the input text features to mitigate spurious correlations caused by text data bias and confounding factors, thereby obtaining initial text features.

[0040] Building a text debiasing module requires a deep learning framework. The most direct approach is to parameterize the module using the softmax function to estimate probabilities. This invention first employs a parametric design to facilitate the differentiation of accurate values ​​between In-Sample Knowledge Selection (ISSKS) and Cross-Sample Knowledge Selection (CSKS). ISKS extracts relevant information only from the current input sample, while CSKS estimates bias and recalibrates by combining the current sample with other samples in the training set.

[0041] Based on this, the method for constructing the text bias removal module in this invention is as follows: First, a global dictionary is initialized on the Mongolian text modalities across the entire training set using a deep adaptive clustering algorithm (DAC). , Next, design a two-branch Transformer network, such as... Figure 2 As shown, the algorithm is divided into a right-hand in-sample knowledge branch and a left-hand cross-sample knowledge branch, which respectively handle "true association within the current sample" and "global cross-sample bias calibration". One branch performs self-attention calculation on the input text features to obtain an estimate of the in-sample knowledge; the other branch uses the input text features as a query vector and employs a global dictionary. Cross-attention is performed as key and value vectors to obtain estimates across sample knowledge branches, where... Indicates the global dictionary. This represents a deep adaptive clustering algorithm. This represents the average pooling operation. This represents the Transformer operation. This represents the number of clusters in the text feature clustering.

[0042] More preferably, the dual-branch Transformer network is composed of L stacked Transformer blocks, each containing two branches. One branch initializes the input text features through embedding, then captures semantic dependencies through self-attention calculation, and subsequently uses a feedforward neural network to perform nonlinear transformation and dimensionality mapping to obtain transformed features. The other branch constructs the association link between the current sample and the global semantic pattern through cross-attention calculation, and then uses a feedforward neural network to perform nonlinear transformation and dimensionality mapping to obtain transformed features. The transformed features obtained from the two branches are merged and used as the dual-branch input of the next Transformer block. The output of the last Transformer block is the initial text features.

[0043] More specifically, for in-sample knowledge branches: Features of input text Embedding initialization is performed, and the internal correlation structure of the sample is simultaneously mined to extract core clues directly related to sentiment from the feature dimension. Based on the initialized text features, a self-attention mechanism is introduced for modeling: by calculating the attention weight distribution between each token and all other tokens in the sample, the complex dependencies at the semantic level of the text are captured in a dynamically adapted manner, achieving differentiated representation of the correlation strength between tokens.

[0044] in, This indicates the self-attention mechanism. These represent the query matrix, key matrix, and value matrix, respectively. This represents the similarity score between the query matrix and the key matrix. This represents the scaling factor, mitigating the vanishing gradient problem. This represents the activation function.

[0045] The output feature vector of the self-attention mechanism will be fed into the feedforward neural network, and through multi-layer weight mapping and non-linear activation functions, the non-linear transformation and dimension mapping of the features will be completed.

[0046] in This is the weight matrix of the fully connected layer. For bias terms, For the hidden layer dimension of the feedforward network, It is a non-linear activation function, and the final output is the transformed feature. .

[0047] Within a single Transformer block, residual connections and layer normalization modules are integrated synchronously: residual connections construct a "shortcut path" for feature propagation, allowing gradients to directly penetrate shallow layers and propagate to deeper layers; layer normalization standardizes the mean and variance for the feature dimensions of each sample, unifying the feature distribution scale.

[0048] After the self-attention module: in, Presentation layer normalization operation, This represents the initialization result of text feature embedding. This represents the initialization result of text feature embedding.

[0049] After the feedforward network module: in, For layer normalization operation, The output features of the self-attention module after layer normalization are... This is a feedforward neural network layer.

[0050] After layer normalization: in, This represents the input features to be normalized. It is input The mean, calculated by feature dimension. Indicates input The variance, calculated by feature dimension, , These are learnable scaling and offset parameters.

[0051] Because of the L-layer stacking, text features can be processed iteratively layer by layer: the self-attention module of each layer continuously optimizes the accuracy of capturing semantic associations between tokens, and the feedforward network simultaneously performs nonlinear reconstruction of features. Under the synergistic effect of the two, the representation strength of sentiment-related semantic patterns is gradually enhanced, while the interference of neutral information and noise components is weakened.

[0052] For cross-sample knowledge branches: Input text features and global dictionary features, designating the text features as query vectors and the global dictionary features as key and value vectors respectively. A cross-attention mechanism is used to construct the association link between the current sample and the global semantic pattern. Specifically, the cross-attention mechanism calculates the similarity score between each token in the text features and all tokens in the global dictionary features, and generates an attention weight matrix after normalization, thereby quantifying the association strength between the current sample token and the global semantic unit. Based on this weight matrix, the value vectors of the global dictionary features are weighted and aggregated to obtain the cross-attention output features. The similarity score matrix is ​​then Softmax normalized to generate the attention weight matrix. And ensure that the sum of the weights is 1. Wherein, It is the activation function that normalizes the similarity score matrix, where S is the similarity score matrix. This represents the exponential operation on the elements in the similarity score matrix and is the numerator of the Softmax function.

[0053] The output features of cross-attention are fed into a feedforward network (FFN) to perform a nonlinear transformation. This feedforward network consists of two fully connected layers and an intermediate ReLU activation function. The first fully connected layer projects the input features to a higher-dimensional space, introducing nonlinear mapping capabilities through the ReLU activation function. The second fully connected layer then maps the high-dimensional features back to the original dimension, completing the abstract reconstruction and enhanced representation of the features. The structure integrating cross-attention and the feedforward network is defined as a single branch.

[0054] Similarly, due to the L-layer stacking, iterative processing can be performed: each branch is based on the features output by the previous layer, repeating the cross-attention association modeling and feedforward network nonlinear transformation process to achieve layer-by-layer optimization of text features.

[0055] After such Transformer block iterative processing, the final output cross-sample knowledge representation will simultaneously retain the semantic information of the current sample and filter out spurious relevance in the local data.

[0056] Step 3: Encode the initial text features, initial audio features, and initial video features using a shared encoder to obtain similar subspace modal indeterminate features.

[0057] In this step, initial text features, initial audio features, and initial video features are used as input. A lightweight structure with fully connected layers (FC layers) and non-linear activations (such as ReLU and Sigmoid) is employed, combined with a self-attention mechanism to enhance the global correlation of features. Contrastive learning loss is used for training: to ensure that semantically related cross-modal pairs (such as audio and text of the same event) are as close as possible in the shared space, and semantically unrelated cross-modal pairs are as far apart as possible. The final output is a feature vector mapped to the shared space, i.e., the modality-indeterminate features of the similar subspace. Simultaneously, classification loss, reconstruction loss, etc., can be used as auxiliary methods to enhance the discriminativeness and robustness of the features.

[0058] Step 4: Encode the initial text features, initial audio features, and initial video features using the corresponding modality-specific encoders to obtain specific subspace modality features.

[0059] In this step, the initial text features, initial audio features, and initial video features are encoded using a text encoder, an audio encoder, and a video encoder, respectively, to obtain the corresponding modal features.

[0060] like Figure 3 As shown, after data preprocessing, this invention forms two specific state spaces: a similarity subspace and a feature subspace. High-level features are extracted from the modality-indeterminate features in the similarity subspace, utilizing common information across different modalities to reduce noise and improve performance. Specifically, feature representation learning of the dual-state space is as follows: Figure 4 As shown, the feature vectors in the similarity subspace represent common features across different modalities. This reduces modal differences and highlights intermodal consistency, facilitating smooth and effective fusion of different modalities. The feature vectors in the feature subspace are modality-specific representations, focusing on the unique features or differences of each modality. Different modalities may contain important information not found in other modalities; fusing feature representations from different modalities plays a crucial role in sentiment analysis.

[0061] After extracting the basic feature representations of each modality, they are projected onto different vector subspaces in two ways: using a shared parameter encoder. Projecting them onto a similarity subspace, parameter sharing ensures that the modality representations in this subspace have a similar distribution. Three modality-specific encoders are used. Learn the unique features of each modality and project them into the feature subspace.

[0062] in .

[0063] Step 5: The modal indeterminate features of similar subspaces are extracted using Mamba features and filtered by local transpose attention of the optimized Transformer encoder based on multipath transpose attention to obtain advanced trimodal fusion indeterminate features.

[0064] In multimodal sentiment analysis, features from different modalities are often extracted independently and then simply concatenated, lacking deep interaction. Simply superimposing modal feature vectors without mining semantic relationships leads to the loss of complementary information between modalities. It is evident that current Mongolian multimodal processing suffers from insufficient deep interaction between different modalities, easily resulting in problems such as untargeted attention calculations and inadequate capture of key local information. Therefore, this invention proposes to construct a Multi-path Transposed Attention Optimized Transformer (MPTT). It embeds a one-dimensional convolution operation into the multi-head attention mechanism of the traditional Transformer encoder to adjust attention calculations. Through the local feature extraction capability of one-dimensional convolution, it finely adjusts the attention weights of Mongolian multimodal data. By interacting with information from different modal features, a transposed attention map is formed, allowing the model to calculate attention in a targeted manner based on the characteristics of different modalities, focusing on more important local information. This improves the flexibility and effectiveness of attention calculations and avoids focusing on useless information, leading to inaccurate predictions.

[0065] refer to Figure 5 As shown, the MPTT of this invention still retains the feedforward layer of the Transformer and the residual and normalization processes. Data is used as input to the Transformer through positional encoding. The MPTA layer (MPTA is an improved attention mechanism in the optimized Transformer) utilizes local attention to calculate the dependencies between parts of the sequence, focusing on key information and discarding interfering information. Finally, through softmax and residual and normalization processes, more accurate multimodal information is output.

[0066] This step extracts high-level features from modal indeterminate features in similar subspaces, leveraging common information across different modalities to reduce noise and improve performance. After modal preprocessing, audio and video representations, like text, possess sequence features.

[0067] First, Mamba feature extraction is performed. During the feature extraction process, Mamba dynamically tracks key information in the sequence through a state update mechanism, which can effectively filter noise and focus on core features.

[0068] in It is the modal representation in the similarity subspace after feature extraction. It is a module that contains Mamba blocks, normalization, and residual joins.

[0069] refer to Figure 6 As shown, the local transposed attention filtering of the present invention is implemented as follows: (a) Input processing: First, the data is used as the input of the encoder through positional encoding and divided into three branches: query, key, and value. Each branch first passes through a linear transformation layer to map the input features to a suitable dimension, and performs preliminary dimension adaptation for subsequent processing. Then, the linear transformation results of the query branch and the key branch are respectively entered into a one-dimensional convolutional layer (Conv1D) to perform preliminary capture and dimension adjustment of local features.

[0070] After passing through the linear layer, the following is obtained: It is the weight matrix of the linear layer.

[0071] Subsequently, the model enters the transposed attention layer. Utilizing a local attention mechanism, it calculates the similarity between features at different positions and across different modalities based on the vectors processed by the one-dimensional convolutional layer. The transposed attention weights are then obtained through a softmax function, constructing a transposed attention map adapted to the multimodal characteristics of Mongolian. This process focuses on the core semantic units in the Mongolian multimodal data, discarding redundant and interfering information, significantly improving the flexibility and effectiveness of attention calculation, and allowing the model to initially and accurately anchor valuable content. Finally, the transposed attention weights and the linear transformation results of the value branch are subjected to a dot product operation to obtain the weighted features.

[0072] Specifically, in the branches that generate Q and K, the results of the linear transformation undergo a one-dimensional convolution operation to extract local features and adjust dimensions, thereby enhancing the model's ability to capture local patterns. Finally, the weighted features are added to the original values ​​to obtain the encoder output.

[0073] Specifically, after the above processing, we obtain The similarity weights are calculated through a dot product operation, and then the transposed attention weights are obtained by passing them through a softmax function. Then, the transposed attention weights are multiplied by V to obtain the weighted features: Finally, the weighted features are added to the original V to obtain the output of the MPTA module: .

[0074] To more effectively integrate common relationships within similarity subspaces, modality fusion techniques such as Multi-Path Transposed Attention Transformer (MPTT) are combined with state space to model bimodal representation sequences. A cross-modal enhancement module based on MPTT is designed, taking two modal representations as input and outputting a composite representation connected to the residuals of the two modal representations. in It is a composite representation of text and audio, as well as text and video, in a similarity subspace. It is a cross-modal reinforcement block. It is root mean square layer normalization.

[0075] Finally, to obtain high-level trimodal fusion uncertain features containing all association information of the similarity subspace. ,Will and After performing elementwise addition, the sum is input into the linear layer: The modal representation of the entire similarity subspace will be used together with the modality-specific output of MFRF for comprehensive information fusion in the subsequent sentiment prediction stage.

[0076] Therefore, after the learning and processing steps above, the initial features of the vector similarity subspace and feature subspace are obtained. In this step, the modal indeterminate features of the similarity subspace are used to extract high-level features through Mamba, and then different modal features are fused through MPTT. The resulting supermodal features are the high-level trimodal fused indeterminate features. .

[0077] Step 6: Complementary fusion of modal indeterminate features in similar subspaces and modal features in specific subspaces is performed, and then local transpose attention filtering based on multipath transpose attention of the optimized Transformer encoder is used to obtain advanced dual-modal fusion specific features.

[0078] Modal heterogeneity refers to the significant differences in the distribution of data across different modalities, making direct fusion prone to conflicts. Insufficient semantic mining manifests in merely matching surface features while neglecting deeper connections such as metaphors. This necessitates using modality transformation and semantic alignment networks to map heterogeneous features to a unified space, mining cross-modal implicit semantics, and improving fusion performance. Therefore, to alleviate the significant heterogeneity in the distribution of different Mongolian modalities and the problem of insufficient semantic mining, and to enable full interaction among the various modal representations of Mongolian in the feature subspace to provide a comprehensive emotional perspective, this paper proposes a modal feature complementary fusion module (MFCF) based on a dual-state space, combining Mongolian modal information from similar subspaces.

[0079] To more effectively integrate the common relationships of Mongolian multimodal expressions within the similarity subspace, fusion techniques adapted to the modal characteristics of Mongolian, such as the multi-path transposed attention Transformer, are combined with the state space to model the bimodal representation sequence. This fully captures the association between Mongolian phonetic prosody and text semantics, as well as the relationship between video visual features and text characters, ultimately resulting in a trimodal fusion feature covering Mongolian text, video, and audio.

[0080] In this step, the modal features of similar subspaces and modal features of specific subspaces are complementary and fused using a modal feature complementarity fusion module to extract certain effective similarity of emotional features and specific features unique to each modality. Specifically, such as... Figure 7 As shown, in the modality feature complementarity fusion module, the similarity representation and specificity representation of each modality are concatenated along the feature dimension. Then, a simple feedforward neural layer is used to achieve the fusion of complementary information of a single modality, as represented by: in For a unified representation that includes modality-specific information and invariant information, This represents the activation function. This indicates a splicing operation. This is a specific representation of the modality. This represents the similarity of modes.

[0081] To enable further cross-modal interaction of the unified representation containing modal information, a multi-path transposed attention Transformer is combined to achieve deeper intermodal information exchange, ultimately forming text-video fusion feature vectors and text-audio fusion feature vectors for final Mongolian multimodal fusion. Specifically, the high-level bimodal fusion-specific features obtained in this step are represented as follows: in, For specific features of text-audio bimodal fusion, For text-video bimodal fusion specific features, This represents a cross-modal interaction function, used to achieve deep-level exchange and fusion of information between different modalities. , , These are unified representations that contain modal-specific information and invariant information, respectively, for audio, text, and video modes. These are learnable parameters in the audio-text interaction process. These are learnable parameters in the visual-text interaction process; for ,by For query, Calculate the similarity matrix between the text and audio modal for the key. : for ,by For query, Calculate the similarity matrix between the text and video modalities for the key. : in and It is a learnable attention matrix. It is the sum of the dimensions of each attention head.

[0082] Composite mode representation It is possible and get: .

[0083] Composite mode representation It is possible and get: .

[0084] Step 7: Concatenate the high-level trimodal fusion indeterminate features and the high-level bimodal fusion specific features to obtain a joint multimodal representation, and finally realize multimodal sentiment classification through a fully connected layer.

[0085] In a further embodiment, such as Figure 8As shown, this invention can also concatenate initial text features, initial audio features, initial video features, high-level trimodal fusion indeterminate features, and high-level bimodal fusion specific features together, and obtain the joint multimodal representation through a cross-modal Transformer and a cross-modal attention mechanism. That is, the input is a prediction layer composed of a cross-modal Transformer encoder and multiple attention mechanisms for sentiment prediction. . This is the result of sentiment prediction. and This indicates a cross-modal Transformer encoder. ,yes The concatenated representation is the joint multimodal representation.

[0086] To evaluate the performance of the Mongolian multimodal emotion feature fusion model, we plan to use accuracy, precision, recall, and F1 score as performance metrics, and verify the effectiveness of the model through multiple sets of comparative experiments. 1. Dataset and Evaluation Criteria The dataset used in this invention comprises 2100 multimodal samples. It is a Mongolian multimodal emotion dataset containing seven discrete emotions, constructed by the Artificial Intelligence Laboratory of Inner Mongolia University of Technology: happiness, anger, sadness, surprise, fear, disgust, and neutral (i.e., a relatively stable voice without emotion). Each emotion category uses 300 sentences. For each emotion category, 60% of the data is randomly selected as the training set, 10% as the validation set, and 30% as the test set.

[0087] This invention proposes to use accuracy, precision, recall, and F1 score as metrics for evaluating model performance. Specifically, the formula for calculating accuracy is as follows: The formula for calculating accuracy is: The formula for calculating recall rate is: The formula for calculating the F1 value is: Accuracy represents the proportion of correctly predicted samples out of the total number of samples; Precision represents the proportion of samples predicted as positive by the model that are actually positive; Recall represents the proportion of samples that are actually positive that are correctly predicted as positive; F1 represents the F1 score, which is the harmonic mean of precision and recall, reflecting the combined performance of both. TP represents the number of samples that are actually positive and predicted as positive by the model. FN represents the number of samples that are predicted as negative by the model but are actually positive. FP represents the number of samples that are predicted as positive by the model but are actually negative. TN represents the number of samples that are actually negative and predicted as negative by the model. Higher scores for accuracy, precision, recall, and F1 score indicate better performance of the sentiment analysis model. Table 1 shows the confusion matrix required for the calculation. Table 1 Confusion Matrix 2. Comparison of Model Training Results From a training perspective, this reflects the model's performance under different component configurations, such as... Figure 9 As shown. Figure 9 The diagram uses four sub-figures to illustrate the changes in accuracy, F1 score, precision, and recall across four core evaluation metrics for classification tasks: the "Full Model" and the removal of different modules (Text Debiasing, MPTT, and Dual Subspace Fusion). The sub-figures show that the Full Model achieved optimal performance across all metrics during training, with an accuracy of 0.924 and an F1 score of 0.922. Removing the Text Debiasing, MPTT, and Dual Subspace Fusion modules sequentially resulted in varying degrees of decline across all metrics. The most significant drop was observed when the MPTT module was removed; for example, accuracy plummeted from 0.924 to 0.848, indicating the crucial role of the MPTT module in model performance. The excellent performance of the complete model across various metrics demonstrates that the model proposed in this invention, through the synergistic effect of its modules, can effectively learn and fit data features during training. Therefore, the model proposed in this invention exhibits good performance, and the above ablation experiment results fully prove the effectiveness of the model and the design of its modules.

Claims

1. A Mongolian multimodal sentiment analysis method based on dual-state space and multi-path transposed attention, characterized in that, Includes the following steps: Step 1: Extract initial text features, initial audio features, and initial video features from the original multimodal dataset. Step 2: Encode the initial text features, initial audio features, and initial video features using a shared encoder to obtain similar subspace modal indeterminate features, and then encode them using a corresponding modality-specific encoder to obtain specific subspace modal features; Step 3: The modal indeterminate features of the similar subspace are extracted using Mamba features and filtered by local transpose attention of the optimized Transformer encoder based on multipath transpose attention to obtain advanced trimodal fusion indeterminate features; Step 4: Complementarily fuse the modal indeterminate features of the similar subspace with the modal features of the specific subspace, and then obtain the advanced dual-modal fusion specific features by filtering the local transpose attention of the optimized Transformer encoder based on multi-path transpose attention. Step 5: The advanced trimodal fusion indeterminate features and the advanced bimodal fusion specific features are concatenated to obtain a joint multimodal representation, and finally multimodal sentiment classification is achieved through a fully connected layer.

2. The Mongolian multimodal sentiment analysis method based on dual-state space and multi-path transposed attention as described in claim 1, characterized in that, In step 1, text features are extracted using BERT, and then bias factors are eliminated using a text debiasing module to obtain the initial text features. The text debiasing module is implemented based on a global dictionary and a two-branch Transformer network. In the two-branch Transformer network, one branch performs self-attention calculation on the input text features to obtain an estimate of the knowledge within the sample; the other branch uses the input text features as a query vector and performs cross-attention calculation using the global dictionary as a key and value vector to obtain an estimate of the knowledge across samples.

3. The Mongolian multimodal sentiment analysis method based on dual-state space and multi-path transposed attention as described in claim 2, characterized in that, Use a deep adaptive clustering algorithm across the entire training set. The global dictionary is initialized on the text modality as follows: in, Indicates the global dictionary. This represents a deep adaptive clustering algorithm. This represents the average pooling operation. This represents the Transformer operation. This represents the number of clusters in the text feature clustering.

4. The Mongolian multimodal sentiment analysis method based on dual-state space and multi-path transposed attention as described in claim 2, characterized in that, The dual-branch Transformer network consists of L stacked Transformer blocks. Each Transformer block contains two branches. One branch initializes the input text features through embedding, then captures semantic dependencies through self-attention calculation, and subsequently uses a feedforward neural network to perform nonlinear transformation and dimension mapping to obtain transformed features. The other branch constructs the association link between the current sample and the global semantic pattern through cross-attention calculation, and then uses a feedforward neural network to perform nonlinear transformation and dimension mapping to obtain transformed features. The transformed features obtained from the two branches are merged and used as the dual-branch input of the next Transformer block. The output of the last Transformer block is the initial text features.

5. The Mongolian multimodal sentiment analysis method based on dual-state space and multi-path transposed attention as described in claim 1, characterized in that, In step 1, audio features are extracted using Librosa, and then the initial audio features are obtained using a temporal LSTM encoder; video features are extracted using OpenFace, and then the initial video features are obtained using a temporal LSTM encoder.

6. The Mongolian multimodal sentiment analysis method based on dual-state space and multi-path transposed attention as described in claim 1, characterized in that, Step 2 involves encoding the initial text features, initial audio features, and initial video features using a shared encoder to obtain similar subspace modality-deterministic features. The implementation method is as follows: Using the initial text features, initial audio features, and initial video features as input, a lightweight structure with fully connected layers and non-linear activation is adopted, and a self-attention mechanism is combined to enhance the global correlation of features. Contrastive learning loss is used for training: to make semantically related cross-modal pairs as close as possible in the shared space, and semantically unrelated cross-modal pairs as far apart as possible. Finally, the feature vector mapped to the shared space is output, which is the modality-indeterminate feature of the similar subspace.

7. The Mongolian multimodal sentiment analysis method based on dual-state space and multi-path transposed attention as described in claim 1, characterized in that, The optimized Transformer encoder based on multi-path transposed attention embeds a one-dimensional convolution operation into the multi-head attention mechanism of the traditional Transformer encoder to adjust the attention calculation. The local transposed attention filtering is implemented as follows: First, the data is used as the input of the encoder through position encoding and is divided into three branches: query, key and value. Each branch first goes through a linear transformation layer to perform preliminary dimension adaptation. Then, the linear transformation results of the query branch and the key branch are respectively entered into a one-dimensional convolutional layer to perform preliminary capture and dimension adjustment of local features. Subsequently, the transposed attention layer is entered. Using the local attention mechanism, the similarity between features at different positions and different modalities is calculated based on the vector processed by the one-dimensional convolutional layer. The transposed attention weights are obtained by processing with the softmax function. Then, the transposed attention weights and the linear transformation result of the value branch are subjected to a dot product operation to obtain the weighted features. Finally, the weighted features are added to the original values ​​to obtain the encoder output.

8. The Mongolian multimodal sentiment analysis method based on dual-state space and multi-path transposed attention as described in claim 1 or 7, characterized in that, In step 4, the modal features of similar subspaces and specific subspaces are fused together using a modal feature complementarity fusion module. This module concatenates the similarity and specificity representations of each modality along the feature dimension, and then uses a feedforward neural layer to achieve the fusion of complementary information for a single modality, as shown below: in For a unified representation that includes modality-specific information and invariant information, This represents the activation function. This indicates a splicing operation. This is a specific representation of the modality. For modal similarity representation, These refer to text, audio, and video modalities, respectively.

9. The Mongolian multimodal sentiment analysis method based on dual-state space and multi-path transposed attention as described in claim 8, characterized in that, The high-level bimodal fusion specific features obtained by local transpose attention filtering of the optimized Transformer encoder based on multipath transpose attention are expressed as follows: in, For specific features of text-audio bimodal fusion, For text-video bimodal fusion specific features, This represents a cross-modal interaction function, used to achieve deep-level exchange and fusion of information between different modalities. , , These are unified representations that contain modality-specific information and invariant information for audio, text, and video modes, respectively. These are learnable parameters in the audio-text interaction process. These are learnable parameters in the visual-text interaction process; for ,by For query, Calculate the similarity matrix between the text and audio modal for the key. : for ,by For query, Calculate the similarity matrix between the text and video modalities for the key. : in and It is a learnable attention matrix. It is the sum of the dimensions of each attention head.

10. The Mongolian multimodal sentiment analysis method based on dual-state space and multi-path transposed attention according to claim 1, characterized in that, In step 5, the initial text features, initial audio features, initial video features, and advanced trimodal fusion indeterminate features and advanced bimodal fusion specific features are concatenated together, and the joint multimodal representation is obtained through cross-modal Transformer and cross-modal attention mechanism.

Images