Multi-modal fusion identification modeling method based on cross structured state space sequence

Abstract

The invention belongs to the technical field of multi-modal data processing, sentiment analysis and artificial intelligence, and particularly relates to a multi-modal fusion identification modeling method based on a cross structured state space sequence. Comprising the following steps: step 1, constructing a multi-modal representation dictionary module based on consistent feature learning; 2, constructing a cross-modal information representation module based on attention low-rank operation; step 3, constructing a cross-modal interaction feature representation module based on a cross-structured architecture; and 4, constructing a multilevel feature fusion module based on a state space sequence architecture. The multi-modal emotion recognition modeling method has the beneficial effects that experimental results show that the multi-modal emotion recognition modeling method provided by the invention has remarkable effects on recognition accuracy and robustness on different public data sets, and has better model generalization.

Description

Technical Field

[0001] This invention belongs to the fields of multimodal data processing, sentiment analysis and artificial intelligence technology, and specifically relates to a multimodal fusion identification and modeling method based on cross-structured state space sequences. Background Technology

[0002] With the rapid development of internet technology and the widespread adoption of mobile communication devices, internet social media platforms have become important channels for people to obtain information and exchange ideas. At the same time, upgrades in communication technology and the widespread use of photographic and video equipment have significantly changed the content format of internet social media, gradually shifting from primarily text-based to a multimedia format that integrates video, images, and text. This multimedia data contains rich emotional information, which is of great significance for understanding user needs, predicting market trends, and optimizing service experiences. Therefore, emotion recognition research has gradually become a hot topic of interest in both academia and industry.

[0003] Despite progress in multimodal emotion recognition research, current technologies still have many shortcomings. In multimodal data, information between different modalities (such as video, images, and text) is interconnected and complex, making it difficult for single-modal analysis to comprehensively capture emotional information. For example, images, sounds, and text in a video may collectively express a certain emotion, but analyzing only one modality may not accurately convey the overall emotion. Furthermore, different feature extraction methods for different modalities lead to differences in feature encoding semantics. How to extract single-modal features suitable for multimodal recognition and effectively model the potential relationships between multiple modalities is a current challenge in multimodal emotion recognition. Traditional feature fusion methods often struggle to balance the information contributions of different modalities, resulting in inaccurate emotion recognition results. Traditional multimodal fusion methods, such as tensor product calculation, while preserving temporal alignment information, suffer from excessively large computational matrix dimensions, leading to low training efficiency. This is particularly evident when processing large-scale multimedia data, severely limiting the practical application of multimodal emotion recognition technology. When processing long-sequence data, traditional models such as recurrent neural networks (RNNs) suffer from vanishing and exploding gradients, making it difficult to effectively capture long-range dependencies. This is crucial for understanding emotional changes in long-sequence data such as video and audio. Summary of the Invention

[0004] The purpose of this invention is to provide a multimodal fusion identification and modeling method based on cross-structured state space sequences, which can effectively solve the problems of complex intermodal relationships, difficulty in feature extraction and fusion, high computational complexity, and difficulty in modeling long sequence dependencies in multimodal data fusion, thereby improving the performance and efficiency of multimodal emotion recognition.

[0005] The technical solution of the present invention is as follows: A multimodal fusion identification and modeling method based on cross-structured state space sequences, comprising the following steps:

[0006] Step 1: Construct a multimodal representation dictionary module based on consistent feature learning;

[0007] A module for preprocessing high-dimensional features from visual, audio, and text modalities, and then performing joint high-dimensional feature embedding and feature extraction after encoding all three modalities using ReFormer;

[0008] Step 2: Construct a cross-modal information representation module based on attention-based low-rank operations;

[0009] Efficient fusion of multimodal information is achieved through low-rank tensor computation, which reduces computational cost and hyperparameter quantity while ensuring model performance;

[0010] Step 3: Construct a cross-modal interaction feature representation module based on a cross-structured architecture;

[0011] By employing a cross-structured architecture and a multi-head self-attention mechanism, common semantics among multiple modalities can be captured.

[0012] Step 4: Construct a multi-level feature fusion module based on a state-space sequence architecture;

[0013] Leveraging selective state-space characteristics enhances long sequence modeling capabilities and cross-modal dynamic response performance.

[0014] Step 1 includes:

[0015] Step 11: Dimensionality reduction of the three-modal high-dimensional features is performed using the TCN temporal convolution model to obtain low-dimensional semantic space mapping vectors. The corresponding indices of the two modalities used in the calculation are defined as 1 and 2, respectively. H1 and H2 are the original input high-dimensional features, L1 and L2 are the output low-dimensional semantic space mapping vectors, and p1 and p2 are the network parameters of the TCN1 and TCN2 models, respectively. The calculation process is as follows:

[0016] L1 = TCN1(H1; p1)

[0017] L2 = TCN2(H2; p2)

[0018] Step 12: After obtaining the dimensionality-reduced modal features, calculate the correlation coefficients between them. Based on the relationship between the covariance and variance quantization feature sets, the correlation coefficient calculation method is as follows:

[0019]

[0020] Where corr(L1,L2) is the correlation coefficient. Let cov(L1, L2) be the covariance of the low-dimensional semantic space mapping vectors L1 and L2, and var(L1) and var(L2) be the variances of L1 and L2, respectively. The covariance and variance are used to analyze the features between the two sets of features to obtain their correlation. In the application of this module, it is called TCN. t TCN v and TCN a When training three models simultaneously, the original optimization objective is calculated as follows:

[0021] Step 13: Generate the gating vector G corresponding to each mode through linear transformation. t G v and G a Construct a dynamic gating mechanism;

[0022] Step 14: Optimize the dimensions of the gated feature vectors, add absolute position vectors of each feature point to preserve temporal information, sort the feature activation values ​​from high to low, retain the core dimensions according to the needs of subsequent modules, and restore the position order.

[0023] Step 2 includes:

[0024] Step 21: Define the abstract representation of multimodal fusion;

[0025]

[0026] When using a linear layer to output multimodal fusion results, the output formula is as follows:

[0027] h = W * H + b,

[0028] Here, the vector dimension of H is M, and the weights of its linear output layer matrix W are M+1 dimensional.

[0029] Step 22: Decompose the original weight vector W, dividing the M+1 dimensions into M parts for independent calculation. Consider it as being made by d y M-order tensors Composition, for each W k Tensors can also be decomposed using a similar method as tensors h, with the following decomposition formula:

[0030]

[0031] The smallest value of R that makes the equation true is called the rank of the tensor, which is the set of two-dimensional vectors. It becomes a low-rank factor of the original tensor;

[0032] Step 23: Optimize the calculation process.

[0033] Step 3 includes:

[0034] Step 31: During the input phase, the visual, audio, and text modalities remain independent, and each modality generates a query vector Q, a key-value vector K, and a value vector V, respectively.

[0035] Step 32: After concatenating the three sets of attention vectors, perform unified multi-head self-attention calculation. After the calculation results are sized, perform independent residual connections with the original inputs of each modality. By analyzing the attention calculation results by modality partition, the off-diagonal region is the feature fusion result of the two modalities, so as to obtain at least one set of modal interaction information in a single round of attention calculation.

[0036] Step 33: Cross-domain semantic extension is completed through the encoder and decoder. The encoder's calculation process is as follows:

[0037] f T→A =CATransformerEncoder(f A f T )

[0038] Its input audio features f A With text features f T First, a cross-modal domain calculation will be performed to obtain an f. T→A Features, and in the decoder calculation in the following formula, it will f T→A Cross-modal semantic information in the text is obtained through text modal information f T As a cross-modal semantic fixed representation and "text-visual" embedded modal feature f T→V Interacting with the content results in a text-based modality.

[0039] Step 4 includes:

[0040] Step 41: Construct the basic framework of the state-space model, whose core equations are the continuous-time state equation and output equation as follows:

[0041] h(t) = A*h(t) + B*x(t)

[0042] y(t)=C*h(t)+D*x(t)

[0043] First, the input sequence x(t) passes through matrix B. Then, the output of matrix B is summed with the output of matrix A to obtain the latent state representation h(t). The input of matrix A is h(t) from the previous input process. After the latent state representation h(t) is input into matrix C, it is summed with the result of the input sequence x(t) passing through matrix D to obtain the final predicted output y(t).

[0044] Step 42: Optimize the performance of the state-space model. The state-space model is improved using the HiPPO matrix, which is initialized for training by constructing a specific expression.

[0045] Step 43: Cross-Mamba model fusion process, used to combine the features after fusion of two sets of multimodal features. Its three module inputs are first integrated in length, and features are concatenated in the direction of the number of feature channels. First, the selection input for each feature dimension is obtained through its matrix B, and the input of this dimension and matrix A are used to complete the transformation from each state to the hidden state h. k The update operation involves matrix Δ, which determines the importance of the input data for each channel based on its characteristics, and indirectly optimizes matrix A through matrix Δ to complete the update operation in the hidden state h. k The core content is retained, and the information from each state-space model is summarized and output through the output matrix C to obtain the output sequence y. k After obtaining the output sequence y k Then, the results are mapped through a linear output layer to obtain the output results of various task indicators.

[0046] The beneficial effects of this invention are as follows: Experimental results show that the multimodal emotion recognition modeling method proposed in this invention has significant effects on recognition accuracy and robustness on different public datasets, and also has better model generalization. To address the problems of inconsistent alignment and heterogeneous discrimination among different modalities, a multimodal representation dictionary based on consistent feature learning is designed. This transforms the complex problem of relationships between different modalities into feature representation and reconstruction in a unified space, effectively solving challenges such as heterogeneity, misalignment, and modality loss, laying the foundation for reliable representation and fusion of multimodal data. To address the problems of high computational complexity and difficulty in representing interactive features in cross-modal information representation learning, a cross-modal information representation module based on low-rank attention operations is designed. By introducing a low-rank weight decomposition method for tensor calculation, the computational complexity and hyperparameter quantity in feature representation and information fusion are reduced without affecting model performance. Furthermore, a cross-modal interaction feature representation module is designed based on a cross-structured architecture, utilizing a multi-head self-attention mechanism to optimize allocation, fully capturing the semantic relationships between different modalities such as text, vision, and audio, and realizing interaction feature representation and cross-modal fusion. To address the lack of long-sequence modeling capability in multimodal emotion recognition, this paper proposes a multimodal emotion recognition modeling method based on cross-structured state space sequences. By integrating spatiotemporal deep neural networks, cross-structured representations, and selective state space characteristics, this method achieves efficient long-sequence multimodal data modeling, optimizes data access and parallel computation order during the computation process, and improves the model's ability to model long-sequence data and its cross-modal dynamic response capability. Attached Figure Description

[0047] Figure 1 Here is a structural diagram of the multimodal representation dictionary module based on consistency feature learning;

[0048] Figure 2 A structural diagram of the cross-modal semantic information embedding module;

[0049] Figure 3 This is a schematic diagram of the original tensor product calculation process;

[0050] Figure 4 To simplify the computational structure diagram for low-rank tensors;

[0051] Figure 5 This is a feature fusion structure diagram of a cross-modal information representation module based on attention-based low-rank operations;

[0052] Figure 6 A cross-modal interaction feature representation model based on a cross-structured architecture;

[0053] Figure 7 This is a diagram of the state-space model structure.

[0054] Figure 8 Here is a structural diagram of the Mamba selective state-space model;

[0055] Figure 9 A bar chart showing the experimental results of a multi-level feature fusion model based on a state-space sequence architecture (MOSI dataset);

[0056] Figure 10 Comparison and display of long sequence segmentation results for single-scene test samples - MOSI dataset;

[0057] Figure 11 A bar chart showing the experimental results of a multi-level feature fusion model based on a state-space sequence architecture (MOSEI dataset);

[0058] Figure 12 Comparison and display of long sequence segmentation results for single-scene test samples - MOSEI dataset;

[0059] Figure 13 A bar chart showing the experimental results of a multi-level feature fusion model based on a state-space sequence architecture (SIMSv2 dataset);

[0060] Figure 14 Comparison of partial results for test samples - SIMSv2 dataset. Detailed Implementation

[0061] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0062] This invention provides a multimodal fusion identification and modeling method based on cross-structured state space sequences. Addressing the issues of insufficient performance and poor computational efficiency in multimodal emotion recognition, this method constructs a multimodal representation dictionary based on consistent feature learning, mines cross-modal information through low-rank attention operations, represents interactive features using a cross-structured architecture, and employs a multimodal fusion strategy based on state space sequences. This method (CrossMT-Mamba) can control the feature interaction information of different modalities, represent multimodal consistency patterns and potential causal relationships, and, relying on a better fusion strategy and a more efficient model architecture, provides a reliable and stable approach for multimodal emotion recognition and multimodal data mining tasks.

[0063] This invention designs a multimodal representation dictionary based on consistent feature learning, constructs a cross-modal feature association mechanism by calculating the correlation coefficients between features of different modalities, and combines a gating dynamic screening strategy to accurately retain highly correlated features and suppress redundant noise. This enhances the consistency of multimodal features while fully preserving temporal sequence information, building a stable data foundation for subsequent feature fusion. A cross-modal information description method based on attention-based low-rank operations is constructed, introducing a low-rank tensor decomposition method. Addressing the problem of dimensionality explosion and low training efficiency caused by traditional tensor products in multimodal fusion, the low-rank processing reconstructs the multimodal feature tensor operation logic. While ensuring that model performance is not compromised, computational complexity is significantly reduced, training efficiency is improved, and efficient processing of large-scale multimedia data is achieved. This paper constructs a cross-modal interaction feature representation method based on a cross-structured architecture. Relying on the cross-structured architecture and the core design of Transformer, it optimizes weight allocation through a multi-head self-attention mechanism to deeply interact with and mine multimodal semantic information such as text, vision, and audio. This fully captures potential semantic relationships between modalities, providing refined interaction feature support for cross-modal fusion and improving the semantic capture accuracy of multimodal sentiment recognition. A multi-level feature fusion architecture based on state-space sequences is designed, constructing a Cross-Mamba multimodal fusion module. Utilizing its selective state-space characteristics, it dynamically captures dependencies in long-sequence data and efficiently integrates multimodal features output from preceding modules. By optimizing data access and parallel computation order during the computation process, deep fusion of multi-level features is achieved, generating a holistic multimodal fusion representation. This provides strong representational support for subsequent tasks such as sentiment classification, improving the model's modeling ability and dynamic response performance for long-sequence multimodal data.

[0064] This invention effectively addresses the challenges of complex intermodal relationships, difficult feature extraction and fusion, high computational complexity, and difficulties in modeling long-sequence dependencies in multimodal emotion recognition tasks by utilizing a structured state-space sequence learning framework combined with multi-stage feature cross-structured fusion. Specifically, it first uses a Temporal Convolutional Network (TCN) to perform dimensionality reduction and coarse-grained feature extraction on multimodal input data such as video, speech, and text. By calculating the correlation coefficients between features of different modalities, a gating vector is designed and generated to achieve adaptive filtering and dimensionality aggregation of coarse-grained features, resulting in a multimodal representation dictionary with consistent feature learning. Secondly, a low-rank weight decomposition method is introduced to decompose the original weight vector into multiple low-rank fusion factors, achieving the fusion of intermodal correlations and characteristics. This reduces computational complexity while preserving temporal alignment information, resulting in a cross-modal information representation vector. Subsequently, a cross-structured representation framework based on an encoder-decoder neural network was designed, leading to the proposal of a cross-modal semantic interaction feature representation module (CA-Transformer). This module takes text modality feature vectors and cross-modal feature embedding vectors as input, and mines the common semantic content among different modalities through multi-head attention mechanisms to obtain cross-modal semantic interaction information vectors. Finally, a multi-level feature fusion module based on a state-space sequence architecture was designed to control the feature interaction information of different modalities, achieving feature concatenation and mixing, and latent causal representation. This leads to the proposal of a cross-structured state-space sequence multimodal fusion identification and modeling method (CrossMT-Mamba), which outputs complete, comprehensive, and high-dimensional abstract multimodal fusion representation rules, improving the performance of multimodal data fusion and the accuracy of sentiment recognition.

[0065] This invention introduces a multimodal representation dictionary module based on consistent feature learning, a cross-modal information representation module based on attention-based low-rank operations, and a cross-modal interaction feature representation module based on a cross-structured architecture. Combined with a Cross-Mamba multi-level feature fusion module based on a state-space sequence architecture, this invention achieves efficient fusion of multimodal cross-domain representations and ultimately constructs a multimodal sentiment analysis model with higher sentiment recognition accuracy, more refined feature fusion, and stronger model robustness.

[0066] The principles of the present invention will be described below with reference to the accompanying drawings and Embodiments 1, 2, and 3.

[0067] A multimodal fusion identification and modeling method based on cross-structured state space sequences includes the following steps:

[0068] Step 1: Construct a multimodal representation dictionary module based on consistent feature learning;

[0069] To address the issues of heterogeneity, misalignment, and missing modalities in multimodal data, a cross-modal representation dictionary module is constructed. The overall structural design of its cross-modal semantic space mapping gating module is shown in the attached figure. Figure 1 As shown, complex modal relationships are transformed into feature representations and reconstructions in a unified space. First, high-dimensional features from the visual, audio, and text modalities are preprocessed: visual features are extracted using traditional motion unit encoding and visual semantic combination; audio features are extracted using a combination of global and local multivariate features; and text features are extracted using the RoBERTa model. All three modalities are then encoded using ReFormer to complete the joint high-dimensional feature embedding and feature extraction module structure diagram, as shown in the attached diagram. Figure 2 As shown.

[0070] The core process of the module is as follows:

[0071] The first step involves reducing the dimensionality of the three high-dimensional features using the TCN temporal convolution model to obtain low-dimensional semantic space mapping vectors. The corresponding indices of the two modalities used in the calculation are defined as 1 and 2, respectively. H1 and H2 are the original input high-dimensional features, L1 and L2 are the output low-dimensional semantic space mapping vectors, and p1 and p2 are the network parameters of the TCN1 and TCN2 models, respectively. The calculation process is shown in the following formula:

[0072] L1 = TCN1(H1; p1)

[0073] L2 = TCN2(H2; p2)

[0074] The second step, after obtaining the dimensionality-reduced modal features, is to calculate the correlation coefficients between them, quantizing the relationship between the feature sets based on covariance and variance. The optimization objective of this module is designed to maximize the correlation coefficient corr(L1,L2) calculated between the low-dimensional semantic space mapping vectors obtained by the two networks during the training phase, thereby obtaining the optimal network parameters for the two computational networks. The actual process of calculating the correlation coefficient is shown below:

[0075]

[0076] Where cov(L1,L2) is the covariance of the low-dimensional semantic space mapping vectors L1 and L2, and var(L1) and var(L2) are the variances of L1 and L2 respectively. The covariance and variance are used to analyze the features between the two sets of features to obtain their correlation. Since there are three modalities, in practical applications of the module, it is TCN. t TCN v and TCN a When training three models simultaneously, the original optimization objective is calculated as follows:

[0077]

[0078] The third step, after calculating the correlation, is to map the correlation information to gating information, and then use a dynamic gating mechanism to filter data with different correlations to varying degrees and from different angles. A linear transformation is used to generate the gating vector G corresponding to each mode. t G v and G a A dynamic gating mechanism is constructed: the absolute value-based correlation coefficient is used as the weight to adjust the retention ratio of gating information of different modalities. Taking audio and visual modal interaction as an example, its output formula is as follows:

[0079] H` a =|corr av |*(H a ⊙G v )+(1-|corr av |)*(H a ⊙G a )

[0080] This formula adjusts the degree to which the vector retains the two gating modes by using the correlation coefficient. When the two modes are highly correlated, we will retain more of the output from the gating information of the other mode, so as to facilitate the multimodal feature fusion in the subsequent model.

[0081] Finally, the gated feature vectors (such as H`) are processed. a Dimensional optimization is performed by adding absolute position vectors for each feature point to preserve temporal information, sorting them by feature activation values ​​from high to low, retaining core dimensions according to the needs of subsequent modules, and restoring the positional order. This process simplifies redundant parameters caused by sparse and low-activation dimensions while fully preserving temporal information, laying a stable foundation for subsequent multimodal feature fusion.

[0082] Step 2: Cross-modal information representation module based on attention-based low-rank operation;

[0083] To address the issues of high computational complexity and difficulty in representing interactive features in traditional tensor fusion models, a low-rank weight decomposition method is introduced based on TFN-type models. This method achieves efficient fusion of multimodal information through low-rank tensor computation, reducing computational costs and the number of hyperparameters while ensuring model performance.

[0084] The core process of the module is as follows:

[0085] The first step is to clarify the abstract expression of multimodal fusion. The traditional TFN model integrates multimodal features through tensor product, and its fusion formula is as follows:

[0086]

[0087] When using a linear layer to output multimodal fusion results, the output formula is as follows:

[0088] h = W * H + b, Here, the vector dimension of H is M, and the weights of its linear output layer matrix W are M+1 dimensional. This formula shows that the dimension of the weight tensor W increases exponentially with the number of modes, and its computational complexity is O(W). This presents an inherent disadvantage for model training. To address the high computational complexity in tensor fusion, we chose a low-rank fusion method that decomposes the weights in the above calculation process using a set of low-rank fusion factors that are independent of each modality, and obtains the low-rank weight tensor through new calculations.

[0089] The second step is to decompose the original weight vector W, breaking it down into M+1 dimensions into M parts for independent computation. We can similarly consider it as being caused by d y M-order tensors Composition, for each W k Tensors can also be decomposed using a similar method as tensors h, as shown below:

[0090]

[0091] The smallest value of R that makes the equation true can be called the rank of the tensor, which is the set of two-dimensional vectors within it. It becomes a low-rank decomposition factor of the original tensor.

[0092] In the decomposition process, we first start the calculation using a fixed rank value r, and then use r factorization factors. By parameterizing the model, these factors can reconstruct W. k The low-rank form can also be used to recombine and splice to form low-rank factor combinations for a specific mode, as shown in the following formula:

[0093]

[0094] After fixing the mode m, the factor set in a single rank is d. y There are _ ... The corresponding low-rank tensor recovery method is shown in the following formula:

[0095]

[0096] The corresponding formula for calculating the original tensor product is as follows:

[0097]

[0098] The calculation process of the original tensor product is graphically illustrated in the appendix. Figure 3 As shown, a dual-modal fusion process is chosen for demonstration because for higher modal processes, the W matrix dimension will reach four dimensions or more, which will lead to difficulties in visualization and comparison.

[0099] The third step involved optimizing the computational process. To further improve computational efficiency and reduce the risk of overfitting, the computation tensors were decomposed and the computation order was adjusted, thus completing performance optimization. The computational structure diagram is shown below. Figure 4 The optimization process for the structure shown is illustrated by the following formula:

[0100]

[0101] The operator ∧ is defined as performing element-wise multiplication of the matrices within the upper and lower bounds. When two modalities are fused, it can be expressed in the following transformed form:

[0102]

[0103] At the code implementation level, to facilitate the computation process and to use high-dimensional vectors for model parameterization with the same computational complexity to preserve more complete global semantic information, this study reversed the computation order. The reversed calculation formula is shown below:

[0104]

[0105] Its structure diagram under the state of fusion of three modes is shown in the attached figure. Figure 5 As shown, after the calculation of different modal decomposition groups of the low-rank tensor, their overall information is input into different attention modules. By mapping the fused modal information through consistent feature learning, cross-modal information representation vectors under the basic feature information of each modality are obtained.

[0106] Through this low-rank tensor product calculation and fusion module, the study performs feature fusion calculation on two cross-modal temporal feature embedding vectors and text vectors. This method is used to mine the correlation between global information of the modality and dynamic information of each modality and the time dimension.

[0107] Step 3: Construct a cross-modal interaction feature representation module based on a cross-structured architecture;

[0108] To address the need to extend the dual-domain interaction features of "text-visual" and "text-audio" to multimodal global semantics, a cross-modal interaction feature representation module based on a cross-structured architecture is constructed. Its model structure is shown in the attached figure. Figure 6As shown, through a cross-structured architecture and a multi-head self-attention mechanism, common semantics among multiple modalities are fully captured.

[0109] The core process of the module is as follows:

[0110] The first step involves maintaining the independence of the visual, audio, and text modalities during the input phase. Each modality generates a query vector (Q), key-value vector (K), and value vector (V) separately, avoiding the loss of modal information caused by direct concatenation. The second step involves concatenating the three attention vectors and performing unified multi-head self-attention computation. After resizing the computation results, independent residual connections are made with the original inputs of each modality, reducing computational resource consumption and preventing the model from getting trapped in local optima. By analyzing the attention computation results by modality partitioning, the off-diagonal regions represent the feature fusion results of the two modalities, ensuring that at least one set of modal interaction information is obtained in a single round of attention computation. The third step involves cross-domain semantic extension through the encoder and decoder: the encoder's computation process is shown in the following formula:

[0111] f T→A =CATransformerEncoder(f A ,f T )

[0112] Its input audio features f A With text features f T First, a cross-modal domain calculation will be performed to obtain an f. T→A Features, and in the decoder calculation in the following formula, it will f T→A Cross-modal semantic information is obtained by using text modal information fT as a fixed representation of cross-modal semantics and embedding it with "text-visual" modal features f. T→V Interacting with each other, and thus obtaining features f that include text-based modality, shifting from "text-audio" to "text-visual" interaction. A(T)→V(T) .

[0113] f A(T)→V(T) =CATransformerDecoder(f T→A ,f T→A )

[0114] The model constructed in this step takes the text modality feature vector and the cross-modality feature embedding vector as input to this part. Its purpose is to perform feature fusion modeling on the three sets of feature vectors that tend to embed semantic information through this part of the model, so as to fully explore the common semantic content between the various modalities.

[0115] Step 4: Construct a multi-level feature fusion module based on a state-space sequence architecture;

[0116] To efficiently integrate the feature vectors output by the preceding modules, a Cross-Mamba multimodal feature fusion module based on a state-space sequence architecture is constructed. This module leverages the selective state-space characteristics of the module to enhance long sequence modeling capabilities and cross-modal dynamic response performance.

[0117] The core process of the module is as follows:

[0118] The first step is to construct the basic framework of the state-space model, whose core equations are the continuous-time state equation and the output equation, as shown in the following formulas:

[0119] h`(t) = A*h(t) + B*x(t)

[0120] y(t)=C*h(t)+D*x(t)

[0121] The two sets of equations mentioned above are also the core of the state-space model. In the early algorithm design, the four matrices ABCD are the learnable parameters of the model, and these four matrices remain fixed after the model's learning phase. The learning process of this model can be understood as learning and adjusting the parameters in the four matrices ABCD to represent the internal hidden state, h(t), so as to obtain a sequence-to-sequence task from input to output. The model module is written and encapsulated using an input-to-output information transfer method, so the structure of the state-space model is as shown in the attached figure. Figure 7 As shown, the input sequence x(t) first passes through matrix B. Then, the output of matrix B is summed with the output of matrix A to obtain the latent state representation h(t). The input to matrix A is h(t) from the previous input process. The latent state representation h(t), after being input into matrix C, is summed with the result of the input sequence x(t) passing through matrix D to obtain the final predicted output y(t). In this model structure, it can be seen that the process from the input sequence x(t) through matrix D to the predicted output y(t) is similar to the residual connection proposed in ResNet. Therefore, in the construction of the state space, it is usually simplified to a core model constructed using three matrices A, B, and C.

[0122] The second step is to optimize the performance of the state-space model. The state-space model is improved using the HiPPO matrix, which is initialized during training by constructing a specific expression, as shown below:

[0123]

[0124] A set of orthogonal polynomials is used to capture near-sequence features while compressing and fitting long-term sequence features. The matrix size is related to the number of polynomials, ensuring that more long-range dependency information is accommodated in a limited storage space. For multi-dimensional input data, the state-space model does not choose to increase the matrix dimension to avoid an exponential increase in training parameters. Instead, one dimension of the matrix is ​​designed as the input data dimension, and the state-space group is reused by the matrix in the output equation, thus maintaining linear computational complexity. Furthermore, since the model achieves sequence compression and reconstruction from input to output, its training process is transformed into calculating the similarity loss between input and output.

[0125] The third step, the Cross-Mamba model fusion process, leverages its more efficient ability to model sequence data and its effective data-driven capabilities to efficiently combine the features obtained from the fusion of two sets of multimodal data. Its structure during application is shown in the attached figure. Figure 8 As shown, the three modules first integrate the inputs in terms of length, then concatenate the features in the direction of the number of feature channels. First, matrix B is used to obtain the selection input for each feature dimension, and then the input of that dimension and matrix A are used to complete the transition from each state to the hidden state h. k The update operation involves matrix Δ, which determines the importance of the input data for each channel based on its characteristics, and indirectly optimizes matrix A through matrix Δ to complete the update operation in the hidden state h. k The core content is retained, and the information from each state-space model is summarized and output through the output matrix C to obtain the output sequence y. k Meanwhile, in the process of summarizing the output, the fused features of each modality are further fused.

[0126] And in obtaining the output sequence y k Then, it will map the results through a linear output layer to obtain the output results of various task indicators. In our final output results, we designed two output formats, namely "2+3+5+1" and "2+3+7+1", to complete multiple tasks such as binary classification, three-class classification, five-class classification, seven-class classification and regression value output, depending on the selected dataset. This output format will be applied to subsequent multi-task training to complete the overall optimization of the model.

[0127] Example:

[0128] This invention was tested on four challenging multimodal sentiment datasets, including the MOSI dataset, the MOSEI dataset, and the SIMSv2 dataset.

[0129] The MOSI dataset is extracted from YouTube opinion-sharing videos, segmented by single sentences, with each sentence serving as a single sample. The data requires only one narrator in the frame, preferably facing the camera directly, with no restrictions on recording equipment, distance, background, or lighting conditions. The dataset contains 93 original videos ranging from 2 to 5 minutes in length, involving 89 narrators (41 women and 48 men), primarily aged 20 to 30, representing multiple ethnic groups, but all videos are in English. 2199 sentiment analysis sentences were extracted, with an average sentence length of 4.2 seconds and a total of 26295 words. 3107 words appeared only once, and 557 words appeared more than ten times. For sentiment tagging, the MOSI dataset provides multimodal sentiment analysis. Data annotation was completed by five people, with classification results in a seven-category system ({-3, -2, -1, 0, 1, 2, 3}), which can be further divided into seven-category, three-category, and two-category indicators. The dataset can be downloaded from: https: / / github.com / AashitaK / CMU-MultimodalSDK.

[0130] The MOSEI dataset, proposed by the MOSI team, is a larger-scale multimodal sentiment dataset. It maintains the video content constraints of MOSI, ensuring the inclusion of text, audio, and facial visual information. Unlike MOSI, which only uses "#vlog" for filtering, MOSEI employs 250 commonly used YouTube hashtags for broader filtering and limits each creator's channel to a maximum of 10 videos. Ultimately, from 5000 videos by 1000 narrators (57% male, 43% female), 3228 videos were selected by 14 experts, covering commentary, debate, and consultation scenarios. This dataset averages 7.3 sentence samples per video, with each sample lasting 7.28 seconds and containing 447,143 words (23,026 of which are uncommon). High-frequency words (appearing more than 10 / 20 / 50 times) number 3,413, 1,971, and 888, respectively. As the largest multimodal sentiment dataset for a single language to date, MOSEI boasts a total sequence duration of 65 hours and 53 minutes, far exceeding MOSI's 2 hours and 36 minutes, making it a crucial benchmark for validating video sentiment analysis tasks. The dataset can be downloaded from: https: / / github.com / AashitaK / CMU-MultimodalSDK.

[0131] The SIMSv2 dataset is an improvement upon SIMS, including more video types such as sitcoms, interviews, talk shows, and video blogs. It also adds videos with a short side resolution of at least 720p and supplements video samples from oblique or side views. The dataset size has expanded to 4402 entries, involving 145 video narrators, with individual sample lengths reaching up to 20 seconds. The common parts with SIMS have been re-annotated, and the labeled data has been improved, including setting up a 7-person labeling group (removing the highest and lowest scores to improve consistency) and updating the modality isolation strategy for single-modality labeling tasks (such as injecting noise to obscure speech content in audio labeling tasks). The dataset can be downloaded at: https: / / github.com / thuiar / ch-sims-v2.

[0132] Example 1:

[0133] This invention was tested on the MOSI dataset. For classification tasks, binary classification accuracy (Acc2), F1 score (F1-Score), three-class classification accuracy (Acc3), and five-class classification accuracy (Acc5) or seven-class classification accuracy (Acc7) were used as evaluation metrics. For regression tasks, correlation coefficient (Corr) and mean absolute error (MAE) were used as evaluation metrics. The performance comparison results of the model of this invention with the baseline model are shown in Table 1.

[0134] Table 1 Experimental Results of Multi-Level Fusion Model - MOSI

[0135]

[0136]

[0137] Table 1 shows the performance comparison between the proposed method and the baseline model on the MOSI dataset. A lower mean absolute error (MAE) value indicates better model performance, while higher values ​​for all other metrics also indicate better model performance. The corresponding bar charts are attached. Figure 9 As shown.

[0138] The performance metrics for each segment of a long-sample, multi-sentence test from a single narrator are presented to compare the performance and stability of different models when emotions change continuously in a single scenario. A comparison of the test sample's long-sequence segmentation results for a single scenario is shown below. Figure 10 As shown.

[0139] In this diagram, C2, C3, and C5 represent different classification tasks, while RE represents the regression task. The data in the ground truth are the one-hot encoded results. Subsequent model classification tasks output the confidence score at the corresponding output header position of the ground truth. By outputting these confidence scores, the model's fit to the distribution of the ground truth data can be demonstrated. Therefore, for the data in the diagram, the closer the output results of C2, C3, and C5 are to 1.00, the better. For the regression task, the closer to the ground truth, the better. Furthermore, incorrect output results for classification tasks are highlighted in blue, while optimal results are highlighted in red.

[0140] Example 2:

[0141] This invention was also tested on the MOSEI and SIMSv2 datasets. For classification tasks, binary classification accuracy (Acc2), F1 score (F1-Score), ternary classification accuracy (Acc3), and septial classification accuracy (Acc7) were used as evaluation metrics. For regression tasks, correlation coefficient (Corr) and mean absolute error (MAE) were used as evaluation metrics. The performance comparison results of the model of this invention with the baseline model are shown in Table 2.

[0142] Table 2 Experimental Results of Multi-Level Fusion Model - MOSEI

[0143]

[0144]

[0145] Table 2 shows the evaluation index calculation results of the proposed method and the baseline model in the MOSEI dataset, and the corresponding bar chart is attached. Figure 11 .

[0146] The performance metrics for each segment of a long-sample, multi-sentence test from a single narrator are presented to compare the performance and stability of different models when emotions change continuously in a single scenario. A comparison of the test sample's long-sequence segmentation results for a single scenario is shown below. Figure 12 As shown.

[0147] exist Figure 10 and Figure 12 In our study, we selected a set of samples with sentiment ranging from weakly positive to strongly positive and then to neutral, as well as samples where the last sentence was determined to be negative within an overall neutral-to-positive range. In both sets of samples, we observed that other methods might produce inconsistent outputs or significant deviations in samples with neutral results or abrupt changes in sentiment. Our proposed model, however, consistently delivers accurate outputs for all classification indices under these conditions, and its regression indices more closely approximate the true values.

[0148] Example 3:

[0149] This invention was also tested on the SIMSv2 dataset. For the classification task, binary classification accuracy (Acc2), F1 score (F1-Score), ternary classification accuracy (Acc3), and quinary classification accuracy (Acc5) were used as evaluation metrics. For the regression task, correlation coefficient (Corr) and mean absolute error (MAE) were used as evaluation metrics. The performance comparison results of the model of this invention with the baseline model are shown in Table 3.

[0150] Table 3 Experimental Results of Multi-Level Fusion Model - SIMSv2

[0151]

[0152] Table 3 shows the evaluation index calculation results of the proposed method and the baseline model in the SIMSv2 dataset, and the corresponding bar chart is attached. Figure 13 As shown.

[0153] Compared to the relatively smaller MOSI dataset, which is a similarly sized dataset in the same language, the overall evaluation metrics of this dataset have declined. This is mainly because the dataset contains a wider variety of video content and includes more narrators and other information, making the data modeling and fitting process more difficult. However, this also means that the modeled results are more practically meaningful and have greater generalizability.

[0154] To illustrate the performance gap between the model and several other mainstream Transformer-based methods in more detail, Example 3 presents the performance metrics for various tasks using a subset of samples, as shown in the attached figures. Figure 14 As shown.

[0155] In the appendix Figure 14 In the second sample, we can see that the MMIM and MulT models may have over-relyed on textual modality information, misclassifying it as a negative result. In other sample demonstrations, we can observe that our proposed method, compared to other models, better fits the data distribution, thus achieving higher overall accuracy and robustness.

[0156] Results Analysis: Experiments were conducted on the MOSI, MOSEI, and SIMSv2 datasets to verify the results. The experimental results show that the method of the present invention outperforms the comparative model in terms of overall performance on each dataset. The experiments demonstrate that multi-stage and multi-angle fusion of different feature data can effectively improve the performance of multimodal data fusion models.

Claims

1. A multimodal fusion identification and modeling method based on cross-structured state-space sequences, characterized in that, Includes the following steps: Step 1: Construct a multimodal representation dictionary module based on consistent feature learning; A module for preprocessing high-dimensional features from visual, audio, and text modalities, and then performing joint high-dimensional feature embedding and feature extraction after encoding all three modalities using ReFormer; Step 2: Construct a cross-modal information representation module based on attention-based low-rank operations; Efficient fusion of multimodal information is achieved through low-rank tensor computation, which reduces computational cost and hyperparameter quantity while ensuring model performance; Step 3: Construct a cross-modal interaction feature representation module based on a cross-structured architecture; By employing a cross-structured architecture and a multi-head self-attention mechanism, common semantics among multiple modalities can be captured. Step 4: Construct a multi-level feature fusion module based on a state-space sequence architecture; Leveraging selective state-space characteristics enhances long sequence modeling capabilities and cross-modal dynamic response performance.

2. The multimodal fusion identification and modeling method based on cross-structured state space sequences as described in claim 1, characterized in that, Step 1 includes: Step 11: Dimensionality reduction of the three-modal high-dimensional features is performed using the TCN temporal convolution model to obtain low-dimensional semantic space mapping vectors. The corresponding indices of the two modalities used in the calculation are defined as 1 and 2, respectively. H1 and H2 are the original input high-dimensional features, L1 and L2 are the output low-dimensional semantic space mapping vectors, and p1 and p2 are the network parameters of the TCN1 and TCN2 models, respectively. The calculation process is as follows: L1 = TCN1(H1; p1) L2 = TCN2(H2; p2) Step 12: After obtaining the dimensionality-reduced modal features, calculate the correlation coefficients between them. Based on the relationship between the covariance and variance quantization feature sets, the correlation coefficient calculation method is as follows: Where corr(L1,L2) is the correlation coefficient. Here, we define the optimal network parameters for two computational networks. `cov(L1,L2)` represents the covariance of the low-dimensional semantic space mapping vectors L1 and L2, while `var(L1)` and `var(L2)` represent the variances of L1 and L2, respectively. By analyzing the covariance and variance, we can determine the correlation between the two sets of features. In this module application, we use TCN. t TCN v and TCN a When training three models simultaneously, the original optimization objective is calculated as follows: Step 13: Generate the gating vector G corresponding to each mode through linear transformation. t G v and G a Construct a dynamic gating mechanism; Step 14: Optimize the dimensions of the gated feature vectors, add absolute position vectors of each feature point to preserve temporal information, sort the feature activation values ​​from high to low, retain the core dimensions according to the needs of subsequent modules, and restore the position order.

3. The multimodal fusion identification and modeling method based on cross-structured state space sequences as described in claim 1, characterized in that, Step 2 includes: Step 21: Define the abstract representation of multimodal fusion; When using a linear layer to output multimodal fusion results, the output formula is as follows: Here, the vector dimension of H is M, and the weights of its linear output layer matrix W are M+1 dimensional. Step 22: Decompose the original weight vector W, dividing the M+1 dimensions into M parts for independent calculation. Consider it as being made by d y M-order tensors Composition, for each W k Tensors can also be decomposed using a similar method as tensors h, with the following decomposition formula: The smallest value of R that makes the equation true is called the rank of the tensor, which is the set of two-dimensional vectors. It becomes a low-rank factor of the original tensor; Step 23: Optimize the calculation process.

4. The multimodal fusion identification and modeling method based on cross-structured state space sequences as described in claim 1, characterized in that, Step 3 includes: Step 31: During the input phase, the visual, audio, and text modalities remain independent, and each modality generates a query vector Q, a key-value vector K, and a value vector V, respectively. Step 32: After concatenating the three sets of attention vectors, perform unified multi-head self-attention calculation. After the calculation results are sized, perform independent residual connections with the original inputs of each modality. By analyzing the attention calculation results by modality partition, the off-diagonal region is the feature fusion result of the two modalities, so as to obtain at least one set of modal interaction information in a single round of attention calculation. Step 33: Cross-domain semantic extension is completed through the encoder and decoder. The encoder's calculation process is as follows: f T→A =CATransformerEncoder(f A ,f T ) Its input audio features f A With text features f T First, a cross-modal domain calculation will be performed to obtain an f. T→A Features, and in the decoder calculation in the following formula, it will f T→A Cross-modal semantic information in the text is obtained through text modal information f T As a cross-modal semantic fixed representation and "text-visual" embedded modal feature f T→V Interacting with the content results in a text-based modality.

5. The multimodal fusion identification and modeling method based on cross-structured state space sequences as described in claim 4, characterized in that, Step 4 includes: Step 41: Construct the basic framework of the state-space model, whose core equations are the continuous-time state equation and output equation as follows: h`(t) = A*h(t) + B*x(t) y(t)=C*h(t)+D*x(t) First, the input sequence x(t) passes through matrix B. Then, the output of matrix B is summed with the output of matrix A to obtain the latent state representation h(t). The input of matrix A is h(t) from the previous input process. After the latent state representation h(t) is input into matrix C, it is summed with the result of the input sequence x(t) passing through matrix D to obtain the final predicted output y(t). Step 42: Optimize the performance of the state-space model. The state-space model is improved using the HiPPO matrix, which is initialized for training by constructing a specific expression. Step 43: Cross-Mamba model fusion process, used to combine the features after fusion of two sets of multimodal features. Its three module inputs are first integrated in length, and features are concatenated in the direction of the number of feature channels. First, the selection input for each feature dimension is obtained through its matrix B, and the input of this dimension and matrix A are used to complete the transformation from each state to the hidden state h. k The update operation involves matrix Δ, which determines the importance of the input data for each channel based on its characteristics, and indirectly optimizes matrix A through matrix Δ to complete the update operation in the hidden state h. k The core content is retained, and the information from each state-space model is summarized and output through the output matrix C to obtain the output sequence y. k After obtaining the output sequence y k Then, the results are mapped through a linear output layer to obtain the output results of various task indicators.

Images