A multimodal emotion recognition method based on cross-modal interaction using a state-space model

Abstract

This invention discloses a multimodal emotion recognition method based on cross-modal interaction using a state-space model, belonging to the field of multimodal emotion recognition technology. The steps include: First, extracting low-level features from speech and text, and using a diagonal version of the structured state-space sequence model (S4D) to extract S4D features, thereby fully utilizing its unique sequence modeling capabilities and robust feature extraction capabilities to effectively capture the complex temporal correlations of sound and text data; Second, to fuse the modal features of sound and text, this invention proposes multiple cross-modal fusion strategies to dynamically learn the weights between modalities; Furthermore, to better distinguish different categories of emotion, this invention designs a triplet loss and combines it with cross-entropy loss to improve the model's accuracy in emotion classification; Finally, ablation experiments were conducted to verify the effectiveness of the method. This invention effectively improves the performance and generalization ability of multimodal emotion recognition systems.

Description

Technical Field

[0001] This invention belongs to the field of multimodal emotion recognition technology, specifically relating to a multimodal emotion recognition method based on cross-modal interaction using a state-space model. Background Technology

[0002] Emotion plays a crucial role in information transmission and interpersonal communication. In recent years, multimodal emotion recognition has received increasing attention and has been widely applied in fields such as human-computer interaction, intelligent customer service, and healthcare. Accurately detecting emotions from multimodal data can greatly enhance user experience and significantly improve system performance. Multimodal emotion recognition combines information from multiple modalities, including speech, images, and text, to identify and analyze emotional states. Compared to the potential for incomplete information or noise interference in single-modal data, multimodal fusion provides more comprehensive and accurate emotional information.

[0003] Feature extraction is a crucial step in multimodal emotion recognition tasks, primarily aimed at acquiring features that effectively represent different emotions. In recent years, researchers have mainly relied on traditional deep sequence models, such as Long Short-Term Memory (LSTMs), Convolutional Neural Networks (CNNs), and transformers, to extract features from different modalities to accomplish emotion recognition tasks. However, these models may encounter vanishing or exploding gradients when processing long sequences, making it difficult for them to capture long-term dependencies. Yet, in emotion analysis, considering the long-term evolution of a speaker's emotional state is often crucial. Furthermore, these models require processing each element of the sequence step-by-step, leading to high computational costs and slow training and inference speeds when processing long sequences. The introduction of State-Space Models (SSMs) has proven superior to traditional deep sequence models. Gu et al. proposed the Structured State-Space Sequence Model (S4) and its variant (the diagonal version of S4, S4D), which can process the entire sequence in parallel, more effectively capture long-term dependencies in the sequence, and exhibit higher computational efficiency. The S4 method can handle variable-length input sequences, adapting to different lengths of different text or speech segments. Therefore, compared with traditional deep sequence models, state-space models have advantages in sentiment analysis, as they can more accurately capture the complexity and long-term relevance of sentiment expression.

[0004] The core challenge of multimodal emotion recognition lies in how to efficiently integrate information from different modalities for emotion identification. Since data from different modalities may have varying feature representations and expressions, effective integration is crucial. Intramodal interactions aim to reveal and enhance the correlations and temporal features within each modality, thereby improving the representational power of a single modality. Intermodal interactions, on the other hand, leverage complementary information between different modalities to improve emotion recognition performance through effective information transfer mechanisms. In existing technologies, Yoon et al. constructed a multimodal dual-loop encoder model, learning speech and text representations and combining them for emotion classification. However, this method does not explicitly capture multimodal interactions. Tsai et al. used a multi-head attention mechanism to calculate the correlations between modalities at different locations and in subspaces, directly extracting multimodal representations from unaligned multimodal sequence data; however, they neglected the correlations of specific features within each modality.

[0005] In daily life, speech and text are important means of emotional expression, providing crucial discriminative information for multimodal emotion recognition systems. However, due to the heterogeneity of speech and text information in real-world applications, multimodal emotion recognition technology faces the critical challenge of effectively extracting key and complementary information from speech and text data. Furthermore, the effective fusion of the extracted speech and text information is essential for the performance of multimodal emotion recognition systems, particularly in ensuring the system's accuracy and robustness. Summary of the Invention

[0006] This invention aims to overcome the shortcomings and deficiencies of existing technologies by proposing a multimodal emotion recognition method based on cross-modal interaction using a state-space model. It extracts S4D features by employing a diagonal version of the Structured State-Space Sequence (S4D) model, fully utilizing its independent sequence modeling and robust feature extraction capabilities to effectively capture the complex temporal correlations of acoustic and textual data. Furthermore, it dynamically learns the weights between modalities through various cross-modal fusion strategies. In addition, to better distinguish different categories of emotions, the proposed method designs a triple loss model and combines it with cross-entropy loss to improve the model's accuracy in emotion classification. This innovative method provides a new solution for multimodal emotion recognition and has broad application prospects.

[0007] The technical solution adopted by this invention to solve its technical problem is: a multimodal emotion recognition method based on cross-modal interaction using a state-space model, the method comprising the following steps:

[0008] Step 1: Perform low-level feature extraction on the original input speech and text modalities respectively;

[0009] Step 2: Input the extracted low-level features into the channel grouping feature extraction network to obtain high-level features. The network consists of different numbers of one-dimensional convolutional layers and channel grouping convolutional layers.

[0010] Step 3: Input the extracted high-level features of speech and text into the cross-modal feature mapping module respectively, and learn the interaction between acoustic and text features in detail to extract rich cross-modal information from speech segments and sentences;

[0011] Step 4: Process the low-level features of speech and text separately using two-layer diagonalized structured state space sequence (S4D) modules to extract higher-level and more robust S4D feature representations, and then concatenate them to obtain S4D multimodal features.

[0012] Step 5: Input the S4D multimodal features and high-level features of speech and text into the cross-modal interactive attention module to adaptively enhance the modal sequence representation;

[0013] Step 6: Fuse the intrinsic features of the two modalities, S4D multimodal features, and intermodal interaction features together. Input the fused feature vector into a bidirectional LSTM layer, and use a self-attention mechanism to select key sentiment features. Finally, output the sentiment classification result through the fully connected layer of the model;

[0014] Step 7: Use the designed triplet loss function combined with cross-entropy loss to constrain model convergence and improve the accuracy of the model in sentiment classification;

[0015] Step 8: Evaluate the performance of the proposed multimodal emotion recognition method based on cross-modal interaction using a state-space model.

[0016] Furthermore, the specific steps of step 1 include:

[0017] Step 1-1: For text embedding X t We use a 300-dimensional pre-trained GloVe embedding model to map words to fixed-length vectors for each utterance transcription. The dimension of the linear operator used to fine-tune the pre-trained word embeddings is set to 256.

[0018] Steps 1-2: For acoustic embedding X a First, the audio signal of the data sample is converted into 25ms frames with a frame shift of 10ms. Then, a Python library is used to extract 40-dimensional filter bank features and 13-dimensional MFCC features from each frame.

[0019] Furthermore, the specific steps of step 2 include:

[0020] Step 2-1: Input low-level text features X tThe channel-group convolution module consists of one one-dimensional convolutional layer and one channel-group convolutional layer, with kernel sizes of 3 and 2 respectively, a stride of 1 for each kernel, and 4 groups in the channel-group convolutional layer. The one-dimensional convolutional layer comprises one-dimensional convolution, a GeLU activation function, and max pooling. The channel-group convolutional layer comprises group convolution, a GeLU activation function, max pooling, and layer normalization.

[0021] Step 2-2: Input low-level speech features X a The next step is the channel grouping convolution module, which consists of two one-dimensional convolutional layers and one channel grouping convolutional layer. The kernel sizes are 3, 2, and 3, respectively, and the stride of each kernel is 1. The number of groups in the channel grouping convolutional layer is set to 4. The composition of the one-dimensional convolutional layer and the channel grouping convolutional layer is the same as in step 2-1.

[0022] Furthermore, step 3 specifically includes the following steps:

[0023] Step 3-1: Employ a dimension-dependent gating mechanism with a sigmoid activation function to learn X” output from Step 2. a and X' t 'Weight parameters of the embedding dimension,'

[0024] M a =σ(W a X' t '),M t =σ(W t X' a '),

[0025] Where σ(·) represents the sigmoid function. and These are linear projection operators for speech and text representations, used to map input features to the corresponding representation spaces, M. a and M t It is a mapping matrix, serving as weights to adapt to the corresponding features;

[0026] Step 3-2: Adaptively calibrate speech and text representations using a cross-modal mapping matrix.

[0027] X″′ a =M a +X″ a ,X″′ t =M t +X″ t .

[0028] Furthermore, step 4 specifically includes the following steps:

[0029] Step 4-1: Replace traditional deep sequence networks such as LSTM with two-layer S4D modules for multimodal feature extraction. The S4D module includes residual connections, dropout, one-dimensional convolutional layers, gated linear units (GLUs), and layer normalization.

[0030] Step 4-2: Concatenate the feature sequences of speech and text modalities processed by their respective S4D modules to obtain the S4D multimodal feature X. c .

[0031] Furthermore, step 5 specifically includes the following steps:

[0032] Step 5-1: Use the features of the sound and text modalities as query vectors Q and the S4D multimodal features as key and value vectors V. Perform cross-modal attention mechanism operations to dynamically adjust the weights of sound and text information. The cross-modal attention mechanism module includes a multi-head attention mechanism, a fully connected layer, a skip connection, and layer normalization. The number of heads in the multi-head attention mechanism is 8.

[0033] Step 5-2: Convert the output CA(Q') of the cross-modal attention mechanism a ) and CA(Q' t The inputs are fed into a single-output gated fusion layer, where a sigmoid function is applied to process the inputs and generate a gate vector. This gate vector is used to adjust the weights of each modality feature in the subsequent feature fusion process. A linear transformation weight matrix is ​​used to perform a linear transformation on the input, and then the result is multiplied element-wise with the gate vector to obtain the fused feature F. a and F t .

[0034] Furthermore, step 6 specifically includes the following steps:

[0035] Step 6-1: Extract the intrinsic features X” of the speech-text modality n '、S4D Multimodal Features X c Interaction features F between modes n The features are fused, where n∈{a,t}, and then the fused feature vector is input into a bidirectional LSTM network to capture long-term dependencies and contextual information.

[0036] Step 6-2: A self-attention mechanism is added after the bidirectional LSTM network to filter key features. Then, the order information of all hidden states is aggregated using a max pooling layer.

[0037] Step 6-3: The aggregated information is used to perform the final sentiment prediction through a two-layer fully connected feedforward network.

[0038] Furthermore, step 7 specifically includes the following steps:

[0039] Step 7-1: Define each data instance as a triple, where the anchor sample y represents the predicted sentiment category, where positive sample y is the correct sentiment label and negative sample e is a randomly selected incorrect sentiment label.

[0040] Step 7-2: Use a neural network to convert each sample into a corresponding embedding vector. The cosine similarity f(y) and f(e) is calculated to minimize the distance between positive samples and anchor points while maximizing the distance between negative samples and anchor points. The triplet loss function is expressed as:

[0041]

[0042] in Representing vectors The dot product with f(e), Representing vectors The norm of , where m is the boundary value hyperparameter;

[0043] Step 7-3: Combine triplet loss with cross-entropy loss to optimize the accuracy of sentiment prediction during training.

[0044] Furthermore, the specific method for performance evaluation of the proposed multimodal emotion recognition method based on state-space model cross-modal interaction in step 8 includes the following steps:

[0045] Step 8-1: Compare a new multimodal emotion recognition method based on cross-modal interaction using a state-space model with existing multimodal emotion recognition methods to verify its overall performance;

[0046] Step 8-2: Compare the state-space model-based multimodal emotion recognition method with the traditional sequence model-based multimodal emotion recognition method (e.g., LSTM network) to verify the effectiveness of the state-space model-based speech emotion recognition.

[0047] Step 8-3: Remove the cross-modal feature mapping module and the cross-modal interaction attention module respectively, and conduct ablation experiments to verify the effectiveness of the cross-modal interaction fusion strategy in this invention;

[0048] Step 8-4: Remove the triplet loss function and conduct ablation experiments to verify its importance in model training and its impact on performance.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] 1. This invention proposes a multimodal emotion recognition based on a state-space model. It utilizes the S4D model to extract and connect features from both audio and text modalities to form powerful S4D multimodal features. This enables the model to dynamically capture the temporal relationship and changes of audio and text signals, thereby enhancing the model's adaptability and generalization ability.

[0051] 2. This invention introduces a cross-modal feature mapping module and a cross-modal interactive attention module, enabling the model to fully capture the intrinsic correlation between acoustic features, text features and S4D multimodal features, and make full use of the complementary information between these features.

[0052] 3. This invention designs a triple loss function, which is combined with cross-entropy loss during the training process to ensure the closeness between sentiment categories and to better distinguish different categories of sentiment. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the operation process of the present invention.

[0054] Figure 2 This is a block diagram of the channel grouping feature extraction module of the present invention.

[0055] Figure 3 This is a block diagram of the S4D state space model module of the present invention.

[0056] Figure 4 This is a block diagram of the cross-modal feature mapping module of the present invention.

[0057] Figure 5 This is a block diagram of the cross-modal interactive attention module of the present invention. Detailed Implementation

[0058] The invention will now be described in further detail with reference to the accompanying drawings.

[0059] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings, tables, and specific embodiments. It should be noted that, to avoid obscuring the invention with unnecessary details, only structures and / or processing steps closely related to the solutions of this invention are shown in the drawings, while other details not closely related to this invention are omitted.

[0060] Additionally, it should be noted that the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0061] like Figures 1 to 5 As shown, this invention proposes a multimodal emotion recognition method based on cross-modal interaction using a state-space model. By applying the state-space model to multimodal emotion recognition to extract high-level features, a higher-performance multimodal emotion recognition system can be constructed. The method specifically includes the following steps:

[0062] Step 1: Perform low-level feature extraction on the original input speech and text modalities respectively. The dataset used in this invention is the IEMOCAP sentiment database, which contains 5 different conversations, each performed by two actors (one male and one female), with approximately 12 hours of speech data.

[0063] Step 1 specifically includes:

[0064] Step 1-1: For text embedding X t We use a 300-dimensional pre-trained GloVe embedding model to map words to fixed-length vectors for each utterance transcription. The dimension of the linear operator used to fine-tune the pre-trained word embeddings is set to 256.

[0065] Steps 1-2: For acoustic embedding X a First, the audio signal of the data sample is converted into 25ms frames with a frame shift of 10ms. Then, a Python library is used to extract 40-dimensional filter bank features and 13-dimensional MFCC features from each frame.

[0066] Step 2: Input the extracted low-level features into the channel grouping feature extraction network to obtain high-level features. This network consists of different numbers of one-dimensional convolutional layers and channel grouping convolutional layers.

[0067] Step 2 specifically includes:

[0068] Step 2-1: Input low-level text features X t The channel-group convolution module consists of one one-dimensional convolutional layer and one channel-group convolutional layer, with kernel sizes of 3 and 2 respectively. Each kernel has a stride of 1, and the channel-group convolutional layer has 4 groups. The one-dimensional convolutional layer consists of one-dimensional convolution, a GeLU activation function, and max pooling. The channel-group convolutional layer consists of group convolution, a GeLU activation function, max pooling, and layer normalization.

[0069] Step 2-2: Input low-level speech features X a The next module is the channel grouping convolution module, which consists of two one-dimensional convolutional layers and one channel grouping convolutional layer. The kernel sizes are 3, 2, and 3, respectively, and the stride of each kernel is 1. The number of groups in the channel grouping convolutional layer is set to 4. The components of the one-dimensional convolutional layer and the channel grouping convolutional layer are consistent with those described in step 2-1.

[0070] Step 3: Input the extracted high-level features of speech and text into the cross-modal feature mapping module respectively, and learn the interaction between acoustic and text features in detail to extract rich cross-modal information from speech segments and sentences.

[0071] Step 3 specifically includes:

[0072] Step 3-1: Use a dimension-dependent gating mechanism with a sigmoid activation function to learn X” output from Step 2. a and X' t 'Weight parameters of the embedding dimension,'

[0073] M a =σ(W a X' t '),M t =σ(W t X' a '),

[0074] Where σ(·) represents the sigmoid function. and These are linear projection operators for speech and text representations, used to map input features to the corresponding representation spaces, M. a and M t It is a mapping matrix, serving as weights to adapt to the corresponding features;

[0075] Step 3-2: Finally, the speech and text representations are adaptively calibrated using the cross-modal mapping matrix.

[0076] X″′ a =M a +X″ a ,X″′ t =M t +X″ t .

[0077] Step 4: Process the low-level features of speech and text separately using two-layer diagonalized structured state space sequence (S4D) modules to extract higher-level and more robust S4D feature representations, and then concatenate them to obtain S4D multimodal features.

[0078] Step 4 specifically includes:

[0079] Step 4-1: Replace traditional deep sequence networks such as LSTM with two-layer S4D modules for multimodal feature extraction. The S4D module includes residual connections, dropout, one-dimensional convolutional layers, gated linear units (GLUs), and layer normalization.

[0080] Step 4-2: Concatenate the feature sequences of speech and text modalities processed by their respective S4D modules to obtain the S4D multimodal feature X. c .

[0081] Step 5: Input the S4D multimodal features along with the high-level features of speech and text into the cross-modal interactive attention module to adaptively enhance the modal sequence representation.

[0082] Step 5 specifically includes:

[0083] Step 5-1: Use the features of the sound and text modalities as query vectors Q and the S4D multimodal features as key and value vectors V. Perform cross-modal attention mechanism operations to dynamically adjust the weights of sound and text information. The cross-modal attention mechanism module includes a multi-head attention mechanism, a fully connected layer, a skip connection, and layer normalization. The number of heads in the multi-head attention mechanism is 8.

[0084] Step 5-2: Convert the output CA(Q') of the cross-modal attention mechanism a ) and CA(Q' t The inputs are fed into a single-output gated fusion layer, where a sigmoid function is applied to process the inputs and generate a gate vector. This gate vector is used to adjust the weights of each modality feature in the subsequent feature fusion process. A linear transformation is performed on the input using a linear transformation weight matrix, and then the result is multiplied element-wise with the gate vector to obtain the fused feature F. a and F t .

[0085] Step 6: Fuse the intrinsic features of the two modalities, S4D multimodal features, and intermodal interaction features together. Input the fused feature vector into a bidirectional LSTM layer, and use a self-attention mechanism to select key sentiment features. Finally, output the sentiment classification result through the fully connected layers of the model.

[0086] Step 6 specifically includes:

[0087] Step 6-1: Extract the intrinsic features X” of the speech-text modality n '、S4D Multimodal Features X c Interaction features F between modes n The features are fused, where n∈{a,t}, and then the fused feature vector is input into a bidirectional LSTM network to capture long-term dependencies and contextual information.

[0088] Step 6-2: A self-attention mechanism is added after the bidirectional LSTM network to filter key features. Then, the order information of all hidden states is aggregated using a max pooling layer.

[0089] Step 6-3: The aggregated information is used to perform the final sentiment prediction through a two-layer fully connected feedforward network.

[0090] Step 7: Use the designed triplet loss function combined with cross-entropy loss to constrain model convergence and improve the accuracy of the model in sentiment classification.

[0091] Step 7 specifically includes:

[0092] Step 7-1: Define each data instance as a triple, where the anchor sample y represents the predicted sentiment category, where positive sample y is the correct sentiment label and negative sample e is a randomly selected incorrect sentiment label.

[0093] Step 7-2: Use a neural network to convert each sample into a corresponding embedding vector. The cosine similarity f(y) and f(e) is calculated to minimize the distance between positive samples and anchor points while maximizing the distance between negative samples and anchor points. The triplet loss function is expressed as:

[0094]

[0095] in Representing vectors The dot product with f(e), Representing vectors The norm of , where m is the boundary value hyperparameter;

[0096] Step 7-3: Combine triplet loss with cross-entropy loss to optimize the accuracy of sentiment prediction during training.

[0097] Step 8: Evaluate the performance of the proposed multimodal emotion recognition method based on cross-modal interaction using a state-space model.

[0098] Step 8 specifically includes:

[0099] Step 8-1: Compare a new multimodal emotion recognition method based on cross-modal interaction using a state-space model with existing multimodal emotion recognition methods to verify its overall performance;

[0100] Step 8-2: Compare the multimodal emotion recognition method based on the state space model (S4D) with the multimodal emotion recognition method based on the traditional sequence model (e.g., LSTM network) to verify the effectiveness of the speech emotion recognition based on the state space model.

[0101] Step 8-3: Remove the cross-modal feature mapping module and the cross-modal interaction attention module respectively, and conduct ablation experiments to verify the effectiveness of the cross-modal interaction fusion strategy in this invention;

[0102] Step 8-4: Remove the triplet loss function and conduct ablation experiments to verify its importance in model training and its impact on performance.

[0103] In this embodiment, two widely adopted sentiment classification metrics are used to evaluate and compare the performance of different methods: weighted accuracy (WA), which is the overall classification accuracy across all categories, calculated by weighting the accuracy of each category by the number of samples; and unweighted accuracy (UA), which is the average recall for each sentiment category, with each category given the same weight regardless of sample size. The IEMOCAP database contains various emotions, including anger, excited, happy, sad, neutral, and frustrated. Samples of four types—angry, happy, sad, and neutral—are selected for the four-class multimodal sentiment recognition task. Five-fold cross-validation is used to test the performance of the IEMOCAP dataset. During the experiment, Python version 3.7 was used, along with the PyTorch deep learning framework. The torch version was 1.13.0+cu116, and the torchaudio version was 0.13.0+cu116. The hardware consisted of an NVIDIA RTX 3090 graphics card, an Intel(R) Xeon(R) Silver 4214 CPU @ 2.20GHz, and the Adam optimizer was used for the network model. The initial learning rate was 0.0001, the batch size was 16, and the number of iterations was 125.

[0104] A comparative experiment was conducted between a multimodal emotion recognition method based on the State-Space Model (S4D) and a multimodal emotion recognition method based on the traditional Sequence Model (LSTM network). Ablation experiments were performed after removing the cross-modal feature mapping module, the cross-modal interaction attention module, and the triplet loss function. The results are shown in Table 1.

[0105] Table 1 shows the ablation experiments of the proposed method on the IEMOCAP dataset.

[0106] method Weighted accuracy (WA) Unweighted accuracy (UA) Replace S4D with LSTM 0.763 0.772 Remove only the cross-modal feature mapping module 0.767 0.773 Remove only the cross-modal interaction attention module 0.770 0.775 Loss function with only triples removed 0.768 0.775 The method proposed in this invention 0.777 0.785

[0107] Table 1 shows that replacing the S4D state-space model module with an LSTM network reduces the overall weighted accuracy by 1.4% and the unweighted accuracy by 1.3%, indicating that the sentiment recognition model using the S4D module outperforms the one using the LSTM network. Without the cross-modal feature mapping module, the overall model performance decreases, with weighted accuracy decreasing by 1% and unweighted accuracy by 1.2%. Removing the cross-modal interaction attention module reduces the overall weighted accuracy by 0.7% and the unweighted accuracy by 1%. This demonstrates that using a state-space model for feature extraction and employing a cross-modal interaction fusion strategy improves various model parameters. To verify the importance of the triplet loss in model training and its impact on performance, this loss function was removed, and an ablation experiment was conducted. It can be seen that without the triplet loss, both the weighted and unweighted accuracies decreased by approximately 1%.

[0108] In summary, this embodiment proposes a multimodal emotion recognition method based on state-space model cross-modal interaction. Utilizing the S4D model, it extracts and connects features from acoustic and textual modalities to form powerful S4D multimodal features. This enables the model to dynamically capture the temporal relationships and changes in sound and text signals, thereby enhancing its adaptability and generalization ability. To further improve model performance, a cross-modal feature mapping module and a cross-modal interaction attention module are introduced, aiming to comprehensively capture features from both modalities and fully utilize cross-modal information. Experimental results verify the rationality and effectiveness of these two modules. Furthermore, the combination of triplet loss and cross-entropy loss during training can better distinguish different emotion categories. Compared with other methods, the method in this embodiment achieves improvements in both measurement metrics and has significant reference value in practical applications.

[0109] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multimodal emotion recognition method based on cross-modal interaction using a state-space model, characterized in that, Includes the following steps: Step 1: Perform low-level feature extraction on the original input speech and text modalities respectively; Step 2: Input the extracted low-level features into the channel grouping feature extraction network to obtain high-level features. The network consists of different numbers of one-dimensional convolutional layers and channel grouping convolutional layers. Step 3: Input the extracted high-level features of speech and text into the cross-modal feature mapping module respectively, and learn the interaction between acoustic and text features in detail to extract rich cross-modal information from speech segments and sentences; Step 4: The low-level features of speech and text are processed by two layers of diagonalized structured state space sequence S4D modules to extract higher-level and more robust S4D feature representations, and then concatenated to obtain S4D multimodal features. Step 5: Input the S4D multimodal features and high-level features of speech and text into the cross-modal interactive attention module to adaptively enhance the modal sequence representation; Step 5-1: Use the high-level features of the sound and text modalities as query vectors Q, and the S4D multimodal features as key vectors K and value vectors V. Perform cross-modal attention mechanism operations to dynamically adjust the weights of sound and text information. The cross-modal attention mechanism module includes a multi-head attention mechanism, fully connected layers, skip connections, and layer normalization. The number of heads in the multi-head attention mechanism is 8. Step 5-2: Convert the output CA(Q') of the cross-modal attention mechanism a ) and CA(Q' t The inputs are fed into a single-output gated fusion layer. A sigmoid function is applied to process the inputs, generating a gate vector. This gate vector is used to adjust the weights of each modality feature in the subsequent feature fusion process. A linear transformation weight matrix is ​​used to perform a linear transformation on the inputs. Then, the result is multiplied element-wise with the gate vector to obtain the fused feature F. a and F t ; Step 6: Combine the intrinsic features of the two modalities extracted in Step 3, the S4D multimodal features, and the interaction features between the modalities F. n The features are then fused together and fed into a bidirectional LSTM layer. A self-attention mechanism is used to select key emotional features. Finally, the emotional classification result is output through the fully connected layer of the model. Step 6-1: Extract the intrinsic features X” of the speech-text modality from Step 3. n '、S4D Multimodal Features X c Interaction features F between modes n The features are fused, where n∈{a,t}, a is the speech modality and t is the text modality. The fused feature vector is then input into a bidirectional LSTM network to capture long-term dependencies and contextual information. Step 6-2: A self-attention mechanism is added after the bidirectional LSTM network to filter key features. Then, the order information of all hidden states is aggregated using a max pooling layer. Step 6-3: The aggregated information is used to perform the final sentiment prediction through a two-layer fully connected feedforward network; Step 7: Use the designed triplet loss function combined with cross-entropy loss to constrain model convergence and improve the accuracy of the model in sentiment classification; Step 7-1: Define each data instance as a triple, where the anchor sample y represents the predicted sentiment category, where positive sample y is the correct sentiment label and negative sample e is a randomly selected incorrect sentiment label. Step 7-2: Use a neural network to convert each sample into a corresponding embedding vector. The cosine similarity f(y) and f(e) is calculated to minimize the distance between positive samples and anchor points while maximizing the distance between negative samples and anchor points. The triplet loss function is expressed as: in Representing vectors The dot product with f(e), Representing vectors The norm of , where m is the boundary value hyperparameter; Step 7-3: Combine triplet loss with cross-entropy loss to optimize the accuracy of sentiment prediction during training; Step 8: Evaluate the performance of the proposed multimodal emotion recognition method based on cross-modal interaction using a state-space model.

2. The multimodal emotion recognition method based on cross-modal interaction using a state-space model according to claim 1, characterized in that, The specific steps of step 1 include: Step 1-1: For text embedding X t We use a 300-dimensional pre-trained GloVe embedding model to map words to fixed-length vectors for each utterance transcription. The dimension of the linear operator used to fine-tune the pre-trained word embeddings is set to 256. Steps 1-2: For acoustic embedding X a First, the audio signal of the data sample is converted into 25ms frames with a frame shift of 10ms. Then, a Python library is used to extract 40-dimensional filter bank features and 13-dimensional MFCC features from each frame.

3. The multimodal emotion recognition method based on cross-modal interaction using a state-space model according to claim 1, characterized in that, The specific steps of step 2 include: Step 2-1: Input low-level text features X t The channel-group convolution module consists of one one-dimensional convolutional layer and one channel-group convolutional layer, with kernel sizes of 3 and 2 respectively, and a stride of 1 for each kernel. The number of groups in the channel-group convolutional layer is set to 4. The one-dimensional convolutional layer consists of three parts: one-dimensional convolution, GeLU activation function, and max pooling. The channel-group convolutional layer consists of four parts: group convolution, GeLU activation function, max pooling, and layer normalization. Step 2-2: Input low-level speech features X a The channel grouping convolution module consists of two one-dimensional convolutional layers and one channel grouping convolutional layer, with kernel sizes of 3, 2, and 3 respectively, and a stride of 1 for each kernel. The number of groups in the channel grouping convolutional layer is set to 4. The composition of the one-dimensional convolutional layer and the channel grouping convolutional layer is the same as in step 2-1.

Images