Multi-modal emotion recognition method based on state space model cross-modal interaction

Abstract

The invention discloses a multi-modal emotion recognition method based on cross-modal interaction of a state space model, and belongs to the technical field of multi-modal emotion recognition, and the method comprises the steps: firstly, extracting low-level features of voice and text, and extracting S4D features through employing a structured state space sequence model of a diagonal version, the unique sequence modeling capability and the robust feature extraction capability are fully utilized, and the complex time correlation of sound and text data is effectively captured; secondly, in order to fuse the modal features of the sound and the text, a plurality of cross-modal fusion strategies are provided to dynamically learn the weight between the modals; besides, in order to better distinguish different types of sentiments, a triple loss is designed and is combined with a cross entropy loss, so that the accuracy of the model in sentiment classification is improved; finally, an ablation experiment is carried out, and the effectiveness of the method is verified. According to the invention, the performance and generalization ability of the multi-modal emotion recognition system are effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal emotion recognition, and specifically relates to a multimodal emotion recognition method based on cross-modal interaction of a state-space model. Background Art

[0002] Emotion plays an important role in information transmission and interpersonal communication. In recent years, multimodal emotion recognition has attracted more and more attention and has been widely used in human-computer interaction, intelligent customer service, medical and health care and other fields. By accurately detecting emotions from multimodal data, the user experience can be greatly enhanced and the system performance can be significantly improved. Multimodal emotion recognition combines multiple modal information such as speech, image, and text to identify and analyze emotional states. Compared with the incomplete information or noise interference that may exist in a single modality, multimodal fusion can provide more comprehensive and accurate emotional information.

[0003] Feature extraction is a key step in multimodal emotion recognition tasks, and its main purpose is to obtain features that effectively represent different emotions. In recent years, in order to complete emotion recognition tasks, researchers have mainly relied on traditional deep sequence models, such as long short-term memory networks (LSTMs), convolutional neural networks (CNNs), transformers, etc., to extract features from different modalities. However, these models may encounter the problem of gradient vanishing or exploding when processing long sequences, which makes it difficult for them to capture long-term dependencies. However, in sentiment analysis, it is often crucial to consider the long-term evolution of the speaker's emotional state. Secondly, these models need to process each element of the sequence step by step, which leads to high computational cost and slow training and inference speed when processing long sequences. The introduction of state-space models (SSMs) has been shown to outperform traditional deep sequence models. Gu et al. proposed the structured state-space sequence model (S4) and its variants (the diagonal version of S4, S4D), which can process the entire sequence in parallel, more effectively capture long-term dependencies in the sequence, and show higher computational efficiency. The S4 method can handle variable-length input sequences and adapt to the different lengths of different text or speech paragraphs. Therefore, compared with traditional deep sequence models, state-space models have advantages in sentiment analysis and can more accurately capture the complexity and long-term correlation of emotional expressions.

[0004] The core challenge of multimodal emotion recognition is how to efficiently integrate information from different modalities for emotion recognition. Since data from different modalities may have different feature representations and expressions, effective integration is crucial. Intra-modal interaction aims to reveal and enhance the correlation and temporal characteristics within each modality, thereby improving the representation ability of a single modality. Inter-modal interaction uses the complementary information between different modalities to improve the performance of emotion recognition through an effective information transfer mechanism. In the prior art, Yoon et al. constructed a multimodal dual-loop encoder model to learn speech and text representations and combine them for emotion classification. However, this method does not explicitly capture multimodal interactions. Tsai et al. used a multi-head attention mechanism to calculate the correlation between modalities at different positions and subspaces, and directly extracted multimodal representations from unaligned multimodal sequence data, but ignored the correlation of specific features in each modality.

[0005] In daily life, speech and text are important ways to express emotions, providing key discriminant information for multimodal emotion recognition systems. However, due to the heterogeneity of speech and text information in practical application scenarios, multimodal emotion recognition technology faces the key challenge of effectively extracting key and complementary information from speech and text data. In addition, the effective fusion of extracted speech and text information is crucial to the performance of multimodal emotion recognition systems, especially in ensuring the accuracy and robustness of the system. Summary of the invention

[0006] The present invention aims to overcome the defects and shortcomings of the prior art, and proposes a multimodal emotion recognition method based on cross-modal interaction of state-space models. By adopting the diagonal version of the structured state-space sequence (S4D) model to extract S4D features, it fully utilizes its independent sequence modeling and robust feature extraction capabilities, effectively captures the complex temporal correlations of acoustic and text data, and dynamically learns the weights between modalities through various cross-modal fusion strategies. In addition, in order to better distinguish different categories of emotions, the method proposed in the present invention designs a triple loss model and combines it with the cross-entropy loss to improve the accuracy of the model in emotion classification. This innovative method provides a new solution for multimodal emotion recognition and has broad application prospects.

[0007] The technical solution adopted by the present invention to solve the technical problem is: a multimodal emotion recognition method based on cross-modal interaction of a state space model, the method comprising the following steps:

[0008] Step 1: Perform low-level feature extraction on the original input speech and text modalities respectively;

[0009] Step 2: Input the extracted low-level features into a channel grouping feature extraction network to obtain high-level features, wherein the network is composed of different numbers of one-dimensional convolutional layers and channel grouping convolutional layers;

[0010] Step 3: The extracted high-level features of speech and text are respectively input into the cross-modal feature mapping module to finely learn the interaction between acoustic and text features and extract rich cross-modal information from speech fragments and sentences;

[0011] Step 4: The low-level features of speech and text are processed separately through two layers of diagonal structured state space sequence (S4D) modules to extract higher-level and more robust S4D feature representations, and then concatenate them to obtain S4D multimodal features;

[0012] Step 5: Input the S4D multimodal features and high-level features of speech and text into the cross-modal interactive attention module to adaptively enhance the modal sequence representation;

[0013] Step 6: Combine the intrinsic features of the two modalities, the S4D multimodal features, and the interaction features between the modalities. Input the fused feature vector into the bidirectional LSTM layer, and select the key emotional features in combination with the self-attention mechanism. Finally, output the emotional classification results through the fully connected layer of the model;

[0014] Step 7: Use the designed triplet loss function combined with the cross entropy loss to constrain the model convergence and improve the accuracy of the model in sentiment classification;

[0015] Step 8: Perform a performance evaluation on the proposed multimodal emotion recognition method based on cross-modal interaction of state-space model.

[0016] Furthermore, the specific steps of step 1 include:

[0017] Step 1-1: For text embedding X t , a 300-dimensional pre-trained GloVe embedding model is used to map words into fixed-length vectors for each utterance transcription, and the dimension of the linear operator used to fine-tune the pre-trained word embeddings is set to 256;

[0018] Step 1-2: For acoustic embedding X a ,First, the audio signal of the data sample is converted into 25ms frames with a frame shift of 10ms, and the 40-dimensional filter bank features and 13-dimensional MFCC features are extracted from each frame using the Python ,library.

[0019] Furthermore, the specific steps of step 2 include:

[0020] Step 2-1: Input low-level text features X tTo the channel grouping convolution module, the module consists of 1 one-dimensional convolution layer and 1 channel grouping convolution layer, the convolution kernel sizes are 3 and 2 respectively, the stride of each kernel is 1, and the number of groups of the channel grouping convolution layer is set to 4. The one-dimensional convolution layer consists of three parts: one-dimensional convolution, GeLU activation function and maximum pooling. The channel grouping convolution layer consists of four parts: grouped convolution, GeLU activation function, maximum pooling and layer normalization;

[0021] Step 2-2: Input low-level speech features X a To the channel grouping convolution module, the module consists of 2 one-dimensional convolution layers and 1 channel grouping convolution layer, the convolution kernel sizes are 3, 2 and 3 respectively, the step size of each kernel is 1, and the number of groups of the channel grouping convolution layer is set to 4. The components of the one-dimensional convolution layer and the channel grouping convolution layer are consistent with the step 2-1.

[0022] Furthermore, the specific steps of step 3 include:

[0023] Step 3-1: Use a dimension-dependent gating mechanism with a sigmoid activation function to learn X output by step 2 a and X' t 'Weight parameter of the embedding dimension,

[0024] M a =σ(W a X' t '),M t =σ(W t X' a '),

[0025] where σ(·) represents the sigmoid function, and are the linear projection operators for speech representation and text representation, respectively, used to map the input features to the corresponding representation space, M a and M t is the mapping matrix, which serves as the weight to adapt the corresponding features;

[0026] Step 3-2: Adaptively calibrate speech and text representations through a cross-modal mapping matrix,

[0027] X″′ a =M a +X″ a ,X″′ t =M t +X″ t .

[0028] Furthermore, the specific steps of step 4 include:

[0029] Step 4-1: Replace the traditional deep sequence network such as LSTM with a two-layer S4D module for multimodal feature extraction. The S4D module includes residual connection, dropout, one-dimensional convolution layer, gated linear unit (GLU) and layer normalization;

[0030] Step 4-2: Connect the feature sequences of speech and text modes processed by their respective S4D modules to obtain the S4D multimodal feature X c .

[0031] Furthermore, the specific steps of step 5 include:

[0032] Step 5-1: Take the features of the sound and text modalities as the query vector Q (query), and the S4D multimodal features as the key K (key) and value V (value) vectors, and perform cross-modal attention mechanism operations respectively to dynamically adjust the weights of the sound and text information. The cross-modal attention mechanism module includes a multi-head attention mechanism, a fully connected layer, a skip connection, and layer normalization. The number of heads of the multi-head attention mechanism is 8;

[0033] Step 5-2: The output of the cross-modal attention mechanism CA(Q' a ) and CA(Q' t ) are respectively input to the single output gated fusion layer, and the input is processed by applying the sigmoid function to generate a gated vector. The gated vector is used to adjust the weights of each modal feature in the subsequent feature fusion process, and the input is linearly transformed using a linear transformation weight matrix, and then the result is multiplied element-by-element by the gated vector to obtain the fused feature F a and F t .

[0034] Furthermore, the specific steps of step 6 include:

[0035] Step 6-1: Transform the intrinsic features X of the speech-to-text modality n ', S4D multimodal feature X c and the interaction feature F between the modalities n Perform fusion, where n∈{a,t}, and then input the fused feature vector into a bidirectional LSTM network to capture long-term dependencies and contextual information;

[0036] Step 6-2: A self-attention mechanism is added after the bidirectional LSTM network to filter key features, and then a maximum pooling layer is used to aggregate the sequential information of all hidden states;

[0037] Step 6-3: The aggregated information is passed through a two-layer fully connected feed-forward network for final sentiment prediction.

[0038] Furthermore, the specific steps of step 7 include:

[0039] Step 7-1: Define each data instance as a triplet, where the anchor sample is the predicted emotion category, the positive sample y is the correct emotion label, and the negative sample e is a randomly selected wrong emotion label;

[0040] Step 7-2: Use a neural network to convert each sample into a corresponding embedding vector f(y) and f(e), calculate the cosine similarity, the purpose is to minimize the distance between the positive sample and the anchor point, while maximizing the distance between the negative sample and the anchor point. The triplet loss function is expressed as:

[0041]

[0042] in Representation vector The dot product with f(e), Representation vector The norm of , m is the boundary value hyperparameter;

[0043] Step 7-3: Combine triplet loss with cross entropy loss to optimize the accuracy of sentiment prediction during training.

[0044] Furthermore, the specific method for performing performance evaluation on the proposed multimodal emotion recognition method based on cross-modal interaction of state-space model in step 8 includes the following steps:

[0045] Step 8-1: Compare a multimodal emotion recognition method based on cross-modal interaction of a state-space model with existing multimodal emotion recognition methods to verify their overall performance;

[0046] Step 8-2: Conduct a comparative experiment between the multimodal emotion recognition method based on the state-space model and the multimodal emotion recognition method based on the traditional sequence model (such as LSTM network) to verify the effectiveness of speech emotion recognition based on the state-space model;

[0047] Step 8-3: Remove the cross-modal feature mapping module and the cross-modal interactive attention module respectively, and perform an ablation experiment to verify the effectiveness of the cross-modal interactive fusion strategy in the present invention;

[0048] Step 8-4: Delete the triplet loss function and conduct ablation experiments to verify its importance in model training and its impact on performance.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] 1. The present invention proposes a multimodal emotion recognition based on the state-space model, which uses the S4D model to extract and connect the features of the sound and text modalities to form a powerful S4D multimodal feature, which enables the model to dynamically capture the temporal relationship and changes of the sound and text signals, thereby enhancing the adaptability and generalization ability of the model.

[0051] 2. The present invention introduces a cross-modal feature mapping module and a cross-modal interactive attention module, which enables the model to comprehensively capture the intrinsic correlation between acoustic features, text features and S4D multimodal features, and make full use of the complementary information between these features.

[0052] 3. The present invention designs a triple loss function and combines it with the cross entropy loss during training to ensure the closeness between emotion categories and better distinguish different categories of emotions. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a schematic diagram of the operation flow of the present invention.

[0054] Figure 2 This is a block diagram of the channel grouping feature extraction module of the present invention.

[0055] Figure 3 This is a block diagram of the state-space model S4D module of the present invention.

[0056] Figure 4 This is a block diagram of the cross-modal feature mapping module of the present invention.

[0057] Figure 5 This is a block diagram of the cross-modal interactive attention module of the present invention. DETAILED DESCRIPTION

[0058] The invention will be further described in detail below in conjunction with the accompanying drawings.

[0059] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is described in detail below in conjunction with the accompanying drawings, attached tables and specific embodiments. It should be noted that, in order to avoid blurring the present invention due to unnecessary details, only structures and / or processing steps closely related to the scheme of the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0060] In addition, it should be noted that the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or apparatus.

[0061] like Figures 1 to 5 As shown, the present invention proposes a multimodal emotion recognition method based on cross-modal interaction of a state-space model, which applies the state-space model to multimodal emotion recognition to extract high-level features and build a higher-performance multimodal emotion recognition system, specifically comprising the following steps:

[0062] Step 1: Perform low-level feature extraction on the original input speech and text modalities respectively. The dataset used in the present invention is the IEMOCAP emotion database, which contains 5 different sessions, each of which is conducted by two actors (one male and one female), with approximately 12 hours of speech data.

[0063] Step 1 specifically includes:

[0064] Step 1-1: For text embedding X t , a 300-dimensional pre-trained GloVe embedding model is used to map words into fixed-length vectors for each utterance transcription, and the dimension of the linear operator used to fine-tune the pre-trained word embeddings is set to 256;

[0065] Step 1-2: For acoustic embedding X a ,First, the audio signal of the data sample is converted into 25ms frames with a frame shift of 10ms, and the 40-dimensional filter bank features and 13-dimensional MFCC features are extracted from each frame using the Python ,library.

[0066] Step 2: Input the extracted low-level features into the channel grouping feature extraction network to obtain high-level features. The network consists of different numbers of one-dimensional convolutional layers and channel grouping convolutional layers.

[0067] Step 2 specifically includes:

[0068] Step 2-1: Input low-level text features X t To the channel grouping convolution module, this module consists of 1 one-dimensional convolution layer and 1 channel grouping convolution layer, the convolution kernel sizes are 3 and 2 respectively, the stride of each kernel is 1, and the number of groups of the channel grouping convolution layer is set to 4. The one-dimensional convolution layer consists of three parts: one-dimensional convolution, GeLU activation function, and maximum pooling. The channel grouping convolution layer consists of four parts: grouped convolution, GeLU activation function, maximum pooling, and layer normalization;

[0069] Step 2-2: Input low-level speech features X a To the channel grouping convolution module, the module consists of 2 one-dimensional convolution layers and 1 channel grouping convolution layer, the convolution kernel sizes are 3, 2, and 3 respectively, the stride of each kernel is 1, and the number of groups of the channel grouping convolution layer is set to 4. The components of the one-dimensional convolution layer and the channel grouping convolution layer are consistent with those described in step 2-1.

[0070] Step 3: The extracted high-level features of speech and text are respectively input into the cross-modal feature mapping module to finely learn the interaction between acoustic and text features and extract rich cross-modal information from speech fragments and sentences.

[0071] Step 3 specifically includes:

[0072] Step 3-1: Use a dimension-dependent gating mechanism with a sigmoid activation function to learn X output by step 2" a and X' t 'Weight parameter of the embedding dimension,

[0073] M a =σ(W a X' t '),M t =σ(W t X' a '),

[0074] where σ(·) represents the sigmoid function. and are the linear projection operators for speech representation and text representation, respectively, used to map the input features to the corresponding representation space, M a and M t is the mapping matrix, which serves as the weight to adapt the corresponding features;

[0075] Step 3-2: Finally, the speech and text representations are adaptively calibrated through the cross-modal mapping matrix,

[0076] X″′ a =M a +X″ a ,X″′ t =M t +X″ t .

[0077] Step 4: The low-level features of speech and text are processed separately through two layers of diagonal structured state-space sequence (S4D) modules to extract higher-level and more robust S4D feature representations, which are then concatenated to obtain S4D multimodal features.

[0078] Step 4 specifically includes:

[0079] Step 4-1: Replace the traditional deep sequence network such as LSTM with a two-layer S4D module for multimodal feature extraction. The S4D module includes residual connection, dropout, one-dimensional convolution layer, gated linear unit (GLU) and layer normalization;

[0080] Step 4-2: Connect the feature sequences of speech and text modes processed by their respective S4D modules to obtain the S4D multimodal feature X c .

[0081] Step 5: The S4D multimodal features and high-level features of speech and text are input into the cross-modal interactive attention module to adaptively enhance the modal sequence representation.

[0082] Step 5 specifically includes:

[0083] Step 5-1: Take the features of the sound and text modalities as the query vector Q (query), and the S4D multimodal features as the key K (key) and value V (value) vectors, and perform cross-modal attention mechanism operations respectively to dynamically adjust the weights of the sound and text information. The cross-modal attention mechanism module includes a multi-head attention mechanism, a fully connected layer, a skip connection, and layer normalization. The number of heads of the multi-head attention mechanism is 8;

[0084] Step 5-2: The output of the cross-modal attention mechanism CA(Q' a ) and CA(Q' t ) are input to the single output gated fusion layer respectively, and the input is processed by applying the sigmoid function to generate a gated vector. The gated vector is used to adjust the weights of each modal feature in the subsequent feature fusion process, and the input is linearly transformed using the linear transformation weight matrix, and then the result is multiplied element by element with the gated vector to obtain the fused feature F a and F t .

[0085] Step 6: Fuse the intrinsic features of the two modalities, the S4D multimodal features, and the interaction features between the modalities. Input the fused feature vector into the bidirectional LSTM layer, and select the key emotional features in combination with the self-attention mechanism. Finally, output the emotional classification results through the fully connected layer of the model.

[0086] Step 6 specifically includes:

[0087] Step 6-1: Transform the intrinsic features X of the speech-to-text modality n ', S4D multimodal feature X c and the interaction feature F between the modalities n Perform fusion, where n∈{a,t}, and then input the fused feature vector into a bidirectional LSTM network to capture long-term dependencies and contextual information;

[0088] Step 6-2: A self-attention mechanism is added after the bidirectional LSTM network to filter key features, and then a maximum pooling layer is used to aggregate the sequential information of all hidden states;

[0089] Step 6-3: The aggregated information is passed through a two-layer fully connected feed-forward network for final sentiment prediction.

[0090] Step 7: Use the designed triplet loss function in combination with the cross entropy loss to constrain the model convergence and improve the accuracy of the model in sentiment classification.

[0091] Step 7 specifically includes:

[0092] Step 7-1: Define each data instance as a triplet, where the anchor sample is the predicted emotion category, the positive sample y is the correct emotion label, and the negative sample e is a randomly selected wrong emotion label;

[0093] Step 7-2: Use a neural network to convert each sample into a corresponding embedding vector f(y) and f(e), calculate the cosine similarity, the purpose is to minimize the distance between the positive sample and the anchor point, while maximizing the distance between the negative sample and the anchor point. The triplet loss function is expressed as:

[0094]

[0095] in Representation vector The dot product with f(e), Representation vector The norm of , m is the boundary value hyperparameter;

[0096] Step 7-3: Combine triplet loss with cross entropy loss to optimize the accuracy of sentiment prediction during training.

[0097] Step 8: Perform a performance evaluation on the proposed multimodal emotion recognition method based on cross-modal interaction of state-space model.

[0098] Step 8 specifically includes:

[0099] Step 8-1: Compare a multimodal emotion recognition method based on cross-modal interaction of a state-space model with existing multimodal emotion recognition methods to verify their overall performance;

[0100] Step 8-2: Conduct a comparative experiment between the multimodal emotion recognition method based on the state-space model (S4D) and the multimodal emotion recognition method based on the traditional sequence model (such as LSTM network) to verify the effectiveness of speech emotion recognition based on the state-space model;

[0101] Step 8-3: Remove the cross-modal feature mapping module and the cross-modal interactive attention module respectively, and perform an ablation experiment to verify the effectiveness of the cross-modal interactive fusion strategy in the present invention;

[0102] Step 8-4: Delete the triplet loss function and conduct ablation experiments to verify its importance in model training and its impact on performance.

[0103] In this embodiment, two widely used sentiment classification indicators are used to evaluate and compare the performance of different methods: weighted accuracy (WA), which is the overall classification accuracy of all categories, calculated by weighting the accuracy of each category by the number of samples; unweighted accuracy (UA), which is the average recall of each sentiment category, and each category is given the same weight regardless of the sample size. The IEMOCAP database contains a variety of emotions, including anger, excitement, happiness, sadness, neutrality, fear, etc. Four types of samples, anger, happiness, sadness and neutrality, are selected for the four-category multimodal sentiment recognition task. At the same time, the IEMOCAP dataset is tested for performance using five-fold cross validation. During the experiment, the Python version was 3.7, and the Pytorch deep learning framework was used. The torch version was 1.13.0+cu116, the torchaudio version was 0.13.0+cu116, the hardware was NVIDIARTX 3090 graphics card, the CPU processor was Intel(R)Xeon(R)Silver 4214CPU@2.20GHz, and the optimizer of the network model was Adam. The initial learning rate was 0.0001, the batch size was 16, and the number of iterations was 125.

[0104] The multimodal emotion recognition method based on the state-space model (S4D) was compared with the multimodal emotion recognition method based on the traditional sequence model (LSTM network), and the cross-modal feature mapping module, cross-modal interactive attention module and triple loss function were removed respectively, and ablation experiments were performed. The results are shown in Table 1:

[0105] Table 1 Ablation experiment of the proposed method on the IEMOCAP dataset

[0106] method Weighted Accuracy (WA) Unweighted Accuracy (UA) Replacing S4D with LSTM 0.763 0.772 Only remove the cross-modal feature mapping module 0.767 0.773 Only remove the cross-modal interaction attention module 0.770 0.775 Only remove triplet loss function 0.768 0.775 The method proposed by the present invention 0.777 0.785

[0107] From Table 1, we can see that when the state-space model S4D module is replaced with the LSTM network, the overall weighted accuracy of the model decreases by 1.4% and the unweighted accuracy decreases by 1.3%, which shows that the emotion recognition model using the S4D module is better than the emotion recognition model using the LSTM network. When there is no cross-modal feature mapping module, the overall performance of the model decreases, the weighted accuracy decreases by 1%, and the unweighted accuracy decreases by 1.2%. When the cross-modal interactive attention module is removed, the overall weighted accuracy of the model decreases by 0.7% and the unweighted accuracy decreases by 1%. It can be seen that when the state-space model is used to extract features and the cross-modal interactive fusion strategy is used, the various parameter indicators of the model have been improved to a certain extent. In order to verify the importance of triplet loss in model training and its impact on performance, the loss function is deleted and an ablation experiment is performed. It can be seen that without the triplet loss, the weighted accuracy and unweighted accuracy of the model decrease by about 1% respectively.

[0108] In summary, the present embodiment proposes a multimodal emotion recognition method based on cross-modal interaction of state-space model, which uses S4D model to extract and connect features from acoustic and text modalities to form a powerful S4D multimodal feature. This enables the model to dynamically capture the temporal relationship and changes of sound and text signals, thereby enhancing the adaptability and generalization ability of the model. In order to further improve the performance of the model, a cross-modal feature mapping module and a cross-modal interaction attention module are introduced to comprehensively capture the features of the two modalities and make full use of cross-modal information. The experimental results verify the rationality and effectiveness of these two modules. In addition, the combination of triplet loss and cross entropy loss is used in the training process, which can better distinguish different emotion categories. Compared with other methods, the method of this embodiment has been improved in both measurement indicators, which has a good reference significance in practical applications.

[0109] The above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A multimodal emotion recognition method based on cross-modal interaction of state space model, characterized in that: The steps include: Step 1: Perform low-level feature extraction on the original input speech and text modalities respectively; Step 2: Input the extracted low-level features into a channel grouping feature extraction network to obtain high-level features, wherein the network is composed of different numbers of one-dimensional convolutional layers and channel grouping convolutional layers; Step 3: The extracted high-level features of speech and text are respectively input into the cross-modal feature mapping module to finely learn the interaction between acoustic and text features and extract rich cross-modal information from speech fragments and sentences; Step 4: The low-level features of speech and text are processed separately through two layers of diagonal structured state space sequence (S4D) modules to extract higher-level and more robust S4D feature representations, and then concatenate them to obtain S4D multimodal features; Step 5: Input the S4D multimodal features and high-level features of speech and text into the cross-modal interactive attention module to adaptively enhance the modal sequence representation; Step 6: The intrinsic features of the two modalities, the S4D multimodal features, and the interaction features between the modalities are fused together, and the fused feature vector is input into the bidirectional LSTM layer. The key emotional features are selected in combination with the self-attention mechanism. Finally, the emotional classification results are output through the fully connected layer of the model. Step 7: Use the designed triplet loss function combined with the cross entropy loss to constrain the model convergence and improve the accuracy of the model in sentiment classification; Step 8: Perform a performance evaluation on the proposed multimodal emotion recognition method based on cross-modal interaction of state-space model.

2. According to claim 1, a multimodal emotion recognition method based on cross-modal interaction of state space model is characterized in that: The specific steps of step 1 include: Step 1-1: For text embedding X t , a 300-dimensional pre-trained GloVe embedding model is used to map words into fixed-length vectors for each utterance transcription, and the dimension of the linear operator used to fine-tune the pre-trained word embeddings is set to 256; Step 1-2: For acoustic embedding X a ,First, the audio signal of the data sample is converted into 25ms frames with a frame shift of 10ms, and the 40-dimensional filter bank features and 13-dimensional MFCC features are extracted from each frame using the Python ,library.

3. According to claim 1, a multimodal emotion recognition method based on cross-modal interaction of state space model is characterized in that: The specific steps of step 2 include: Step 2-1: Input low-level text features X t To the channel grouping convolution module, the module consists of 1 one-dimensional convolution layer and 1 channel grouping convolution layer, the convolution kernel sizes are 3 and 2 respectively, the step size of each kernel is 1, and the number of groups of the channel grouping convolution layer is set to 4; the one-dimensional convolution layer consists of three parts: one-dimensional convolution, GeLU activation function and maximum pooling; the channel grouping convolution layer consists of four parts: grouped convolution, GeLU activation function, maximum pooling and layer normalization; Step 2-2: Input low-level speech features X a To the channel grouping convolution module, the module consists of 2 one-dimensional convolution layers and 1 channel grouping convolution layer, the convolution kernel sizes are 3, 2 and 3 respectively, the step size of each kernel is 1, and the number of groups of the channel grouping convolution layer is set to 4; the components of the one-dimensional convolution layer and the channel grouping convolution layer are consistent with the step 2-1.

4. According to claim 1, a multimodal emotion recognition method based on cross-modal interaction of state space model is characterized in that: The specific steps of step 3 include: Step 3-1: Use a dimension-dependent gating mechanism with a sigmoid activation function to learn X output by step 2 a and X” t The weight parameter of the embedding dimension, M a =σ(W a "X" t ),M t =σ(W t "X" a ), where σ(·) represents the sigmoid function, and are the linear projection operators for speech representation and text representation, respectively, used to map the input features to the corresponding representation space, M a and M t is the mapping matrix, which serves as the weight to adapt the corresponding features; Step 3-2: Adaptively calibrate speech and text representations through a cross-modal mapping matrix, X” a =M a +X” a ,X” t =M t +X” t 。 5. According to claim 1, a multimodal emotion recognition method based on cross-modal interaction of state space model is characterized in that: The specific steps of step 4 include: Step 4-1: Replace the traditional deep sequence network such as LSTM with a two-layer S4D module for multimodal feature extraction. The S4D module includes residual connection, dropout, one-dimensional convolution layer, gated linear unit (GLU) and layer normalization; Step 4-2: Connect the feature sequences of speech and text modes processed by their respective S4D modules to obtain the S4D multimodal feature X c .

6. The multimodal emotion recognition method based on cross-modal interaction of state space model according to claim 1, characterized in that: The specific steps of step 5 include: Step 5-1: Take the features of the sound and text modalities as the query vector Q (query), and the S4D multimodal features as the key K (key) and value V (value) vectors, and perform cross-modal attention mechanism operations respectively to dynamically adjust the weights of the sound and text information. The cross-modal attention mechanism module includes a multi-head attention mechanism, a fully connected layer, a skip connection, and layer normalization. The number of heads of the multi-head attention mechanism is 8; Step 5-2: The output of the cross-modal attention mechanism CA(Q' a ) and CA(Q' t ) are respectively input into the single output gated fusion layer, and the input is processed by applying the sigmoid function to generate a gated vector, which is used to adjust the weights of each modal feature in the subsequent feature fusion process, and the input is linearly transformed using the linear transformation weight matrix, and then the result is multiplied one by one by the gated vector at the element level to obtain the fused feature F a and F t .

7. The multimodal emotion recognition method based on cross-modal interaction of state space model according to claim 1, characterized in that: The specific steps of step 6 include: Step 6-1: Transform the intrinsic features X' of the speech-to-text modality n , S4D multimodal feature X c and the interaction feature F between the modalities n Perform fusion, where n∈{a,t}, and then input the fused feature vector into a bidirectional LSTM network to capture long-term dependencies and contextual information; Step 6-2: A self-attention mechanism is added after the bidirectional LSTM network to filter key features, and then a maximum pooling layer is used to aggregate the sequential information of all hidden states; Step 6-3: The aggregated information is passed through a two-layer fully connected feed-forward network for final sentiment prediction.

8. The multimodal emotion recognition method based on cross-modal interaction of state space model according to claim 1, characterized in that: The specific steps of step 7 include: Step 7-1: Define each data instance as a triplet, where the anchor sample is the predicted emotion category, the positive sample y is the correct emotion label, and the negative sample e is a randomly selected wrong emotion label; Step 7-2: Use a neural network to convert each sample into a corresponding embedding vector f(y) and f(e), calculate the cosine similarity, the purpose is to minimize the distance between the positive sample and the anchor point, while maximizing the distance between the negative sample and the anchor point. The triplet loss function is expressed as: in Representation vector The dot product with f(e), Representation vector The norm of , m is the boundary value hyperparameter; Step 7-3: Combine triplet loss with cross entropy loss to optimize the accuracy of sentiment prediction during training.

9. The multimodal emotion recognition method based on cross-modal interaction of state space model according to claim 1, characterized in that: The specific method for performing performance evaluation on the proposed multimodal emotion recognition method based on cross-modal interaction of the state space model in step 8 includes the following steps: Step 8-1: Compare a multimodal emotion recognition method based on cross-modal interaction of a state-space model with existing multimodal emotion recognition methods to verify their overall performance; Step 8-2: Conduct a comparative experiment between the multimodal emotion recognition method based on the state space model and the multimodal emotion recognition method based on the traditional sequence model to verify the effectiveness of speech emotion recognition based on the state space model; Step 8-3: Remove the cross-modal feature mapping module and the cross-modal interactive attention module respectively, and perform an ablation experiment to verify the effectiveness of the cross-modal interactive fusion strategy in the present invention; Step 8-4: Delete the triplet loss function and conduct ablation experiments to verify its importance in model training and its impact on performance.

Images