Speech emotion recognition method combining attention mechanism and transpose mamba

Abstract

The application discloses a speech emotion recognition method combining an attention mechanism and a transpose Mamba, relates to the technical field of artificial intelligence, and inputs speech to be recognized into a trained joint model to output an emotion recognition result of the speech; training of the joint model is as follows: a plurality of emotions are obtained from speech, and are respectively converted into initial speech feature vectors through a general language pre-training model; the initial speech feature vectors are input into the joint model, global semantic features of emotional speech are obtained through multi-head attention coding processing; the global semantic features of the emotional speech are input into a transpose Mamba module, forward branch state selection output and transpose branch state selection output are obtained, the forward branch state selection output and the transpose branch state selection output are fused to obtain complex speech timing features; the complex speech timing features are input into a classifier module to obtain a final emotion prediction value; the speech emotion recognition method improves the speech emotion recognition capability.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a speech emotion recognition method using a joint attention mechanism and transposed Mamba. Background Art

[0002] Speech emotion recognition is key to human-computer interaction. To make human-computer interaction more natural, machines must accurately capture emotions and respond appropriately. This technology acts as a computer-assisted "eye," helping machines analyze the acoustic characteristics and content of speech signals. For example, this technology can be used in telephone customer service and survey research to help companies understand customer emotions and satisfaction. By analyzing customer speech, companies can promptly identify and resolve customer issues, improving customer service quality.

[0003] With the development of modern deep learning technology and audio processing technology, speech emotion analysis methods based on deep learning have made great progress, but they still face several problems:

[0004] First, while convolutional neural networks (CNNs) excel in certain areas, they also have drawbacks. CNNs are limited when processing long audio sequences, as they typically only capture local audio features and struggle to effectively process and understand longer audio sequences. In some cases, CNNs may not be as computationally efficient as other types of models, especially when processing large-scale speech datasets. Second, CNN models require more parameters and a more complex architecture to achieve comparable performance to other models, resulting in higher computational costs and storage requirements.

[0005] For example, in 2023, Surbhi Khurana et al. published "SER: Performance Evaluation of CNN Models and Overview of Available Indian Speech Datasets, and the Evolution of Classifiers from Traditional to Modern" in the journal ACM Asia, an international conference on low-resource speech information processing. This paper used CNNs for speech sentiment analysis and designed a series of related experiments, pointing out that the high computational cost of CNN models limits their practicality in resource-constrained environments. Furthermore, CNN models performed poorly on data across different corpora, demonstrating limited generalization capabilities.

[0006] Second, existing Transformer-based models have also achieved remarkable results in the field of speech emotion recognition. This success is attributed to the multi-head self-attention (MHA) module, which can represent complex data structures within a specific context window. However, the self-attention mechanism faces the challenge of computational complexity, which increases quadratically with the size of the context window. In speech tasks, it is usually necessary to consider the entire speech sample as the context window. This results in the presence of extensive contextual information for the frame-level acoustic feature sequence, which significantly increases the computational complexity.

[0007] For example, in 2023, Chen et al. published an article entitled "DWFormer: Dynamic Window Transformer for Speech Emotion Recognition" at the international conference ICASSP 2023. The article pointed out that the defect of traditional Transformer in speech emotion recognition tasks is that its indiscriminate processing of all input information may lead to redundant information interference and noise affecting accuracy. Summary of the Invention

[0008] Based on the technical problems existing in the background technology, the present invention proposes a speech emotion recognition method that combines the attention mechanism and transposed Mamba, which reduces the computational complexity of the model and improves the ability of speech emotion recognition.

[0009] The speech emotion recognition method proposed in the present invention combines the attention mechanism and the transposed Mamba, inputs the speech to be recognized into the trained joint model to output the emotion recognition result of the speech;

[0010] The training process of the joint model is as follows:

[0011] Step 1: Obtain multiple emotion-embedded speech sounds and convert them into initial speech feature vectors through a general language pre-training model to form a training dataset;

[0012] Step 2: In the i-th training round, the initial speech feature vector is input into the joint model, and the initial speech feature vector is layer-normalized and mapped through multi-head attention encoding to obtain query features, key features, and value features. The query features, key features, and value features are split according to the number of heads of multi-head attention to obtain the query vector, key vector, and value vector of each attention head. The query vector, key vector, and value vector are subjected to dot product attention calculation and spliced ​​to obtain the global semantic features of emotional speech;

[0013] Step 3: Input the global semantic features of emotional speech into the transposed Mamba module, perform a two-way parallel operation on the global semantic features of emotional speech, and add a gating mechanism to the two-way parallel operation to obtain the forward branch state selection output and the transposed branch state selection output. The forward branch state selection output and the transposed branch state selection output are fused to obtain the complex speech time series features;

[0014] Step 4: Input the complex speech time series features into the classifier module to obtain the final emotion prediction value. Based on the emotion prediction value and the corresponding true emotion label, a cross-entropy error loss is constructed to adjust the trainable parameters in the joint model.

[0015] Furthermore, in step 2, dot product attention calculation is performed on the query vector, key vector, and value vector, and the global semantic features of emotional speech are obtained after splicing. Specifically,

[0016] Calculate the attention weight between the query vector and the key vector in each attention head, and calculate the dot product between the attention weight and the value vector to obtain the weighted self-attention encoding feature of each attention head;

[0017] The weighted self-attention encoding features of each attention head are concatenated and output through a linear fully connected layer to obtain the multi-head attention feature. The initial speech feature vector is added to the multi-head attention feature to obtain the global semantic feature of the emotional speech.

[0018] Furthermore, in step 3, a two-way parallel operation is performed on the global semantic features of the emotional speech, and a gating mechanism is added to the two-way parallel operation to obtain the forward branch state selection output and the transposed branch state selection output, as follows:

[0019] Normalize the global semantic features of emotional speech to construct the forward branch input of the transposed Mamba module;

[0020] Performing a transposition flip operation on the global semantic features of the emotional speech to construct a transposition branch input of the transposition Mamba module, wherein the transposition flip operation is a transposition flip operation along the time series feature dimension;

[0021] Calculate the selectively forgotten information of the forward branch input and the transposed branch input respectively to obtain the current state of the forward branch and the current state of the transposed branch;

[0022] Gating mechanisms are added to the current state of the forward branch and the current state of the transposed branch respectively to obtain the forward branch state selection output and the transposed branch state selection output.

[0023] Furthermore, after calculating the selectively forgotten part of the forward branch input and the transposed branch input respectively, the current state of the forward branch and the current state of the transposed branch are obtained. Specifically, the forward branch input and the transposed branch input pass through the one-dimensional convolution layer, the activation layer and the selective state space model in sequence to obtain the current state of the forward branch and the current state of the transposed branch.

[0024] Furthermore, a gating mechanism is added to the current state of the forward branch and the current state of the transposed branch respectively to obtain the forward branch state selection output and the transposed branch state selection output, specifically:

[0025] F i =Gated(M i ,G i );

[0026] Among them, F i Select the output for the forward branch state, Select output for the transposed branch state, M i is the current state of the forward branch, is the current state of the transposed branch, G i represents the forward branch input, represents the transposed branch input, Gated(·) represents the gating mechanism in the transposed Mamba module, and:

[0027] Gated(M i ,G i )=M i ⊙Sigmoid(W l (G i ))

[0028]

[0029] Among them, ⊙ represents the Hadamard product operation of the matrix, Sigmoid represents the Sigmoid activation function, and W l represents a linear fully connected layer.

[0030] The advantages of the joint attention mechanism and transposed Mamba speech emotion recognition method provided by the present invention are: the joint attention mechanism and transposed Mamba speech emotion recognition method provided in the structure of the present invention are. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of the process of the present invention;

[0032] Figure 2 This is the training flow chart of the joint model;

[0033] Figure 3Schematic diagram of multi-head attention encoding;

[0034] Figure 4 Schematic diagram of the transposed Mamba module. DETAILED DESCRIPTION

[0035] The technical solutions of the present invention are described in detail below through specific embodiments. Numerous specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art may make similar modifications without departing from the scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0036] like Figures 1 to 4 As shown, the speech emotion recognition method of the joint attention mechanism and transposed Mamba proposed in the present invention inputs the speech to be recognized into the trained joint model to output the emotion recognition result of the speech;

[0037] The training process of the joint model is as follows:

[0038] Step 1: Obtain multiple emotion-embedded speech sounds and convert them into initial speech feature vectors through a general language pre-training model to form a training dataset;

[0039] The emotional speech is taken as input and converted into the initial speech feature vector using the general language pre-training model (WavLM speech pre-training model), which is recorded as in, represents the real number domain, L represents the speech sequence length, and C represents the feature dimension size. The WavLM speech pre-training model combines self-supervised learning with abundant training data to effectively extract rich features from speech while significantly improving the accuracy of downstream tasks.

[0040] Step 2: In the i-th training round, the initial speech feature vector is input into the joint model, and the initial speech feature vector is layer-normalized and mapped through multi-head attention encoding to obtain query features, key features, and value features. The query features, key features, and value features are split according to the number of heads of multi-head attention to obtain the query vector, key vector, and value vector of each attention head. The query vector, key vector, and value vector are subjected to dot product attention calculation and spliced ​​to obtain the global semantic features of emotional speech;

[0041] In the training process of the joint model, the same batch of training data sets are trained in different training rounds. The training round i = 1 can be initialized and the training round iteration is performed at i = i + 1. The parameter of the joint model in the i-th training is ξ i .

[0042] In this embodiment, the joint model includes multi-head attention encoding, transposed Mamba module and classifier module. The initial speech feature vector obtains the global semantic features and key information of emotional speech through multi-head attention encoding. The selective forgetting and information filtering characteristics of the transposed Mamba module in multiple directions of time are used to capture the temporal features and local patterns in the global semantic features of emotional speech, thereby obtaining complex speech temporal features. Then, the classifier module predicts the complex speech temporal features and the assigned emotional category to achieve efficient and accurate speech emotion recognition. For details, see steps 2 to 4.

[0043] The multi-head attention encoding process of the initial speech feature vector is as follows (a1) to (a4):

[0044] (a1) Multi-head attention encoding first uses formula (1) to train the initial speech feature vector x i Perform layer normalization and then perform linear feature mapping to obtain different subspace features, including the query feature Q of the i-th training i , key feature K i Sum value feature V i :

[0045]

[0046] In formula (1), norm(·) represents the layer normalization operation, W l i (·) represents the fully connected layer during the i-th training.

[0047] Next, the query features Q in different subspaces are i , key feature K i Sum value feature V i Split along the feature dimension according to the number of heads, and get and The number of heads of multi-head attention is denoted as h. For example, in this embodiment, h can be taken as 8. represents the query vector of the h-th attention head during the i-th training, represents the key vector of the h-th attention head during the i-th training, Represents the value vector of the h-th attention head during the i-th training.

[0048] (a2) Multi-head attention encoding uses formula (2) to calculate the internal of the h-th attention head during the i-th training and The attention weight between That is, scaled dot product attention;

[0049]

[0050] In formula (2), Softmax(·) represents the Softmax activation function, T represents the transposition operation, and d represents the scaling factor.

[0051] By using the multi-head self-attention mechanism, the joint model can simultaneously focus on different semantic features and contextual information, thereby encoding richer feature representations. The schematic diagram of the multi-head attention encoding module is shown in Figure 3 As shown, this helps the joint model better understand the complex emotional content in speech.

[0052] (a3) The multi-head attention encoding is used to calculate the weighted self-attention encoding feature of the h-th attention head during the i-th training using formula (3) This allows us to focus on more important global semantic features and contextual information, filter out noise information in speech signals, and focus on important correlation information.

[0053]

[0054] In formula (3), Represents the square root of the scaling factor d.

[0055] (a4) Multi-head attention encoding During the i-th training, the global semantic feature a of emotional speech encoded by multi-head attention is obtained using formula (4) i :

[0056]

[0057] In formula (4), Concat(·) represents the concatenation operation of multiple attention heads, W l i Represents a linear fully connected layer. Global semantic features of emotional speech a i This provides the joint model with richer contextual information, helping to more accurately identify the potential high-dimensional emotional states in speech.

[0058] Step 3: Input the global semantic features of emotional speech into the transposed Mamba module, perform a two-way parallel operation on the global semantic features of emotional speech, and add a gating mechanism to the two-way parallel operation to obtain the forward branch state selection output and the transposed branch state selection output. The forward branch state selection output and the transposed branch state selection output are fused to obtain the complex speech time series features;

[0059] The transposed Mamba module processes the global semantic features of emotional speech as follows:

[0060] (b1) During the i-th training, the transposed Mamba module uses formula (5) to transform the global semantic feature a i Perform two-way parallel operation to construct the forward branch input G iWith transposed branch input

[0061]

[0062] In formula (5), norm(·) represents the layer normalization operation, and Flip(·) represents the transposition flip operation along the time series feature dimension.

[0063] The dual-path parallel Mamba can better capture the local patterns and long- and short-term temporal features in complex emotional speech through selective forgetting and information filtering in multiple directions of time sequence. The transposed Mamba module is shown in the figure. Figure 4 shown.

[0064] (b2) During the i-th training, the transposed Mamba module uses formula (6) to calculate the forward branch input G i With transposed branch input After selectively forgetting some information, the current state M of the forward branch is obtained i and transpose branch current state

[0065]

[0066] In formula (6), W c (·) represents a one-dimensional convolutional layer, where SSM(·) represents the selective state space model after discretization of the continuous state space model, Sigmoid(·) represents the Sigmoid activation function, and W l represents a linear fully connected layer.

[0067] The selective state space model SSM(·) in formula (6) is introduced as follows: The continuous state space model SSM(·) refers to the implicit potential state Use formula (7) to transform an input equation or continuous sequence Mapping to an output equation or sequence

[0068] s′(t)=Ps(t)+QX(t);

[0069] y(t)=Zs′(t); (7)

[0070] In formula (7), represents the learnable matrix parameters, s(t) represents the potential state representation at any given time t, and s′(t) represents the current state after calculation.

[0071] Preferably, for the discrete speech sequence input X t, the output y of the discretized state space model can be calculated using equation (8) using the zero-order hold (ZOH) method t :

[0072]

[0073] y t =Zs t ; (8)

[0074] In formula (8), s t-1 represents the potential state representation at time t-1, s t Indicates the current state after calculation, and Represented as discrete parameters of a discrete state-space model.

[0075] As shown in the formula structure in formula (8), in order to achieve an efficient parallel architecture to maximize the computational efficiency, theoretically, the invariance of the matrix parameters in the selective state space model can be used to obtain formula (9):

[0076]

[0077] In formula (9), N represents the length of the sequence X input to the discrete state space model, Represents a structured convolution kernel.

[0078] To achieve selective processing of input information, the SSM block in the Transposed Mamba module transforms the matrices Q and Z in the state-space model into variable matrix parameters that adapt to the input sequence X. Therefore, due to the dynamic nature of the matrix parameters, the selective state-space equations cannot utilize the parallel training capabilities provided by convolution. Furthermore, the SSM block in the Transposed Mamba module employs a parallel scanning algorithm. This algorithm leverages the multi-level structure of computer memory and optimizes data movement between memory devices of varying speeds, significantly reducing repeated reads and writes to the high-bandwidth but slow-access main memory.

[0079] (b3) During the i-th training, the transposed Mamba module uses Equation (10) to add a gating mechanism to Equation (6) to achieve information filtering and updating of speech features, and obtain the forward branch state selection output F i Select output with transposed branch state

[0080]

[0081] In formula (10), Gated(·) represents the gating mechanism in the transposed Mamba module, and has:

[0082]

[0083] In formula (11), ⊙ represents the Hadamard product operation of the matrix.

[0084] (b4) During the i-th training, the transposed Mamba module uses formula (12) to select the output F of the forward branch state. i Select output with transposed branch state Perform the fusion operation to obtain the complex speech time series feature E after being processed by the transposed Mamba module i ;

[0085]

[0086] This implementation introduces the transposed Mamba block, leveraging its long-term, multi-directional selective forgetting and information filtering properties to effectively address the performance bottlenecks of traditional models when processing long speech sequences. This not only improves the computational efficiency of the joint model but also enhances its ability to capture complex temporal patterns, opening up possibilities for real-time speech emotion recognition applications.

[0087] Step 4: Input the complex speech time series features into the classifier module to obtain the final emotion prediction value. Based on the emotion prediction value and the corresponding true emotion label, a cross-entropy error loss is constructed to adjust the trainable parameters in the joint model.

[0088] During the i-th training, the classifier module uses formula (13) to transform the complex speech time sequence features E obtained by the transposed Mamba module into i Passed into the classifier module to obtain the final emotion prediction value Emo i :

[0089] Emo i =EmotionClassifier(E i ); (13)

[0090] In formula (13), EmotionClassifier(·) represents the classifier module, and has:

[0091] EmotionClassifier(E i )=MLP(Avgpool(E i )); (14)

[0092] In formula (14), Avgpool(·) represents the average pooling operation, and MLP(·) represents the multi-layer perceptron.

[0093] The emotion prediction value Emo obtained by the joint model during the i-th training is converted into i The cross entropy error loss Loss(ξ i), the cross entropy loss function is used to quantify the difference between the joint model prediction probability distribution and the true label distribution, guiding the optimization of the joint model to improve the prediction accuracy:

[0094]

[0095] In formula (15), Emo i represents the sentiment prediction value during the i-th training, Indicates the true emotional label, Represents the cross entropy loss function, then the mathematical expression for a set of N training samples can be expressed as:

[0096]

[0097] Finally, the error Loss(ξ i ) converges, that is, the loss value tends to be unchanged; if it converges, it means that the parameter ξ of the joint model during the i-th training i is the optimal parameter ξ * , and with the optimal parameter ξ * The corresponding joint model is used as the final model for speech emotion recognition; otherwise, the i-th gradient back propagation is continued and the parameter ξ is updated. i , and get the network parameters ξ during the i+1th training i+1 Then, assign i+1 to i and return to step 2 to continue executing until the joint model converges.

[0098] Through steps one to four, this embodiment combines the multi-head attention mechanism and the transposed Mamba network to achieve an organic fusion of global semantic understanding and long-term feature capture in local modes. This unique architectural design not only improves the joint model's ability to understand complex emotional speech, but also significantly enhances the accuracy and robustness of recognition, opening up a new research direction in the field of speech emotion recognition. In addition, the introduction of the transposed Mamba block effectively solves the performance bottleneck of traditional models in processing long-sequence speech through its long-term multi-directional selective forgetting and information filtering characteristics. This innovation not only improves the computational efficiency of the model, but also enhances the ability to capture complex temporal patterns, providing possibilities for real-time speech emotion recognition applications.

[0099] Furthermore, this embodiment builds an end-to-end, efficient speech emotion recognition framework. This approach not only simplifies the traditional recognition process but also significantly improves the model's generalization capabilities, laying a solid foundation for the widespread application of speech emotion recognition technology in multiple fields and possessing significant practical value.

[0100] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A speech emotion recognition method based on the joint attention mechanism and transposed Mamba, characterized by: Input the speech to be recognized into the trained joint model to output the emotion recognition result of the speech; The training process of the joint model is as follows: Step 1: Obtain multiple emotion-embedded speech sounds and convert them into initial speech feature vectors through a general language pre-training model to form a training dataset; Step 2: In the i-th training round, the initial speech feature vector is input into the joint model, and the initial speech feature vector is layer-normalized and mapped through multi-head attention encoding to obtain query features, key features, and value features. The query features, key features, and value features are split according to the number of heads of multi-head attention to obtain the query vector, key vector, and value vector of each attention head. The query vector, key vector, and value vector are subjected to dot product attention calculation and spliced ​​to obtain the global semantic features of emotional speech; Step 3: Input the global semantic features of emotional speech into the transposed Mamba module, perform a two-way parallel operation on the global semantic features of emotional speech, and add a gating mechanism to the two-way parallel operation to obtain the forward branch state selection output and the transposed branch state selection output. The forward branch state selection output and the transposed branch state selection output are fused to obtain the complex speech time series features; Step 4: Input the complex speech time series features into the classifier module to obtain the final emotion prediction value. Based on the emotion prediction value and the corresponding true emotion label, a cross-entropy error loss is constructed to adjust the trainable parameters in the joint model.

2. The speech emotion recognition method of the joint attention mechanism and transposed Mamba according to claim 1 is characterized in that In step 2, dot product attention is calculated on the query vector, key vector, and value vector and concatenated to obtain the global semantic features of emotional speech, specifically: Calculate the attention weight between the query vector and the key vector in each attention head, and calculate the dot product between the attention weight and the value vector to obtain the weighted self-attention encoding feature of each attention head; The weighted self-attention encoding features of each attention head are concatenated and output through a linear fully connected layer to obtain the multi-head attention feature. The initial speech feature vector is added to the multi-head attention feature to obtain the global semantic feature of the emotional speech.

3. The speech emotion recognition method of the joint attention mechanism and transposed Mamba according to claim 1 is characterized in that In step 3, a two-way parallel operation is performed on the global semantic features of emotional speech, and a gating mechanism is added to the two-way parallel operation to obtain the forward branch state selection output and the transposed branch state selection output, as follows: Normalize the global semantic features of emotional speech to construct the forward branch input of the transposed Mamba module; Performing a transposition flip operation on the global semantic features of the emotional speech to construct a transposition branch input of the transposition Mamba module, wherein the transposition flip operation is a transposition flip operation along the time series feature dimension; Calculate the selectively forgotten information of the forward branch input and the transposed branch input respectively to obtain the current state of the forward branch and the current state of the transposed branch; Gating mechanisms are added to the current state of the forward branch and the current state of the transposed branch respectively to obtain the forward branch state selection output and the transposed branch state selection output.

4. The speech emotion recognition method of the joint attention mechanism and transposed Mamba according to claim 3 is characterized in that After calculating the selectively forgotten part of the forward branch input and the transposed branch input respectively, the current state of the forward branch and the current state of the transposed branch are obtained. Specifically, the forward branch input and the transposed branch input pass through the one-dimensional convolution layer, the activation layer and the selective state space model in sequence to obtain the current state of the forward branch and the current state of the transposed branch.

5. The speech emotion recognition method of the joint attention mechanism and transposed Mamba according to claim 3 is characterized in that Add gating mechanisms to the current state of the forward branch and the current state of the transposed branch respectively to obtain the forward branch state selection output and the transposed branch state selection output, specifically: F i =Gated(M i ,G i ); Among them, F i Select the output for the forward branch state, Select output for the transposed branch state, M i is the current state of the forward branch, is the current state of the transposed branch, G i represents the forward branch input, represents the transposed branch input, Gated(·) represents the gating mechanism in the transposed Mamba module, and: Gated(M i ,G i )=M i ⊙Sigmoid(W l (G i )) Among them, ⊙ represents the Hadamard product operation of the matrix, Sigmoid represents the Sigmoid activation function, W l represents a linear fully connected layer.

Images