Voice emotion recognition model and method based on cross-space-time fusion attention network

Abstract

The invention discloses a voice emotion recognition model and method based on a cross-time-space fusion attention network, and belongs to the technical field of voice processing and artificial intelligence. The voice emotion recognition model based on the cross-space-time fusion attention network comprises a frequency domain feature extraction module, a time domain feature extraction module, a feature fusion module and an emotion classification module. The voice signals are extracted through short-time Fourier transform and a Mel filter group to generate a speech spectrogram and a Mel spectrogram, and the speech spectrogram and the Mel spectrogram serve as input of the frequency domain feature extraction module to extract frequency domain features; the eGeMAPS features of the voice signals are subjected to frame-level extraction to form a time sequence feature matrix, the time sequence feature matrix is input into a time domain feature extraction module, and time domain features are extracted; the feature fusion module fuses the frequency domain features and the time domain features into unified high-dimensional features; and finally inputting an emotion classification module and outputting an emotion category. Therefore, the recognition precision of various complex voice emotions is effectively improved, and the method has a good application prospect and popularization value.

Description

Technical Field

[0001] The present invention belongs to the field of speech processing and artificial intelligence technology, and specifically relates to a speech emotion recognition model and method based on a cross-temporal and spatial fusion attention network. Background Art

[0002] With the development of artificial intelligence, speech recognition, and human-computer interaction technologies, speech emotion recognition (SER), as a key component of affective computing, is increasingly being used in fields such as intelligent voice assistants, virtual reality, and intelligent customer service systems. Speech not only carries linguistic information but also contains rich emotional characteristics. Accurately identifying emotional states from speech has become a key technical issue in intelligent interaction research.

[0003] Traditional speech emotion recognition methods primarily rely on low-level acoustic features such as pitch, energy, and formants, and employ machine learning algorithms such as support vector machines (SVMs) or Gaussian mixture models (GMMs) for classification. However, these methods have limited ability to model emotional changes and struggle to accurately capture the complex dynamics of emotional expression.

[0004] In recent years, deep learning has demonstrated its powerful capabilities in automatic feature extraction and modeling. Convolutional neural networks (CNNs), long short-term memory networks (LSTMs), and their variants have been widely used in SER tasks. CNNs can extract spatial features from spectrograms, while LSTMs excel at modeling temporal dependencies within sequences. However, CNNs have limitations in modeling long-term context, and LSTMs also face challenges in capturing global dependencies. To address these shortcomings, researchers have begun introducing the Transformer architecture, which uses a self-attention mechanism to improve its ability to model global information.

[0005] In addition, a single acoustic feature is difficult to fully express the multi-dimensional emotional information in speech, resulting in insufficient model robustness. Summary of the Invention

[0006] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide a speech emotion recognition model and method based on a cross-temporal and spatial fusion attention network, which solves the problems in the existing technology.

[0007] The purpose of the present invention can be achieved through the following technical solutions:

[0008] The speech emotion recognition model based on the cross-temporal and spatial fusion attention network includes: frequency domain feature extraction module, time domain feature extraction module, feature fusion module and emotion classification module;

[0009] The speech signal is extracted through short-time Fourier transform and Mel filter bank to generate a spectrogram and a Mel-spectrogram, which are used as input to the frequency domain feature extraction module to extract frequency domain features. The eGeMAPS features of the speech signal are extracted at the frame level to form a time series feature matrix, which is input into the time domain feature extraction module to extract time domain features. The feature fusion module fuses the frequency domain features and time domain features into a unified high-dimensional feature. Finally, the feature is input into the emotion classification module to output the emotion category.

[0010] Furthermore, the frequency domain feature extraction module includes two independent two-dimensional convolutional neural networks to process the spectrogram and the mel-spectrogram respectively;

[0011] The two-dimensional convolutional neural network consists of multiple convolutional layers. First, a 5×5 convolution kernel is used to perform preliminary feature extraction on the spectrogram and mel-spectrogram, followed by batch normalization, ReLU activation function, and max pooling operations. Then, a 3×3 convolution kernel is used to further extract local features. Batch normalization, ReLU activation function, and max pooling are applied after each convolution to achieve preliminary processing of the two feature maps.

[0012] After the two feature maps are preliminarily processed, they are concatenated and fused. The fused feature maps are again subjected to two sets of 3×3 convolution operations, each of which is followed by batch normalization, ReLU activation function, and maximum pooling. Finally, the output high-dimensional feature map is flattened and reduced to a unified 128-dimensional frequency domain feature vector through a fully connected layer.

[0013] Furthermore, the time domain feature extraction module includes: a bidirectional long short-term memory network and a Transformer encoder; the time series feature matrix is input into the bidirectional long short-term memory network to capture local time series features, and then the output of the bidirectional long short-term memory network is input into three stacked Transformer encoders, which further model the global time series dependency by adding position encoding and utilizing the self-attention mechanism, thereby extracting more discriminative deep time series features.

[0014] Furthermore, the feature fusion module includes: a feature splicing structure and a multi-head attention mechanism. The time domain features and the frequency domain features are spliced by the feature splicing structure to obtain a fused feature vector; then the fused feature vector is fused into a unified high-dimensional feature through the multi-head self-attention mechanism.

[0015] Furthermore, the emotion classification module includes: a two-layer fully connected network and a Softmax classifier to reduce the dimensionality and classify the fused unified high-dimensional features; each layer of the fully connected network includes: Dropout, batch normalization structure and ReLU activation function.

[0016] A conversational robot is equipped with the above-mentioned speech emotion recognition model based on the cross-temporal and spatial fusion attention network.

[0017] The speech emotion recognition method based on the cross-temporal and spatial fusion attention network includes:

[0018] The speech signal is extracted by short-time Fourier transform and Mel filter bank to generate spectrogram and Mel spectrum, and the eGeMAPS features of the speech signal are extracted at the frame level to form a time series feature matrix.

[0019] The spectrogram, mel-spectrogram, and temporal feature matrix are input into the above-mentioned speech emotion recognition model based on the cross-temporal and spatial fusion attention network to output the emotion category.

[0020] A computer storage medium stores a readable program, which, when executed by a processor, can execute the above-mentioned speech emotion recognition method based on a cross-temporal and spatial fusion attention network.

[0021] An electronic device comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus;

[0022] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the above-mentioned speech emotion recognition method based on cross-temporal and spatial fusion attention network.

[0023] A computer program product includes computer instructions, which instruct a computing device to perform operations corresponding to the above-mentioned method for speech emotion recognition based on a cross-temporal and spatial fusion attention network.

[0024] Beneficial effects of the present invention:

[0025] 1. The present invention integrates three types of acoustic features: spectrogram, mel-spectrogram and eGeMAPS, making full use of the complementarity between different modalities to enhance the model's sensitivity and expressiveness to complex emotions.

[0026] 2. The present invention introduces parallel frequency domain and time domain modeling paths to capture spatial and temporal information respectively, solving the problem of incomplete emotion capture by a single modeling path in existing methods.

[0027] 3. The present invention adopts a BiLSTM+Transformer joint architecture, in which BiLSTM is used to model local temporal dependencies and Transformer is used to model global context, effectively improving the ability to capture long-term dependency information while reducing the complexity of model training.

[0028] 4. In the feature fusion stage, the present invention uses a multi-head attention mechanism to model the correlation between frequency domain and time domain features, realizing dynamic weighting and deep semantic understanding among multiple features. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0030] Figure 1 It is a structural diagram of the multi-feature fusion model of the present invention;

[0031] Figure 2 Schematic diagram of the BiLSTM-Transformer structure of the present invention;

[0032] Figure 3 This is an example diagram of the multi-head attention mechanism structure of the present invention;

[0033] Figure 4 This is the architecture diagram of the speech emotion recognition system proposed in Example 4. DETAILED DESCRIPTION

[0034] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0035] Example 1

[0036] The speech emotion recognition model based on the cross-temporal and spatial fusion attention network includes: a frequency domain feature extraction module, a time domain feature extraction module, a feature fusion module and an emotion classification module, forming a dual-channel parallel modeling architecture.

[0037] The speech signal is extracted by the Librosa tool library using short-time Fourier transform and Mel filter bank to generate a spectrogram and a Mel-spectrogram, which are used as input to the frequency domain feature extraction module to extract frequency domain features. The eGeMAPS features of the speech signal are extracted at the frame level to form a time series feature matrix, which is input into the time domain feature extraction module to extract time domain features. The feature fusion module fuses the frequency domain features and time domain features into a unified high-dimensional feature. Finally, the feature is input into the emotion classification module to output the emotion category.

[0038] The eGeMAPS feature set is extracted from the openSMILE toolkit. openSMILE is an open source toolkit for audio feature extraction and classification of speech and music signals, and is widely used in automatic emotion recognition in affective computing.

[0039] 1) Frequency domain feature extraction module

[0040] like Figure 1 As shown in the figure, the frequency domain feature extraction module consists of two independent two-dimensional convolutional neural networks (2D-CNNs). The spectrogram and mel-spectrogram are input into the two 2D-CNNs to extract multi-scale spatial features. The 2D-CNNs contain multiple convolutional layers. First, a 5×5 convolution kernel is used to perform preliminary feature extraction on the spectrogram and mel-spectrogram, respectively. Then, batch normalization, ReLU activation function, and max pooling are applied in sequence. Next, a 3×3 convolution kernel is used to further extract local features. Batch normalization, ReLU activation function, and max pooling are also applied after each convolution to enhance the stability and robustness of the model.

[0041] After initial processing of the two feature maps, they are concatenated and fused to create a richer representation. The fused feature maps are then subjected to two sets of 3×3 convolutions, each followed by batch normalization, ReLU activation, and max pooling. Finally, the high-dimensional feature maps are flattened and reduced to a unified 128-dimensional frequency-domain feature vector using a fully connected layer for subsequent tasks.

[0042] 2) Time domain feature extraction module

[0043] like Figure 2 As shown in the figure, the eGeMAPS features corresponding to the speech signal are extracted at the frame level at intervals of 200ms, and each frame contains 88-dimensional acoustic parameters; a time series feature matrix with a shape of 88×t is generated for each speech segment, where t is the number of time steps.

[0044] The time domain feature extraction module includes a bidirectional long short-term memory network (BiLSTM) and a Transformer encoder. The time series feature matrix is first input into the BiLSTM to capture local time series features. Then, the output of the BiLSTM is input into the Transformer encoder to add position encoding to enhance the global context modeling capability. After adding the position encoding, the time domain features are finally output through CNN.

[0045] BiLSTM consists of two LSTMs, forward and reverse, which process feature sequences in order and reverse order respectively to capture contextual emotional clues. Each time step of BiLSTM is calculated through the following gating mechanism:

[0046] f t =σ(W f *[h t-1 ,x t ]+b f ) (1)

[0047] i t =σ(Wi *[h t-1 ,x t ]+b i ) (2)

[0048]

[0049] o t =σ(W o *[h t-1 ,x t ]+b o ) (5)

[0050] h t =o t *tanh(C t ) (6)

[0051] Among them, σ is the activation function, x t is the input sequence at time t, C t is the cell state, is the temporary cell state, h t is the hidden layer state of the current time step, h t-1 is the hidden state of the previous time step, f t For the forget gate, i t For the memory gate, o t is the output gate, W f is the weight matrix of the forget gate, W i is the weight matrix of the input gate, W c is the weight matrix of the memory unit, W o is the weight matrix of the output gate, b f is the bias term of the forget gate, b i is the bias term of the input gate, b c is the bias term of the memory unit, b o is the bias term of the output gate. The three gates control the transmission of information in and out of the unit, and finally obtain a hidden layer state sequence with the same length as the input sequence.

[0052] In BiLSTM, the forward and backward hidden states are concatenated at each time step to more fully capture contextual information. The forward and backward concatenations are updated according to the following formula:

[0053]

[0054] in, is the state of the forward LSTM at time step t, is the state of the inverse LSTM at time step t.

[0055] The Transformer encoder adds positional encoding to the BiLSTM output:

[0056]

[0057] Among them, P t represents the position encoding vector at the t-th time step, is the state of BiLSTM at time step t, Z t Represents the vector after adding position encoding.

[0058] After adding the position encoding, it is calculated through the multi-head self-attention mechanism, and the multi-head self-attention mechanism is updated according to the following formula:

[0059]

[0060] Among them, Q is the query vector, which is the input Z t Linear mapping Q = Z t W Q , K is the key vector, which is the input Z t Linear mapping K = Z t W K , V is the value vector, which is the input Z t Linear mapping V = Z t W V , d k is the key vector dimension and softmax is the activation function.

[0061] This fusion mechanism helps to explore the complementarity and importance between different modal features.

[0062] 3) Feature fusion module

[0063] The feature fusion module includes a feature concatenation structure and a multi-head attention mechanism (MHA). The time domain features and frequency domain features are concatenated by the feature concatenation structure to obtain a fused feature vector to unify the feature dimensions. The fused feature vector is then fused into a unified high-dimensional feature through a multi-head self-attention mechanism.

[0064] Specifically, let the frequency domain features and time domain features be In order to unify the feature dimension, the two types of features are first concatenated to obtain the fused feature vector F con :

[0065]

[0066] Wherein, d=d1+d2, and in this embodiment, d=256.

[0067] Subsequently, the fused feature vector is fed into the multi-head attention module to fully exploit the interdependencies and importance between different feature components. The multi-head attention mechanism first generates the Q, K, and V matrices through linear mapping:

[0068] Q=F con W Q ,K=F con W K ,V=F con W V (10)

[0069] in, is the learnable weight matrix.

[0070] The output of each attention head is calculated as follows:

[0071]

[0072] After concatenating the outputs of all attention heads, we can get the multi-head attention output through linear transformation:

[0073] MHA(F con )=Concat(head1,...,head h )W O (12)

[0074] Among them, h is the number of heads, W O is the weight matrix of the output linear layer.

[0075] 4) Emotion Classification Module

[0076] The emotion classification module includes: a two-layer fully connected network (FC) and a Softmax classifier, which are used to reduce the dimensionality and classify the fused high-dimensional features. Each layer of the fully connected network contains a random dropout layer (Dropout), a batch normalization layer (BatchNorm), and a ReLU activation function in sequence. Dropout effectively prevents model overfitting and improves generalization ability by randomly blocking some neuron outputs; BatchNorm normalizes the input data, stabilizes the training process and accelerates convergence; the ReLU activation function gives the network nonlinear expression capabilities and enhances the ability to discriminate complex features. The three work together to achieve training stability and efficiency, while improving the generalization performance and classification accuracy of the model;

[0077] Finally, the fused features are input into a Softmax classifier, which outputs a probability distribution of seven emotions: anger, happiness, sadness, surprise, fear, disgust, and neutral. The class label is determined by the emotion category corresponding to the maximum probability.

[0078] Example 2

[0079] In this embodiment, the superiority of the model of the present invention is verified through specific experiments;

[0080] The test was conducted on the CASIA Chinese emotional speech corpus and the EMO-DB German emotion dataset. The test process is as follows:

[0081] This model was implemented using the PyTorch deep learning framework. The hardware configuration included an NVIDIA GeForce RTX 3080 GPU with 32GB of RAM and an Intel Core i7-14700K CPU. Simulations were conducted in Python. To optimize model performance, the Adam optimizer was used, along with the categorical cross-entropy loss function. The parameters were set to a batch size of 32, an initial learning rate of 0.001, and 100 optimization iterations. The optimizer's learning rate was dynamically adjusted during training to ensure optimal convergence.

[0082] The test results show that while ensuring the stability of the model, the emotion recognition accuracy of the model using the present invention reached 93.41% and 92.46% on the CASIA and EMO-DB datasets, respectively, which is significantly better than traditional CNN-LSTM, CNN-Transformer and other structures.

[0083] Example 3

[0084] Based on the speech emotion recognition model based on the cross-temporal and spatial fusion attention network mentioned in Example 1, this embodiment proposes a speech emotion recognition method based on the cross-temporal and spatial fusion attention network, including:

[0085] The speech signal is extracted using short-time Fourier transform and Mel filter group to generate spectrogram and Mel-spectrogram, and the eGeMAPS features of the speech signal are extracted at the frame level to form a temporal feature matrix. The spectrogram, Mel-spectrogram and temporal feature matrix are input into the speech emotion recognition model based on the cross-temporal and spatial fusion attention network to output the emotion category.

[0086] Based on similar inventive concepts, an embodiment of the present invention also provides a computer storage medium storing a readable program, which, when executed by a processor, can execute the above-mentioned speech emotion recognition method based on a cross-temporal and spatial fusion attention network.

[0087] Based on similar inventive concepts, an embodiment of the present invention provides an electronic device, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus;

[0088] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the above-mentioned speech emotion recognition method based on cross-temporal and spatial fusion attention network.

[0089] Based on similar inventive concepts, an embodiment of the present invention further provides a computer program product, including computer instructions, which instruct a computing device to execute operations corresponding to the above-mentioned load forecasting method.

[0090] The method of the present invention can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CDROM, RAM, floppy disk, hard disk or magneto-optical disk), or as computer code that is originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded over a network and will be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a special-purpose processor or programmable or special-purpose hardware (such as an ASIC or FPGA). It will be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown here, the execution of the code converts the general-purpose computer into a special-purpose computer for executing the method shown here.

[0091] Example 4

[0092] In this embodiment, based on the speech emotion recognition model based on the cross-temporal and spatial fusion attention network proposed in Example 1, a multi-feature fusion speech emotion recognition system is proposed; Figure 4 As shown, it includes: data acquisition, parameter initialization, parameter adjustment, interface authentication management, device management, log management, prediction result management and prediction result analysis modules.

[0093] The data acquisition module completes voice signal acquisition through local audio input or calling external device interfaces, and stores raw voice data for subsequent processing. The parameter initialization module is used to load the initial model parameters and configure the operating environment, including feature dimensions, network structure, optimizer settings, etc. The parameter adjustment module supports runtime adjustment of model structure or hyperparameters to meet flexible deployment or experimental requirements. The interface authentication management module ensures the security of interface call permissions and prevents illegal access or parameter tampering. The device management module is used to configure external devices that interact with the recognition system, such as microphones and audio terminals. The log management module is used to record historical operation information such as model calls, parameter modifications, and recognition outputs. The prediction result management and analysis module is used to archive classification results, user behavior statistics, and visualization. The STFNet library management module is responsible for the backbone library management of the emotion recognition model, supporting multi-feature input and attention fusion operations. The multi-feature fusion strategy is constructed into a module STFNet library and the STFNet library is made public so that different models can call the appropriate multi-feature fusion strategy to achieve model optimization.

[0094] The system integrates Interface 8 as a parameter configuration entry point for setting key parameters in speech feature extraction and model calculation, including the number of FFT points, number of Mel filters, window function type, eGeMAPS feature template, convolution kernel size, number of attention heads, and number of classification categories. It also supports setting training hyperparameters such as learning rate, batch size, and number of training rounds. The system's main processing flow is driven by the STFNet (Spatio-Temporal Fusion Network) library, and Interfaces 1 through 6 rely on this library for operation. Interface 1 calls STFNet to complete audio data acquisition and preprocessing. Interface 2 uses its spectrum transformation and feature extraction modules to generate spectrograms, Mel-spectrograms, and eGeMAPS feature sets. Interfaces 3 through 5 use STFNet's built-in CNN, BiLSTM, and Transformer modules to encode and model various features. Interface 6 uses STFNet's multi-feature fusion architecture and multi-head attention mechanism to compress and weight the extracted features, ultimately outputting a fused representation. After obtaining the STFNet output, interface 7 performs sentiment classification confidence calculation and structured analysis for sentiment label output, result recording, and user trend evaluation, forming a complete recognition and feedback closed loop. Table 1 shows an example of the STFNet library simulation process.

[0095] Table 1 STFNet library simulation process processing example

[0096]

[0097]

[0098]

[0099]

[0100]

[0101] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention, and such changes and modifications fall within the scope of the invention as claimed.

Claims

1. A speech emotion recognition model based on a cross-temporal and spatial fusion attention network, characterized by: include: Frequency domain feature extraction module, time domain feature extraction module, feature fusion module and emotion classification module; The speech signal is extracted through short-time Fourier transform and Mel filter bank to generate spectrogram and Mel spectrum, which are used as input of the frequency domain feature extraction module to extract frequency domain features; The eGeMAPS features of the speech signal are extracted at the frame level to form a time series feature matrix, which is then input into the time domain feature extraction module to extract the time domain features. feature The fusion module fuses the frequency domain features and time domain features into a unified high-dimensional feature; finally, it inputs the emotion classification module and outputs the emotion category.

2. The speech emotion recognition model based on the cross-temporal and spatial fusion attention network according to claim 1 is characterized in that The frequency domain feature extraction module includes two independent two-dimensional convolutional neural networks to process the spectrogram and the mel-spectrogram respectively; The two-dimensional convolutional neural network consists of multiple convolutional layers. First, a 5×5 convolution kernel is used to perform preliminary feature extraction on the spectrogram and mel-spectrogram, followed by batch normalization, ReLU activation function, and max pooling operations. Then, a 3×3 convolution kernel is used to further extract local features. Batch normalization, ReLU activation function, and max pooling are applied after each convolution to achieve preliminary processing of the two feature maps. After the two feature maps are preliminarily processed, they are concatenated and fused. The fused feature maps are again subjected to two sets of 3×3 convolution operations, each of which is followed by batch normalization, ReLU activation function, and maximum pooling. Finally, the output high-dimensional feature map is flattened and reduced to a unified 128-dimensional frequency domain feature vector through a fully connected layer.

3. The speech emotion recognition model based on the cross-temporal and spatial fusion attention network according to claim 1 is characterized in that The time domain feature extraction module includes: a bidirectional long short-term memory network and a Transformer encoder; the time series feature matrix is input into the bidirectional long short-term memory network to capture local time series features, and then the output of the bidirectional long short-term memory network is input into three stacked Transformer encoders. By adding position encoding and using the self-attention mechanism to further model the global time series dependency, deep time series features are extracted.

4. The speech emotion recognition model based on the cross-temporal and spatial fusion attention network according to claim 1 is characterized in that The feature fusion module includes: a feature splicing structure and a multi-head attention mechanism. The time domain features and frequency domain features are spliced by the feature splicing structure to obtain a fused feature vector; then the fused feature vector is fused into a unified high-dimensional feature through the multi-head self-attention mechanism.

5. The speech emotion recognition model based on the cross-temporal and spatial fusion attention network according to claim 1 is characterized in that: The emotion classification module includes: a two-layer fully connected network and a Softmax classifier to reduce the dimensionality and classify the fused unified high-dimensional features; each layer of the fully connected network includes: Dropout, batch normalization structure and ReLU activation function.

6. A conversational robot, characterized in that: Equipped with the speech emotion recognition model based on the cross-temporal and spatial fusion attention network as described in any one of claims 1-5.

7. A speech emotion recognition method based on a cross-temporal and spatial fusion attention network, characterized in that: include: The speech signal is extracted by short-time Fourier transform and Mel filter bank to generate spectrogram and Mel spectrum, and the eGeMAPS features of the speech signal are extracted at the frame level to form a time series feature matrix. The spectrogram, mel-spectrogram and time series feature matrix are input into the speech emotion recognition model based on the cross-temporal and spatial fusion attention network described in any one of claims 1-5, and the emotion category is output.

8. A computer storage medium storing a readable program, characterized in that: When the program is executed by the processor, the speech emotion recognition method based on the cross-temporal and spatial fusion attention network as claimed in claim 7 can be executed.

9. An electronic device, characterized in that: include: A processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the speech emotion recognition method based on cross-temporal and spatial fusion attention network as described in claim 7.

10. A computer program product comprising computer instructions, characterized in that The computer instructions instruct the computing device to perform operations corresponding to the speech emotion recognition method based on cross-temporal and spatial fusion attention network as described in claim 7.

Images