Speech emotion recognition method based on multi-scale feature extraction of attention mechanism

Abstract

The application discloses a speech emotion recognition method based on a multi-scale feature extraction of an attention mechanism, and comprises the following steps: constructing a training data set; constructing a speech emotion recognition network model, wherein the speech emotion recognition network model comprises a multi-scale feature extractor module, a multi-scale feature encoder module, a feature fusion module and a speech emotion recognition classifier, the multi-scale feature extractor module is used to obtain multiple speech features of different scales, the multi-scale feature encoder module is used to encode the speech features to obtain speech features of different scales after encoding, the feature fusion module is used to obtain multi-scale speech fusion features, and the speech emotion recognition classifier is used to obtain a final classification result by using the multi-scale speech fusion features; training the speech emotion recognition network model; and obtaining an emotion recognition result of a to-be-recognized speech. The multi-scale feature extractor is used to learn features of speech data under different receptive fields as much as possible, so that the feature diversity is ensured.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a speech emotion recognition method based on multi-scale feature extraction of attention mechanism, which can be used for lie testing, voice customer service and human-computer interaction. Background Technology

[0002] Speech is one of the important ways of human-computer interaction, and as a fundamental research area of ​​affective computing, speech emotion recognition (SER) has become an active research field. Currently, a common approach to SER algorithms is to find as many effective emotional features as possible, which is key to the success of SER, and then perform emotion classification to obtain the recognition result.

[0003] A common approach to speech emotion recognition is to compute low-level features, such as FilterBank (Fbank), Mel-frequency cepstral coefficients (MFCC), linear predictive cepstral coefficients (LPCC), and prosodic features. Then, a neural network is used to obtain high-level feature representations of the data, and finally, emotion classification is performed. Feature representation plays a crucial role in speech emotion recognition (SER), but previous methods mostly used single-type features for recognition tasks or performed early feature fusion. In SER, emotion may exhibit different energy patterns in the spectrograms of different granularities; therefore, a single type of feature cannot accurately reflect the emotion of a speech signal. Furthermore, different features may contain different information, and direct fusion cannot integrate the advantages of different features.

[0004] In recent years, the development of speech emotion recognition has often been driven by deep learning. Thanks to the advancements in deep learning, the emergence of Deep Neural Networks (DNNs) has led to the development of speech emotion recognition methods specifically for DNNs. With further developments in deep learning, Convolutional Neural Networks (CNNs) have achieved promising results in computer vision. Researchers have further demonstrated the effectiveness of CNNs in emotion classification by combining audio features with textual information. Furthermore, research has shown that sequence modeling models based on RNNs can also achieve certain results.

[0005] With the advent of Transformers, attention mechanisms have achieved excellent performance in various fields of deep learning, sparking interest in achieving higher accuracy with attention-based SER models. However, these attention mechanisms can only be computed at a preset granularity and may not be able to dynamically adapt to different regions of interest in the spectrogram.

[0006] The methods mentioned above are all aimed at single-dimensional features or fixed-granularity attention regions. However, speech data is often variable, especially emotional expression, which is dynamic and varies in energy representation in spectrograms. These studies cannot effectively learn different features. Summary of the Invention

[0007] To address the aforementioned problems in existing technologies, this invention provides a speech emotion recognition method based on multi-scale feature extraction using an attention mechanism. The technical problem to be solved by this invention is achieved through the following technical solution:

[0008] This invention provides a speech emotion recognition method based on multi-scale feature extraction using an attention mechanism, comprising:

[0009] S1: Construct a training dataset with language sentiment labels;

[0010] S2: Construct a speech emotion recognition network model, which includes a multi-scale feature extractor module, a multi-scale feature encoder module, a feature fusion module, and a speech emotion recognition classifier. The multi-scale feature extractor module obtains multiple speech features at different scales from the input spectrogram. The multi-scale feature encoder module performs cross-attention calculation on the speech features and the original spectrogram to obtain speech features encoded at different scales. The feature fusion module concatenates the speech features encoded at different scales to obtain multi-scale speech fusion features. The speech emotion recognition classifier uses the multi-scale speech fusion features to obtain the final classification result.

[0011] S3: Train the speech emotion recognition network model using the training dataset to obtain the trained speech emotion recognition network model;

[0012] S4: Train the speech emotion recognition network model with the speech input to be recognized, and obtain the emotion recognition result of the current speech.

[0013] In one embodiment of the present invention, the multi-scale feature extractor module includes multiple feature extractors of different sizes connected in parallel and an adaptive average pooling layer connected after each feature extractor, wherein,

[0014] The last feature extractor in the feature extractor is a fully connected layer, and the remaining feature extractors are all two-dimensional convolutional layers.

[0015] In one embodiment of the present invention, the multi-scale feature extractor module includes a first feature extractor, a second feature extractor, a third feature extractor, a fourth feature extractor, a first adaptive average pooling layer, a second adaptive average pooling layer, a third adaptive average pooling layer, and a fourth adaptive average pooling layer, wherein,

[0016] The first feature extractor is a two-dimensional convolutional layer with a kernel of 3 and an expansion dimension of 1; the second feature extractor is a two-dimensional convolutional layer with a kernel of 5 and an expansion dimension of 2; the third feature extractor is a two-dimensional convolutional layer with a kernel of 7 and an expansion dimension of 3; the fourth feature extractor is a fully connected layer that can map the input data dimension from 640 dimensions to 80 dimensions.

[0017] The input of the first adaptive average pooling layer is connected to the output of the first feature extractor, the input of the second adaptive average pooling layer is connected to the output of the second feature extractor, the input of the third adaptive average pooling layer is connected to the output of the third feature extractor, and the input of the fourth adaptive average pooling layer is connected to the output of the fourth feature extractor.

[0018] In one embodiment of the present invention, the multi-scale feature encoder module includes multiple feature encoders, each feature encoder including a multi-head attention unit, a first residual normalization unit, a feedforward neural network, and a second residual normalization unit, wherein...

[0019] The q-input and k-input of the multi-head attention unit are connected to the output of the corresponding feature extractor, and the v-input of the multi-head attention unit is the original spectrogram. The calculation process of the multi-head attention unit is as follows:

[0020] MultiHead(Q,K,V)=Concat(head1,...,head h W o

[0021] Among them, head i =Attention(QW i Q ,KW i K VW i V ), W i Q W represents the weights of the Q-input of the i-th feature encoder. i K W represents the weights of the K inputs of the i-th feature encoder. i V W represents the weight of the V input of the i-th feature encoder, and h represents the number of heads in the multi-head attention unit; oThis represents the overall weight matrix of the multi-head attention unit. Concat(·) means concatenating the data along a specified dimension, and Attention(·) means performing a dot product attention operation.

[0022] The output of the feature encoder is:

[0023] x ei =FFN(MultiHead(Q,K) i V i )+x i )+x i

[0024] Where, x i This represents the spectrogram of the current input, and FFN represents the feedforward neural network module.

[0025] In one embodiment of the present invention, the multi-scale feature encoder module includes a first feature encoder, a second feature encoder, a third feature encoder, and a fourth feature encoder, wherein,

[0026] The q-input and k-input of the multi-head attention unit of the first feature encoder are connected to the output of the first adaptive average pooling layer;

[0027] The q-input and k-input of the multi-head attention unit of the second feature encoder are connected to the output of the second adaptive average pooling layer;

[0028] The q input and k input of the multi-head attention unit of the third feature encoder are connected to the output of the third adaptive average pooling layer.

[0029] The q-input and k-input of the multi-head attention unit of the fourth feature encoder are connected to the output of the fourth adaptive average pooling layer.

[0030] In one embodiment of the present invention, S3 includes:

[0031] S3.1: Set the parameters of the entire speech emotion recognition network model: initial iteration count K, number of input network samples per iteration, learning rate, and number of cross-validation iterations;

[0032] S3.2: Use the cross-entropy function as the loss function for the entire network to measure the difference between the true probability distribution and the predicted probability distribution;

[0033] S3.3: Input the data in the training dataset into the speech emotion recognition network model, calculate the loss function of the data 100 times in one verification process, and verify a total of 10 times;

[0034] S3.4: Determine whether the loss function has been decreasing throughout the 10 validation cycles: if so, increment K by 1 and return to S3.2; if the loss function stops changing, stop training and save the network model obtained after the current number of validation cycles.

[0035] In one embodiment of the present invention, the loss function is:

[0036]

[0037] Where m is the number of training samples, n is the number of label categories, and p(x) ij q(x) represents the classification probability that the i-th sample actually belongs to the j-th class. ij ) represents the probability that the i-th sample is classified into the j-th category as predicted by the network.

[0038] In one embodiment of the present invention, the method further includes the following step before step S4:

[0039] The pre-constructed test dataset is input into the trained speech emotion recognition network model to obtain the recognition result, which is then compared with the label of the test dataset to verify the speech emotion recognition accuracy of the speech emotion recognition network model.

[0040] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0041] 1. This invention proposes a speech emotion recognition method based on multi-scale feature extraction using an attention mechanism. The proposed multi-scale feature extractor learns the features of speech data under different receptive fields as much as possible, ensuring the diversity of learned features.

[0042] 2. This invention uses a multi-scale feature encoder to fuse multi-scale features with speech data, which improves the feature representation capability of the model and makes better use of the information in the speech features.

[0043] 3. This invention demonstrates, through speech emotion recognition, that the speech emotion recognition method based on attention mechanism proposed in this invention has a higher accuracy than other advanced methods.

[0044] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0045] Figure 1 This is a flowchart of a speech emotion recognition method based on multi-scale feature extraction using an attention mechanism, provided in an embodiment of the present invention.

[0046] Figure 2 This is a schematic diagram of the structure of a speech emotion recognition network model provided in an embodiment of the present invention;

[0047] Figure 3 This is a schematic diagram of the processing procedure of a multi-scale feature extractor provided in an embodiment of the present invention;

[0048] Figure 4 This is a schematic diagram of the processing procedure of a multi-scale encoder provided in an embodiment of the present invention.

[0049] Figure 5 This is a flowchart illustrating the training process of a speech emotion recognition network model provided in an embodiment of the present invention. Detailed Implementation

[0050] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the following describes in detail a speech emotion recognition method based on attention mechanism multi-scale feature extraction proposed according to the present invention, in conjunction with the accompanying drawings and specific embodiments.

[0051] The foregoing and other technical contents, features, and effects of the present invention will be clearly presented in the following detailed description of specific embodiments in conjunction with the accompanying drawings. Through the description of the specific embodiments, a more in-depth and concrete understanding can be gained of the technical means and effects adopted by the present invention to achieve its intended purpose. However, the accompanying drawings are for reference and illustration only and are not intended to limit the technical solutions of the present invention.

[0052] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations are intended to cover non-exclusive inclusion, such that an article or apparatus comprising a list of elements includes not only those elements but also other elements not expressly listed. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the article or apparatus that includes said element.

[0053] Please see Figure 1 , Figure 1 This is a flowchart of a speech emotion recognition method based on multi-scale feature extraction using an attention mechanism, provided by an embodiment of the present invention. The speech emotion recognition method includes:

[0054] S1: Construct a training dataset with language sentiment labels.

[0055] Specifically, from the existing IEMOCAP dataset, which contains 10,039 speech data points, we selected 4,900 data points from the four most valuable categories: anger, sadness, happiness, and calmness. The dataset was divided into training and testing sets at a 10:1 ratio to evaluate the performance of speech emotion recognition under multi-scale feature representation.

[0056] S2: Construct a speech emotion recognition network model. The speech emotion recognition network model includes a multi-scale feature extractor module, a multi-scale feature encoder module, a feature fusion module, and a speech emotion recognition classifier. The multi-scale feature extractor module is used to obtain multiple speech features at different scales from the input spectrogram. The multi-scale feature encoder module is used to perform cross-attention calculation on the speech features and the original spectrogram to obtain speech features encoded at different scales. The feature fusion module is used to concatenate the speech features encoded at different scales to obtain multi-scale speech fusion features. The speech emotion recognition classifier is used to obtain the final classification result using the multi-scale speech fusion features.

[0057] Specifically, the multi-scale feature extractor module includes multiple feature extractors of different sizes connected in parallel, and an adaptive average pooling layer connected after each feature extractor. The last feature extractor in the feature extractor is a fully connected layer, and the remaining feature extractors are all two-dimensional convolutional layers.

[0058] Specifically, please see Figure 2 and Figure 3 The multi-scale feature extractor module includes a first feature extractor, a second feature extractor, a third feature extractor, a fourth feature extractor, a first adaptive average pooling layer, a second adaptive average pooling layer, a third adaptive average pooling layer, and a fourth adaptive average pooling layer. Specifically, the first feature extractor is a two-dimensional convolutional layer with a kernel of 3 and an expanded dimension of 1; the second feature extractor is a two-dimensional convolutional layer with a kernel of 5 and an expanded dimension of 2; the third feature extractor is a two-dimensional convolutional layer with a kernel of 7 and an expanded dimension of 3; and the fourth feature extractor is a fully connected layer capable of mapping the input data dimension from 640 dimensions to 80 dimensions. The input of the first adaptive average pooling layer is connected to the output of the first feature extractor, the input of the second adaptive average pooling layer is connected to the output of the second feature extractor, the input of the third adaptive average pooling layer is connected to the output of the third feature extractor, and the input of the fourth adaptive average pooling layer is connected to the output of the fourth feature extractor.

[0059] Specifically, the raw speech data is processed by a multi-scale feature extractor to obtain speech feature representations at four different scales. An adaptive average pooling layer is added after each feature extractor to reduce the number of parameters in subsequent operations for the extracted multi-scale features.

[0060] It should be noted that in this embodiment, the original speech data is processed through low-level feature extraction to obtain an 80-dimensional Mel-Cepstral (spectral graph), which is then input into four feature extractors of different scales to obtain preliminary feature extraction results. Subsequently, the preliminary feature extraction results are processed through an adaptive average pooling layer to obtain the final multi-scale speech features.

[0061] Further, please see Figure 2 and Figure 4 The multi-scale feature encoder module includes multiple feature encoders, each of which includes a multi-head attention unit, a first residual normalization unit, a feedforward neural network, and a second residual normalization unit.

[0062] The q-input and k-input of the multi-head attention unit are connected to the output of the corresponding feature extractor, and the v-input of the multi-head attention unit is the original spectrogram. The calculation process of the multi-head attention unit is as follows:

[0063] MultiHead(Q,K,V)=Concat(head1,...,head h W o

[0064] Among them, head i =Attention(QW i Q ,KW i K VW i V ), W i Q W represents the weights of the Q-input of the i-th feature encoder. i K W represents the weights of the K inputs of the i-th feature encoder. i V W represents the weight of the V input of the i-th feature encoder, and h represents the number of heads in the multi-head attention unit; o This represents the overall weight matrix of the multi-head attention unit. Concat(·) means concatenating the data along a specified dimension, and Attention(·) means performing a dot product attention operation.

[0065] The output of the feature encoder is:

[0066] x ei =FFN(MultiHead(Q,K) i V i )+x i )+x i

[0067] Where, xi This represents the spectrogram of the current input, and FFN represents the feedforward neural network module.

[0068] In this embodiment, the multi-scale feature encoder module includes a first feature encoder, a second feature encoder, a third feature encoder, and a fourth feature encoder. The q-input and k-input of the multi-head attention unit of the first feature encoder are connected to the output of the first adaptive average pooling layer; the q-input and k-input of the multi-head attention unit of the second feature encoder are connected to the output of the second adaptive average pooling layer; the q-input and k-input of the multi-head attention unit of the third feature encoder are connected to the output of the third adaptive average pooling layer; and the q-input and k-input of the multi-head attention unit of the fourth feature encoder are connected to the output of the fourth adaptive average pooling layer.

[0069] Specifically, the spectrogram of the original speech data is input into four multi-scale feature encoders as the value of the v input in multi-head attention, and is combined with the q input and k input to calculate the encoded speech features.

[0070] Subsequently, the feature fusion module concatenates the speech features encoded at different scales to obtain multi-scale speech fusion features. These multi-scale speech fusion features are then fed into a speech emotion recognition classifier for classification, yielding the final classification result y. i The data is then saved, and the accuracy of the classification results for each emotion is given.

[0071] Classification result y i for:

[0072] y i =Classifier(concat(x) ei ))

[0073] Where i = 1, 2, 3, 4 represent the sequence numbers of the feature encoder, x ei This represents the output of the encoder for the i-th feature.

[0074] S3: Train the speech emotion recognition network model using the training dataset to obtain the trained speech emotion recognition network model.

[0075] In this embodiment, please refer to Figure 2 and Figure 5 S3 includes:

[0076] S3.1: Set the parameters of the entire speech emotion recognition network model: initial iteration count K = 100, batch_size of single input network samples = 32, learning rate lr = 0.001 and cross-validation count kf = 10;

[0077] S3.2: The cross-entropy function is used as the loss function for the entire network to measure the difference between the true probability distribution and the predicted probability distribution. The loss function is:

[0078]

[0079] Where m is the number of training samples, n is the number of label categories, and p(x) ij q(x) represents the classification probability that the i-th sample actually belongs to the j-th class. ij ) represents the probability that the i-th sample is classified into the j-th category as predicted by the network.

[0080] S3.3: Input the data in the training dataset into the speech emotion recognition network model, calculate the loss function of the data 100 times in one verification process, and verify a total of 10 times;

[0081] S3.4: Determine whether the loss function has been decreasing throughout the 10 validation cycles: if so, increment K by 1 and return to S3.2; if the loss function stops changing, stop training and save the network model obtained after the current number of validation cycles.

[0082] S4: Train the speech emotion recognition network model with the speech input to be recognized, and obtain the emotion recognition result of the current speech.

[0083] In this embodiment, the method further includes the following step before step S4:

[0084] The pre-constructed test dataset is input into the trained speech emotion recognition network model to obtain the recognition result, which is then compared with the label of the test dataset to verify the speech emotion recognition accuracy of the speech emotion recognition network model.

[0085] The following simulation further illustrates the effectiveness of the speech emotion recognition method based on attention mechanism multi-scale feature extraction of the present invention.

[0086] (1) Simulation experimental conditions:

[0087] The hardware platform used in the simulation experiment of this invention is as follows: CPU model is Intel(R) i9-7960X; GPU model is GeForce RTX 3090*2, with 24G of video memory; RAM is 16G*8.

[0088] The software platform used in the simulation experiment of this invention is: Ubuntu 20.0LTS system, Python 3.9.2, and PyTorch 1.13.1.

[0089] The dataset used in the simulation experiments of this invention comes from the IEMOCAP dataset.

[0090] (2) Simulation content and result analysis:

[0091] Using the method of this invention, a speech emotion recognition simulation was performed using the IEMOCAP dataset described above. The results are shown in Table 1 below.

[0092] Table 1 Experimental results of the present invention

[0093]

[0094] As shown in Table 1, the speech emotion recognition method based on attention mechanism of the present invention, through multi-scale feature extraction and encoding, has extracted as many emotional features as possible from the speech data. It has achieved good accuracy in classifying and judging the four emotions of anger, happiness, calmness and sadness individually. In particular, it has achieved an accuracy of over 90% for anger and happiness. The overall results also show excellent performance. This indicates that the method proposed in this invention can learn more feature representations in speech data, which has a positive effect on speech emotion recognition and improves the accuracy of speech emotion recognition.

[0095] This invention presents a speech emotion recognition method based on multi-scale feature extraction using an attention mechanism. The proposed multi-scale feature extractor learns features from speech data across different receptive fields, ensuring feature diversity. This invention uses a multi-scale feature encoder to fuse multi-scale features with speech data, enhancing the model's feature representation capability and better utilizing the information within the speech features. Through speech emotion recognition, this invention demonstrates that the accuracy of the proposed attention-based multi-scale feature extraction method is higher than other state-of-the-art methods.

[0096] In the several embodiments provided by this invention, it should be understood that the apparatus and methods disclosed in this invention can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of modules is merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0097] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or in the form of hardware plus software functional modules.

[0098] Another embodiment of the present invention provides a storage medium storing a computer program for executing the steps of the speech emotion recognition method based on attention mechanism multi-scale feature extraction described in the above embodiments. A further aspect of the present invention provides an electronic device including a memory and a processor. The memory stores a computer program, and the processor, when calling the computer program in the memory, implements the steps of the speech emotion recognition method based on attention mechanism multi-scale feature extraction as described in the above embodiments. Specifically, the integrated modules implemented as software functional modules can be stored in a computer-readable storage medium. The software functional modules stored in a storage medium include several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0099] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A speech emotion recognition method based on multi-scale feature extraction using an attention mechanism, characterized in that, include: S1: Construct a training dataset with language sentiment labels; S2: Construct a speech emotion recognition network model, which includes a multi-scale feature extractor module, a multi-scale feature encoder module, a feature fusion module, and a speech emotion recognition classifier. The multi-scale feature extractor module obtains multiple speech features at different scales from the input spectrogram. The multi-scale feature encoder module performs cross-attention calculation on the speech features and the original spectrogram to obtain speech features encoded at different scales. The feature fusion module concatenates the speech features encoded at different scales to obtain multi-scale speech fusion features. The speech emotion recognition classifier uses the multi-scale speech fusion features to obtain the final classification result. S3: Train the speech emotion recognition network model using the training dataset to obtain the trained speech emotion recognition network model; S4: Train the speech emotion recognition network model with the speech input to be recognized, and obtain the emotion recognition result of the current speech.

2. The speech emotion recognition method based on multi-scale feature extraction using an attention mechanism according to claim 1, characterized in that, The multi-scale feature extractor module includes multiple feature extractors of different sizes connected in parallel, and an adaptive average pooling layer connected after each feature extractor. The last feature extractor in the feature extractor is a fully connected layer, and the remaining feature extractors are all two-dimensional convolutional layers.

3. The speech emotion recognition method based on multi-scale feature extraction using an attention mechanism according to claim 2, characterized in that, The multi-scale feature extractor module includes a first feature extractor, a second feature extractor, a third feature extractor, a fourth feature extractor, a first adaptive average pooling layer, a second adaptive average pooling layer, a third adaptive average pooling layer, and a fourth adaptive average pooling layer, wherein... The first feature extractor is a two-dimensional convolutional layer with a kernel of 3 and an expansion dimension of 1; the second feature extractor is a two-dimensional convolutional layer with a kernel of 5 and an expansion dimension of 2; the third feature extractor is a two-dimensional convolutional layer with a kernel of 7 and an expansion dimension of 3; the fourth feature extractor is a fully connected layer that can map the input data dimension from 640 dimensions to 80 dimensions. The input of the first adaptive average pooling layer is connected to the output of the first feature extractor, the input of the second adaptive average pooling layer is connected to the output of the second feature extractor, the input of the third adaptive average pooling layer is connected to the output of the third feature extractor, and the input of the fourth adaptive average pooling layer is connected to the output of the fourth feature extractor.

4. The speech emotion recognition method based on attention mechanism multi-scale feature extraction according to claim 3, characterized in that, The multi-scale feature encoder module includes multiple feature encoders, each of which includes a multi-head attention unit, a first residual normalization unit, a feedforward neural network, and a second residual normalization unit. The q-input and k-input of the multi-head attention unit are connected to the output of the corresponding feature extractor, and the v-input of the multi-head attention unit is the original spectrogram. The calculation process of the multi-head attention unit is as follows: MultiHead(Q,K,V)=Concat(head1,...,head h )W o Among them, head i =Attention(QW i Q ,KW i K VW i V ), W i Q W represents the weights of the Q-input of the i-th feature encoder. i K W represents the weights of the K inputs of the i-th feature encoder. i V W represents the weight of the V input of the i-th feature encoder, and h represents the number of heads in the multi-head attention unit; o This represents the overall weight matrix of the multi-head attention unit. Concat(·) means concatenating the data along a specified dimension, and Attention(·) means performing a dot product attention operation. The output of the feature encoder is: x ei =FFN(MultiHead(Q,K i ,V i )+x i )+x i Where, x i This represents the spectrogram of the current input, and FFN represents the feedforward neural network module.

5. The speech emotion recognition method based on multi-scale feature extraction using an attention mechanism according to claim 4, characterized in that, The multi-scale feature encoder module includes a first feature encoder, a second feature encoder, a third feature encoder, and a fourth feature encoder, wherein, The q-input and k-input of the multi-head attention unit of the first feature encoder are connected to the output of the first adaptive average pooling layer; The q-input and k-input of the multi-head attention unit of the second feature encoder are connected to the output of the second adaptive average pooling layer; The q input and k input of the multi-head attention unit of the third feature encoder are connected to the output of the third adaptive average pooling layer. The q-input and k-input of the multi-head attention unit of the fourth feature encoder are connected to the output of the fourth adaptive average pooling layer.

6. The speech emotion recognition method based on multi-scale feature extraction using an attention mechanism according to claim 5, characterized in that, S3 includes: S3.1: Set the parameters of the entire speech emotion recognition network model: initial iteration count K, number of input network samples per iteration, learning rate, and number of cross-validation iterations; S3.2: Use the cross-entropy function as the loss function for the entire network to measure the difference between the true probability distribution and the predicted probability distribution. S3.3: Input the data in the training dataset into the speech emotion recognition network model, calculate the loss function of the data 100 times in one verification process, and verify a total of 10 times; S3.4: Determine whether the loss function has been decreasing throughout the 10 validation cycles: if so, increment K by 1 and return to S3.2; if the loss function stops changing, stop training and save the network model obtained after the current number of validation cycles.

7. The speech emotion recognition method based on multi-scale feature extraction using an attention mechanism according to claim 6, characterized in that, The loss function is: Where m is the number of training samples, n is the number of label categories, and p(x) ij q(x) represents the classification probability that the i-th sample actually belongs to the j-th class. ij ) represents the probability that the i-th sample is classified into the j-th category as predicted by the network.

8. The speech emotion recognition method based on multi-scale feature extraction using an attention mechanism according to claim 5, characterized in that, The steps preceding step S4 also include: The pre-constructed test dataset is input into the trained speech emotion recognition network model to obtain the recognition result, which is then compared with the label of the test dataset to verify the speech emotion recognition accuracy of the speech emotion recognition network model.

Images