A method and device for speech emotion recognition based on phoneme perception

Abstract

The present invention discloses a method and device for speech emotion recognition based on phoneme perception, the method comprising: obtaining original speech, preprocessing the original speech to obtain an audio sequence and a phoneme sequence; obtaining the prediction probability of the phoneme label through the fully connected layer and softmax function of the initial model, and calculating the phoneme prediction loss through the first cross entropy loss function; fusing the encoding feature and the decoding feature through a squeeze excitation module to obtain an enhanced feature; pooling the enhanced feature in the time dimension and splicing it in the feature dimension to form an aggregated feature. The speech emotion recognition method provided by the present invention reduces the identity information in the pre-trained representation, thereby providing more powerful and more discriminative features for the speech emotion recognition task. In order to further utilize the pre-trained representation, a squeeze excitation module is introduced to model the dependency between feature channels and highlight emotion-related information, thereby improving the performance of the speech emotion recognition task.

Description

Technical Field

[0001] The present application relates to the technical field of speech signal processing, and more specifically to a speech emotion recognition method and device based on phoneme perception. Background Art

[0002] Emotion is an important part of human communication. Human emotions are not only conveyed through facial expressions or body language, but also through many features in speech, such as pitch, tone, speaking speed, volume, and rhythm. Speech signals contain rich emotional information, and the emotional state of the speaker can be inferred by analyzing these parameters. Early speech emotion recognition mostly relied on feature engineering-based methods, training models by manually extracting certain features of speech. With the rapid development of machine learning technology, especially deep learning, modern speech emotion recognition technology mostly uses deep neural networks to automatically extract emotional features from speech signals and perform classification predictions.

[0003] Deep neural networks have powerful capabilities in efficiently recognizing speech emotions, but the scale of speech emotion recognition datasets is limited, usually with only a small number of speakers and speech clips. In addition, the reliance on traditional manually extracted acoustic features also limits the improvement of overall performance. In recent years, the research focus of speech emotion recognition has gradually shifted to pre-trained self-supervised models, such as wav2vec 2.0, HuBERT, and WavLM. These models can effectively utilize a large amount of unlabeled speech data, learn rich information, and can be transferred to a variety of downstream tasks. Based on these advances, researchers are actively exploring the potential of pre-trained representations in speech emotion recognition. Although most schemes using pre-trained representations have achieved good results, researchers often ignore the redundant information in pre-trained representations that is irrelevant to emotional cues, which affects the performance of speech emotion recognition. This problem is particularly serious when the scale of the emotion dataset is limited. The model is prone to overfitting to the characteristics of a specific speaker and ignoring the subtle differences in emotions, which in turn weakens the generalization ability and accuracy of the model. This inspires us to use a feature with identity independence to guide the model to learn emotion-related features from the pre-trained representation. Specifically, we introduced a phoneme recognition task to assist speech emotion recognition. The design of this auxiliary task is based on the speaker invariance of phoneme pronunciation. This invariance is reflected in the fact that despite the differences in timbre, pitch, and speaking style between individuals, the basic acoustic pattern of phoneme pronunciation remains consistent. This characteristic of phonemes allows the model to pay more attention to the pronunciation pattern of phonemes rather than identity-related features during training. Summary of the invention

[0004] In this embodiment, a speech emotion recognition method, device, electronic device and storage medium based on phoneme perception are provided to solve the current problems in the field of speech emotion recognition, such as small sample size, few speakers, poor expression ability of manual features, and poor emotion recognition performance.

[0005] In a first aspect, an embodiment of the present invention provides a method for speech emotion recognition based on phoneme perception, the method for speech emotion recognition comprising:

[0006] Acquire original speech, and preprocess the original speech to obtain an audio sequence and a phoneme sequence;

[0007] Using a pre-trained encoder to embed the audio sequence to obtain encoding features, and inputting the encoding features and the phoneme sequence into a pre-trained decoder to obtain decoding features to predict a target phoneme sequence;

[0008] Input the decoded features into the initial model, obtain the predicted probability of the phoneme label through the fully connected layer and softmax function of the initial model, and calculate the phoneme prediction loss through the first cross entropy loss function;

[0009] The encoding feature and the decoding feature are merged through a squeeze excitation module to obtain an enhanced feature;

[0010] The enhanced features are pooled in the time dimension and concatenated in the feature dimension to form aggregated features, the aggregated features are mapped to the probability of the emotion category through another fully connected layer and Softmax function of the initial model, and the loss of emotion recognition is calculated through the second cross entropy loss function;

[0011] A total loss function is obtained according to the first cross entropy loss function and the second cross entropy loss function. The total loss function is used to optimize the initial model and perform iterative training. The initial model after iterative training is used as a speech emotion recognition model, and the speech emotion recognition model is used for emotion recognition.

[0012] In an optional embodiment, obtaining original speech, and preprocessing the original speech to obtain an audio sequence and a phoneme sequence includes:

[0013] The librosa tool is used to sample the original speech to obtain an initial audio sequence. The length of the initial audio sequence is unified to the maximum length of each batch of speech data, and zeros are filled for the missing parts to obtain a dimension of An audio sequence, where is the batch quantity, is the length of the audio sequence;

[0014] The transcription content corresponding to the speech is converted into an initial phoneme sequence through the CMU phoneme dictionary. The length of the initial phoneme sequence is unified to the maximum length of each batch of phoneme sequences. For the missing parts, blank marks are added to obtain the dimension A phoneme sequence, where is the batch quantity, is the length of the phoneme sequence.

[0015] In an optional embodiment, the decoder includes two standard Transformer decoder layers, and the encoder uses a pre-trained wav2vec 2.0 base model.

[0016] In an optional embodiment, the squeeze excitation module includes a global average pooling layer, two fully connected layers and two activation functions, and its calculation formula is as follows:

[0017] ,

[0018] in, , represents the weights of two fully connected layers, represents the ReLU activation function, represents the Sigmoid activation function, represents the original feature map, Represents the importance coefficient.

[0019] In an optional embodiment, the encoding feature and the decoding feature are fused through a squeeze excitation module to obtain an enhanced feature, which includes:

[0020] The original feature map is averaged in the time dimension through the global average pooling layer to obtain the global feature;

[0021] The global features are passed through two fully connected layers and corresponding activation functions to obtain importance coefficients;

[0022] Importance coefficient and original feature map The dot product is used to obtain enhanced features.

[0023] In an optional embodiment, the iterative training of the speech emotion recognition model includes:

[0024] Define key performance indicators for the phoneme recognition task and the emotion recognition task. The key performance indicator for emotion recognition is the weighted accuracy, and the key performance indicator for phoneme recognition is the character error rate.

[0025] The key performance indicators are updated through the exponential scheme to obtain smoothed key performance indicators;

[0026] The task difficulty is calculated based on the smoothed key performance indicators, and the task weight is determined based on the task difficulty.

[0027] In an optional embodiment, calculating the task difficulty according to the smoothed key performance indicators, and determining the task weight according to the task difficulty includes:

[0028] The task difficulty is calculated based on the smoothed key performance indicators, and the formula is as follows:

[0029] ，

[0030] in, represents the task focus parameter, is the task difficulty, is the smoothed key performance indicator;

[0031] The Softmax function is used to normalize the task difficulty and multiply it by the number of tasks to obtain the task weight.

[0032] Compared with the prior art, the speech emotion recognition method based on phoneme perception of the present invention has the following beneficial effects:

[0033] The speech emotion recognition method provided by the present invention reduces the identity information in the pre-trained representation, thereby providing more powerful and discriminative features for the speech emotion recognition task. In order to further utilize the pre-trained representation, a squeeze excitation module is introduced to model the dependencies between feature channels and highlight emotion-related information. In addition, a dynamic task priority strategy is used to cope with the imbalance in the multi-task training process and utilize the synergy between tasks, thereby improving the performance of the speech emotion recognition task.

[0034] In a second aspect, an embodiment of the present invention provides a speech emotion recognition device based on phoneme perception, comprising:

[0035] A preprocessing module, used to obtain original speech, and preprocess the original speech to obtain an audio sequence and a phoneme sequence;

[0036] A coding and decoding architecture module, which uses a pre-trained encoder to embed the audio sequence to obtain coding features, and inputs the coding features and the phoneme sequence into a pre-trained decoder to obtain decoding features to predict a target phoneme sequence;

[0037] A phoneme prediction module, which inputs the decoded features into an initial model, obtains the predicted probability of the phoneme label through the fully connected layer and softmax function of the initial model, and calculates the phoneme prediction loss through a first cross entropy loss function;

[0038] A feature fusion module, which fuses the encoding feature and the decoding feature through a squeeze excitation module to obtain an enhanced feature;

[0039] The emotion recognition module pools the enhanced features in the time dimension and concatenates them in the feature dimension to form aggregated features, maps the aggregated features to the probability of the emotion category through the fully connected layer and Softmax function of the initial model, and calculates the loss of emotion recognition through the second cross entropy loss function;

[0040] The recognition module is trained to obtain a total loss function based on the first cross entropy loss function and the second cross entropy loss function, the initial model is optimized using the total loss function and iterative training is performed, the initial model after iterative training is used as a speech emotion recognition model, and the speech emotion recognition model is used for emotion recognition.

[0041] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a processor, a communication interface, a memory and a bus, wherein the processor, the communication interface and the memory communicate with each other through the bus, and the processor can call logic instructions in the memory to execute the steps of the method provided in the first aspect.

[0042] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the speech emotion recognition method based on phoneme perception as described in the first aspect.

[0043] Compared with the prior art, the beneficial effects of the speech emotion recognition device, electronic device and storage medium based on phoneme perception of the present invention are the same as those of the speech emotion recognition method based on phoneme perception described in the first aspect, so they will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 is a flow chart of a method for speech emotion recognition based on phoneme perception in an embodiment of the present invention;

[0046] Figure 2 is a schematic diagram of a speech emotion recognition method based on phoneme perception in an embodiment of the present invention;

[0047] Figure 3 is a structural block diagram of a speech emotion recognition device based on phoneme perception in an embodiment of the present invention;

[0048] Figure 4 1 is a structural block diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0050] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with general skills in the technical field to which this application belongs. The words "one", "a", "the", "these" and the like in this application do not indicate a quantitative limitation, and they may be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone. Generally, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0051] In an embodiment of the present invention, a method for speech emotion recognition based on phoneme perception is provided. Figure 1 is a flow chart of the speech emotion recognition method based on phoneme perception of the present invention, such as Figure 1 As shown, the process includes the following steps:

[0052] S100, obtaining original speech, and preprocessing the original speech to obtain an audio sequence and phoneme sequences ;

[0053] Combination Figure 2 As shown, in this embodiment, the original speech is obtained, and the original speech is preprocessed to obtain an audio sequence and phoneme sequences include:

[0054] The original speech is sampled using the librosa tool. It should be noted that the sampling rate of the speech is 16kHz. The initial audio sequence is obtained. The length of the initial audio sequence is unified to the maximum length of each batch of speech data, and zeros are filled for the missing parts. The dimension is Audio sequence ,in, is the batch quantity, is the length of the audio sequence;

[0055] The transcription content corresponding to the speech is converted into the initial phoneme sequence through the CMU phoneme dictionary. The length of the initial phoneme sequence is unified to the maximum length of each batch of phoneme sequences. For the missing parts, blank marks are added to obtain the dimension The phoneme sequence ,in, is the batch quantity, is the length of the phoneme sequence;

[0056] Preprocess the original speech to get the audio sequence and phoneme sequences .

[0057] S200, using pre-trained encoder For audio sequence Perform embedding coding to obtain coding features , the encoded features and phoneme sequences Input pre-trained decoder , get the decoding features To predict the target phoneme sequence;

[0058] It should be noted that, in this embodiment, the decoder includes two standard Transformer decoder layers, and the encoder uses a pre-trained wav2vec 2.0 base model;

[0059] Specifically, the pre-built encoder-decoder architecture uses the pre-trained wav2vec 2.0 Base as the encoder. , perform embedding coding to obtain the coding features .Decoder It consists of two standard Transformer decoder layers, encoding features and phoneme sequences As a decoder The input gets the decoded features To predict the target phoneme sequence. This process can be expressed as:

[0060] ,

[0061] ，

[0062] By predicting the target phoneme sequence, the target phoneme sequence refers to the phoneme sequence obtained by transcription through the CMU phoneme dictionary. The purpose of predicting the target phoneme sequence is to keep the basic acoustic pattern of phoneme pronunciation consistent, that is, phoneme pronunciation has speaker-independent characteristics, even though each person's timbre, pitch, and speaking style are different. By predicting the phoneme sequence, the model can pay more attention to identity-independent information, enhance the generalization ability of the model, and avoid overfitting caused by judging emotions through identity information during training.

[0063] S300, input the decoded features into the initial model, obtain the predicted probability of the phoneme label through the fully connected layer and softmax function of the initial model, and calculate the phoneme prediction loss through the first cross entropy loss function;

[0064] Specifically, from the decoder The decoding features obtained from After a fully connected layer and a Softmax function, the predicted probability of the phoneme label is obtained, and the phoneme prediction loss is calculated by the first cross entropy loss function, which is defined as:

[0065]

[0066] in Represents the number of phoneme label categories, It is The first Phonemes are embedded in the label category The true label on (usually 0 or 1), The model is The first Phonemes are embedded in the label category The predicted probability of

[0067] In a sense, the first cross entropy loss function is the loss function of the entire encoder-decoder architecture as part of the model. wav2vec2.0 only represents the encoder, which can be used as part of the overall model structure, and the loss function is used to optimize the parameters of the overall model.

[0068] S400, the coding feature and decode features Enhanced features are obtained by fusion of squeeze excitation modules;

[0069] First of all, it should be noted that the squeeze excitation module includes a global average pooling layer (GAP), two fully connected layers, and two activation functions;

[0070] Encoding features and decode features The squeeze excitation module is used to fuse the information between different channels. The squeeze excitation module consists of a global average pooling layer (GAP), two fully connected layers and two activation functions. Its calculation formula is as follows:

[0071] ,

[0072] in, , represents the weights of two fully connected layers, represents the ReLU activation function, represents the Sigmoid activation function, represents the original feature map, represents the importance coefficient;

[0073] Encoding features and decode features Through the fusion of the extrusion excitation module, the enhanced features include:

[0074] The original feature map The global features are averaged in the time dimension through the global average pooling layer (GAP);

[0075] The global features pass through two fully connected layers and corresponding activation functions to obtain the importance coefficient ;

[0076] Importance coefficient and the original feature map The dot product is used to obtain enhanced features, and the formula is as follows:

[0077]

[0078] in represents the enhanced features, The symbol uses the broadcast mechanism to The shape expands to Same, then multiplication is performed.

[0079] S500, pooling the enhanced features in the time dimension and concatenating them in the feature dimension to form aggregated features, mapping the aggregated features to the probability of the emotion category through another fully connected layer and the Softmax function of the initial model, and calculating the loss of emotion recognition through the second cross entropy loss function;

[0080] Through the above operations, the encoding features and decode features The enhanced features are obtained by the squeeze excitation module. On this basis, the enhanced features are pooled in the time dimension, and the pooled features are spliced ​​in the feature dimension to aggregate the two features. Similar to the above steps, the aggregated features are mapped to the probability of the emotion category through the fully connected layer and the Softmax function, and the loss of emotion recognition is calculated through the second cross entropy loss function, which is defined as follows:

[0081]

[0082] in Represents the number of sentiment categories, It is Samples in emotional label The true label on (usually 0 or 1), The model is In the sample, the emotional label The predicted probability of .

[0083] S600, obtaining a total loss function according to the first cross entropy loss function and the second cross entropy loss function, optimizing the initial model using the total loss function and performing iterative training, using the initial model after iterative training as a speech emotion recognition model, and performing emotion recognition using the speech emotion recognition model.

[0084] The overall loss function can be expressed as:

[0085] ,

[0086] in, Used to control task weight.

[0087] Furthermore, the task weights of phoneme recognition and emotion recognition are fixed, which may lead to imbalance in task training. To avoid this, the present embodiment further adopts a dynamic task priority training strategy, which assigns weights to tasks according to their training progress.

[0088] Specifically, the iterative training of the speech emotion recognition model includes:

[0089] Define key performance indicators for phoneme recognition and emotion recognition tasks , Represents the index of the task. In this embodiment, the key performance indicator of emotion recognition is the weighted accuracy, and the key performance indicator of phoneme recognition is the character error rate (1-CER);

[0090] The key performance indicators are updated through the exponential scheme to obtain smoothed key performance indicators;

[0091] The task difficulty is calculated based on the smoothed key performance indicators, and the task weight is determined based on the task difficulty.

[0092] More specifically, the key performance indicators are updated through the exponential average scheme, and the process is as follows:

[0093] ,

[0094] in, represents the number of training iterations, The current smoothed key performance indicator is composed of the current key performance indicator and the key performance indicators after smoothing in the previous step Calculated. is a hyperparameter used to balance the current and historical indicators. Calculating task difficulty , thereby determining the weight of each task.

[0095] Furthermore, the task difficulty is calculated based on the smoothed key performance indicators, and the task weight is determined based on the task difficulty, including:

[0096] The task difficulty is calculated based on the smoothed key performance indicators, and the formula is as follows:

[0097] ，

[0098] in, represents the task focus parameter, is the task difficulty, is the smoothed key performance indicator;

[0099] The Softmax function is used to normalize the task difficulty and multiply it by the number of tasks to obtain the task weight.

[0100] The public dataset IEMOCAP dataset was used for five-fold cross validation. The IEMOCAP dataset contains five sessions, each session contains two speakers. We used data from four sessions for training. In the remaining session, data from one speaker was used as a validation set, and data from the other speaker was used as a test set. Then the validation set and the test set were exchanged for evaluation, and the results of each fold were averaged over two evaluations. We froze the parameters of the convolutional layer in wav2vec2.0 Base during training, retained the trainability of the Transformer layer, and trained according to the above method. Finally, the unweighted accuracy and weighted accuracy on the test set of the IEMOCAP dataset were statistically calculated. The statistical results are shown in Table 1. The higher accuracy indicates that the proposed speech emotion recognition method based on phoneme perception is effective in identifying emotions and can better distinguish different emotion categories; and five-fold cross validation is used, which means that the dataset is divided into five subsets, each subset is used as a test set in turn, and the rest are used as training sets. This method can better evaluate the performance of the model on unseen data, reflecting the generalization ability of the model.

[0101] Table 1

[0102]

[0103] The speech emotion recognition method provided in the embodiment of the present invention reduces the identity information in the pre-trained representation, thereby providing more powerful and discriminative features for the speech emotion recognition task. In order to further utilize the pre-trained representation, a squeeze excitation module is introduced to model the dependencies between feature channels and highlight emotion-related information. In addition, a dynamic task priority strategy is customized to cope with the imbalance in the multi-task training process and utilize the synergy between tasks, thereby improving the performance of the speech emotion recognition task.

[0104] The embodiment of the present invention also provides a speech emotion recognition device based on phoneme perception, which is used to implement the above method embodiment, and will not be repeated hereafter. The terms "module", "unit", "subunit", etc. used below can implement a combination of software and / or hardware of predetermined functions. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware or a combination of software and hardware is also possible and conceivable.

[0105] like Figure 3 As shown, Figure 3 : is a structural block diagram of a speech emotion recognition device based on phoneme perception in the present invention, the device comprises:

[0106] A preprocessing module 101 is used to obtain original speech and preprocess the original speech to obtain an audio sequence and a phoneme sequence;

[0107] The encoding and decoding architecture module 102 uses a pre-trained encoder to embed the audio sequence to obtain encoding features, and inputs the encoding features and the phoneme sequence into a pre-trained decoder to obtain decoding features to predict a target phoneme sequence;

[0108] The phoneme prediction module 103 inputs the decoded features into an initial model, obtains the predicted probability of the phoneme label through the fully connected layer and the softmax function of the initial model, and calculates the phoneme prediction loss through the first cross entropy loss function;

[0109] A feature fusion module 104, which fuses the encoding feature and the decoding feature through a squeeze excitation module to obtain an enhanced feature;

[0110] The emotion recognition module 105 pools the enhanced features in the time dimension and concatenates them in the feature dimension to form aggregated features, maps the aggregated features to the probability of the emotion category through the fully connected layer and Softmax function of the initial model, and calculates the loss of emotion recognition through the second cross entropy loss function;

[0111] The training recognition module 106 obtains a total loss function according to the first cross entropy loss function and the second cross entropy loss function, optimizes the initial model using the total loss function and performs iterative training, uses the initial model after iterative training as a speech emotion recognition model, and uses the speech emotion recognition model to perform emotion recognition.

[0112] The speech emotion recognition device based on phoneme perception provided in the embodiment of the present invention reduces the identity information in the pre-trained representation, thereby providing more powerful and discriminative features for the speech emotion recognition task. In order to further utilize the pre-trained representation, a squeeze excitation module is introduced to model the dependency between feature channels and highlight emotion-related information, thereby improving the emotion recognition performance.

[0113] Figure 4 A structural block diagram of an electronic device provided by an embodiment of the present invention, such as Figure 4 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the following method:

[0114] Obtaining original speech, and preprocessing the original speech to obtain an audio sequence and a phoneme sequence;

[0115] The audio sequence is embedded and encoded using a pre-trained encoder to obtain encoding features, and the encoding features and the phoneme sequence are input into a pre-trained decoder to obtain decoding features to predict the target phoneme sequence;

[0116] The decoded features are input into the initial model, and the predicted probability of the phoneme label is obtained through the fully connected layer and softmax function of the initial model. The phoneme prediction loss is calculated through the first cross entropy loss function;

[0117] The encoding features and decoding features are fused through the squeeze excitation module to obtain enhanced features;

[0118] The enhanced features are pooled in the time dimension and concatenated in the feature dimension to form aggregated features. The aggregated features are mapped to the probability of the emotion category through another fully connected layer and Softmax function of the initial model, and the loss of emotion recognition is calculated through the second cross entropy loss function.

[0119] A total loss function is obtained according to the first cross entropy loss function and the second cross entropy loss function. The total loss function is used to optimize the initial model and perform iterative training. The initial model after iterative training is used as a speech emotion recognition model, and the speech emotion recognition model is used for emotion recognition.

[0120] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0121] An embodiment of the present invention further provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method provided in each of the above embodiments is implemented, for example, including:

[0122] Obtaining original speech, and preprocessing the original speech to obtain an audio sequence and a phoneme sequence;

[0123] The audio sequence is embedded and encoded using a pre-trained encoder to obtain encoding features, and the encoding features and the phoneme sequence are input into a pre-trained decoder to obtain decoding features to predict the target phoneme sequence;

[0124] The decoded features are input into the initial model, and the predicted probability of the phoneme label is obtained through the fully connected layer and softmax function of the initial model. The phoneme prediction loss is calculated through the first cross entropy loss function;

[0125] The encoding features and decoding features are fused through the squeeze excitation module to obtain enhanced features;

[0126] The enhanced features are pooled in the time dimension and concatenated in the feature dimension to form aggregated features. The aggregated features are mapped to the probability of the emotion category through another fully connected layer and Softmax function of the initial model, and the loss of emotion recognition is calculated through the second cross entropy loss function.

[0127] A total loss function is obtained according to the first cross entropy loss function and the second cross entropy loss function. The total loss function is used to optimize the initial model and perform iterative training. The initial model after iterative training is used as a speech emotion recognition model, and the speech emotion recognition model is used for emotion recognition.

[0128] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0129] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech emotion recognition method based on phoneme perception, characterized in that: The speech emotion recognition method comprises: Acquire original speech, and preprocess the original speech to obtain an audio sequence and a phoneme sequence; Using a pre-trained encoder to embed the audio sequence to obtain encoding features, and inputting the encoding features and the phoneme sequence into a pre-trained decoder to obtain decoding features to predict a target phoneme sequence; Input the decoded features into the initial model, obtain the predicted probability of the phoneme label through the fully connected layer and softmax function of the initial model, and calculate the phoneme prediction loss through the first cross entropy loss function; The encoding feature and the decoding feature are merged through a squeeze excitation module to obtain an enhanced feature; The enhanced features are pooled in the time dimension and concatenated in the feature dimension to form aggregated features, the aggregated features are mapped to the probability of the emotion category through another fully connected layer and Softmax function of the initial model, and the loss of emotion recognition is calculated through the second cross entropy loss function; A total loss function is obtained according to the first cross entropy loss function and the second cross entropy loss function. The total loss function is used to optimize the initial model and perform iterative training. The initial model after iterative training is used as a speech emotion recognition model, and the speech emotion recognition model is used for emotion recognition.

2. The speech emotion recognition method based on phoneme perception according to claim 1, characterized in that: Acquiring original speech and preprocessing the original speech to obtain an audio sequence and a phoneme sequence includes: The librosa tool is used to sample the original speech to obtain an initial audio sequence. The length of the initial audio sequence is unified to the maximum length of each batch of speech data, and zeros are filled for the missing parts to obtain a dimension of An audio sequence, where is the batch quantity, is the length of the audio sequence; The transcription content corresponding to the speech is converted into an initial phoneme sequence through the CMU phoneme dictionary. The length of the initial phoneme sequence is unified to the maximum length of each batch of phoneme sequences. For the missing parts, blank marks are added to obtain the dimension A phoneme sequence, where is the batch quantity, is the length of the phoneme sequence.

3. The speech emotion recognition method based on phoneme perception according to claim 1, characterized in that: The decoder includes two standard Transformer decoder layers, and the encoder uses a pre-trained wav2vec 2.0 base model.

4. The speech emotion recognition method based on phoneme perception according to claim 1, characterized in that: The squeeze excitation module includes a global average pooling layer, two fully connected layers and two activation functions, and its calculation formula is as follows: ， in, , represents the weights of two fully connected layers, represents the ReLU activation function, represents the Sigmoid activation function, represents the original feature map, Represents the importance coefficient.

5. The speech emotion recognition method based on phoneme perception according to claim 4 is characterized in that: The encoding feature and the decoding feature are fused through a squeeze excitation module to obtain enhanced features including: The original feature map is averaged in the time dimension through the global average pooling layer to obtain the global feature; The global features are passed through two fully connected layers and corresponding activation functions to obtain importance coefficients; Importance coefficient and original feature map The dot product is used to obtain enhanced features.

6. The speech emotion recognition method based on phoneme perception according to claim 1, characterized in that: The iterative training of the speech emotion recognition model includes: Define key performance indicators for the phoneme recognition task and the emotion recognition task. The key performance indicator for emotion recognition is the weighted accuracy, and the key performance indicator for phoneme recognition is the character error rate. The key performance indicators are updated through the exponential scheme to obtain smoothed key performance indicators; The task difficulty is calculated based on the smoothed key performance indicators, and the task weight is determined based on the task difficulty.

7. The method for speech emotion recognition based on phoneme perception according to claim 6, characterized in that: The task difficulty is calculated based on the smoothed key performance indicators, and the task weight is determined based on the task difficulty, including: The task difficulty is calculated based on the smoothed key performance indicators, and the formula is as follows: ， in, represents the task focus parameter, is the task difficulty, is the smoothed key performance indicator; The Softmax function is used to normalize the task difficulty and multiply it by the number of tasks to obtain the task weight.

8. A speech emotion recognition device based on phoneme perception, characterized in that: include: A preprocessing module, used to obtain original speech, and preprocess the original speech to obtain an audio sequence and a phoneme sequence; A coding and decoding architecture module, which uses a pre-trained encoder to embed the audio sequence to obtain coding features, and inputs the coding features and the phoneme sequence into a pre-trained decoder to obtain decoding features to predict a target phoneme sequence; A phoneme prediction module, which inputs the decoded features into an initial model, obtains the predicted probability of the phoneme label through the fully connected layer and softmax function of the initial model, and calculates the phoneme prediction loss through a first cross entropy loss function; A feature fusion module, which fuses the encoding feature and the decoding feature through a squeeze excitation module to obtain an enhanced feature; The emotion recognition module pools the enhanced features in the time dimension and concatenates them in the feature dimension to form aggregated features, maps the aggregated features to the probability of the emotion category through the fully connected layer and Softmax function of the initial model, and calculates the loss of emotion recognition through the second cross entropy loss function; The recognition module is trained to obtain a total loss function based on the first cross entropy loss function and the second cross entropy loss function, the initial model is optimized using the total loss function and iterative training is performed, the initial model after iterative training is used as a speech emotion recognition model, and the speech emotion recognition model is used for emotion recognition.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the speech emotion recognition method based on phoneme perception as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the speech emotion recognition method based on phoneme perception as described in any one of claims 1 to 7 are implemented.

Images