Multitask constraint end-to-end noisy speech emotion recognition method based on teacher-student model

Abstract

The invention discloses a multi-task constraint end-to-end noisy speech emotion recognition method based on a teacher-student model, and the method comprises the steps: proposing an end-to-end multi-task constraint network based on the teacher-student model, completing the multi-task training of the teacher model through reference to the thought of knowledge distillation, and obtaining a multi-task training result; pure and remarkable emotion features are extracted by adopting a maturely trained teacher model to guide the training process of the student model; a multi-level enhanced loss mechanism is introduced, so that a student model can learn key emotion information extracted from clean voice by a teacher model in a middle layer; the thought of a voice enhancement auxiliary task is adopted, and the voice enhancement auxiliary task is combined with multi-level enhancement loss; in the test stage, only a core process specific to speech emotion recognition and classification in the student model needs to be concerned, and the execution rate of the test is effectively improved. The voice emotion recognition task in the noise environment is achieved, the emotion classification accuracy is improved, and the method is suitable for intelligent customer service and mental health monitoring.

Description

Technical Field

[0001] The present invention belongs to the technical field of speech emotion recognition, and in particular relates to a multi-task constrained end-to-end noisy speech emotion recognition method based on a teacher-student model. Background Art

[0002] At present, speech emotion recognition systems have demonstrated excellent performance under ideal environmental conditions. However, when these systems are applied to the real world, environmental noise interferes with the effective information in the speech signal, making it extremely difficult to detect significant speech emotion representations. Researchers have launched targeted attempts from multiple aspects, including preprocessing operations to improve or eliminate noise in the original speech signal, selecting features that are more robust to noise, and building more robust and stable models.

[0003] Preprocessing algorithms improve speech signal quality by reducing or eliminating the effects of noise. Existing methods include spectral subtraction, audio enhancement methods based on optimal modified logarithmic spectrum amplitude estimator, and methods using filters for preprocessing, etc. Speech enhancement based on deep learning is also gradually showing advantages in recognizing emotions in noisy speech. For example, a study used a long short-term memory (LSTM) architecture to build an enhancement model, combined with a post-processing method that estimates the LSTM structure mask with an improved speech presence probability, thereby improving recognition accuracy. Chakraborty et al. studied a feature compensation technique based on the vector Taylor series expansion of the noise Mel-frequency cepstral coefficients, and improved the technique through the vector Taylor series expansion with the auditory masking formula, showing significant performance advantages.

[0004] Feature selection aims to determine the most relevant features, extract features from a wider set of features and reduce data dimensionality to reduce overfitting and increase interpretability. In the study of speech emotion recognition in noisy environments, researchers have explored and proposed a variety of feature extraction strategies, including noise adaptation, speaker adaptation, speaker and noise adaptation, and noise-specific feature selection. These methods are usually combined with correlation-based feature subset selection and sequential floating forward search to improve the performance of the model in noisy environments. Bandela et al. proposed a feature selection method based on the performance of low-level descriptors, using a convolutional neural network (CNN) structure to test 65 low-level descriptors of the ComparE2013 feature set to evaluate the noise robustness of different feature selection methods. Experimental results show that this method improves speech emotion recognition performance by 24.4% (arousal), 23.9% (dominance) and 43.2% (valence) in 10dB noise.

[0005] In the case of noise interference, building a more robust and robust model is one of the key methods to improve the overall model recognition rate. Triantafyllopoulos et al. studied training scalable deep learning architectures to enhance audio signals in low signal-to-noise ratio environments. Zhu Zhou et al. proposed a virtual amplification method and a robust multi-scenario speech emotion recognition system. They synthesized a new database by modifying the environmental conditions of the audio and simulated experiments under different levels of Gaussian white noise, real-world noise, and reverberation. Tan et al. used semi-supervised learning techniques to identify musical emotions and conducted student training experiments in noisy environments, contributing more inspiration and thinking. Guimaraes et al. used noise and reverberation to enhance training data, and combined curriculum learning with multi-task learning methods to build a speech emotion recognition model based on HuBERT. Experimental results show that the model shows certain advantages in "wild" speech applications. Jaiswal et al. studied the impact of noise on human and machine perception of emotions, and explored the integration of denoising feature space and speech enhancement algorithms with emotion recognition models. Summary of the invention

[0006] The purpose of the present invention is to address the defects and shortcomings of the above-mentioned prior art, and propose a multi-task constrained end-to-end noisy speech emotion recognition method based on a teacher-student model. The teacher model is used to extract significant emotional features through multi-task training, and knowledge distillation is used to guide the training of the student model to improve accuracy and robustness; a multi-level enhancement loss mechanism is introduced to enable the student model to learn the key emotional information of the teacher model in the middle layer, so that the noisy speech features are close to the clean speech features, and the model accuracy in a noisy environment is improved by combining the hyperparameter loss function; speech enhancement auxiliary tasks are combined with multi-level enhancement losses to construct a composite multi-level speech enhancement auxiliary structure to enhance robustness in a noisy environment; in the test phase, only the core process of the student model is focused on, without relying on the teacher model or auxiliary tasks, thereby reducing the test complexity and improving the test rate.

[0007] The technical solution adopted by the present invention to solve the technical problem is: to provide a multi-task constrained end-to-end noisy speech emotion recognition method based on a teacher-student model, the method comprising the following steps:

[0008] Step 1: Input the clean speech signal into the teacher model, which is composed of a CNN feature encoder consisting of 7 one-dimensional convolutional layers and a Wav2vec 2.0 model consisting of 12 Transformer layers. Then, through the fully connected layer, the letter vector and emotion soft label are obtained, and the teacher model network is fine-tuned under the joint constraints of speech recognition and emotion recognition.

[0009] Step 2: Input the noisy speech signal into the student model. The student model emotion prediction task is composed of a CNN feature encoder consisting of 7 one-dimensional convolutional layers and a Wav2vec 2.0 model consisting of 12 Transformer layers, and then pass through the fully connected layer to obtain the predicted label;

[0010] Step 3: Input the noisy speech signal into the student model. The student model speech enhancement task consists of a CNN feature encoder consisting of 7 one-dimensional convolutional layers and a speech enhancement structure consisting of 12 Transformer layers and 7 one-dimensional transposed convolutional layers, and outputs the predicted speech;

[0011] Step 4: Based on the proposed multiple joint loss functions, a composite constraint mechanism is formed to guide and regulate the training process of the student model and improve the accuracy of the model in a noisy environment;

[0012] Step 5: Perform a performance evaluation on the proposed multi-task constrained end-to-end noisy speech emotion recognition method based on the teacher-student model.

[0013] Furthermore, the specific steps of step 1 are:

[0014] Step 1-1: The input noisy speech signal audio waveform X is passed through 7 one-dimensional convolutional layers to obtain the downsampled audio feature Z;

[0015] Step 1-2: The downsampled audio feature z output by the feature encoder is passed through 12 Transformer layers to obtain the context representation c;

[0016] Step 1-3: The context representation c is passed through the fully connected layer to obtain the letter vector and sentiment soft label y i t , where the letter vectors are no longer involved in subsequent tasks.

[0017] Furthermore, the specific steps of step 2 are:

[0018] Step 2-1: The input clean speech signal audio waveform x' is passed through 7 one-dimensional convolutional layers to obtain the downsampled audio feature z';

[0019] Step 2-2: The downsampled audio feature z' output by the feature encoder is passed through 12 Transformer layers to obtain the context representation c';

[0020] Step 2-3: The context representation c' is passed through the fully connected layer to obtain the sentiment label Participate in subsequent tasks.

[0021] Furthermore, the specific steps of step 3 are:

[0022] Step 3-1: The input clean speech signal audio waveform x' is passed through 7 one-dimensional convolutional layers to obtain the downsampled audio feature z';

[0023] Step 3-2: The downsampled audio feature z' output by the feature encoder is passed through 12 Transformer layers to obtain the context representation c';

[0024] Step 3-3: The context representation c' is passed through the speech reconstruction module to obtain the predicted speech for subsequent tasks.

[0025] Furthermore, in step 3-3, the specific method for implementing the speech reconstruction module includes the following steps:

[0026] Step 3-3-1: Construct a speech reconstruction module, which consists of 7 transposed convolutional neural networks that match the dimensions of the CNN feature extractor;

[0027] Step 3-3-2: Use the residual connection technology to connect the odd-layer network output of the CNN feature extractor with the corresponding layer output of the speech reconstruction module, which effectively promotes the flow of information within the network.

[0028] Furthermore, the specific steps of step 4 are:

[0029] Step 4-1: The calculation formula of distillation loss KL is expressed as:

[0030]

[0031] in, is the soft label of the i-th sample obtained by the teacher model, is the soft prediction of the i-th sample obtained by the student model, T is the distillation temperature coefficient, is a sentiment soft label, For sentiment labels, we get the distillation loss;

[0032] Step 4-2: Construct the sentiment label loss function as a multi-classification cross entropy loss function:

[0033]

[0034] Among them, M represents the number of samples, N represents the number of emotion categories, and p(x ij ) represents the predicted probability that the observed sample i belongs to category j, and the sentiment label loss is obtained;

[0035] Step 4-3: Construct a multi-level enhancement loss function:

[0036]

[0037] Where M is defined as the number of samples, is the output of the i-th layer of the teacher model, represents the feature vector of the output of the i-th layer of the student network model, that is, the predicted value of the i-th layer of the student model, and i represents the number of layers of the Transformer network of the teacher model and the student model;

[0038] The losses between the even-numbered layers of the teacher model and the student model Transformer network are calculated separately, for a total of 6 layers, and the formula is as follows:

[0039]

[0040] Get multi-level enhancement loss;

[0041] Step 4-4: Construct the scale-invariant signal-to-noise ratio loss function (SI-SNR):

[0042]

[0043] Among them, s targct represents the target signal (i.e., the original clean speech signal), e noisc represents the noise signal (i.e., the difference between the enhanced signal and the target signal), ||·|| 2 represents the L2 norm, which is a measure of signal energy, and obtains the speech enhancement loss;

[0044] Step 4-5: Combine the distillation loss, sentiment label loss, multi-level enhancement loss and speech enhancement loss to constrain the training of the student model:

[0045] L ALL =αL KL +(1-α)L CE +βL S +γL SI-SNR ,

[0046] Among them, α is the weight of distillation loss, β is the weight of multi-level enhancement loss, and γ is the weight of speech enhancement loss.

[0047] Furthermore, in step 5, the specific steps of performing performance evaluation on the proposed multi-task constrained end-to-end noisy speech emotion recognition method based on the teacher-student model are as follows:

[0048] Step 5-1: Conduct ablation experiments on the multi-task constrained end-to-end noisy speech emotion recognition method based on the teacher-student model and the method of removing the teacher model to verify the effectiveness of the teacher-student model;

[0049] Step 5-2: Conduct ablation experiments on the multi-task constrained end-to-end noisy speech emotion recognition method based on the teacher-student model and the method of removing multi-level enhanced connections to verify the effectiveness of the teacher-student model;

[0050] Step 5-3: Conduct ablation experiments on the multi-task constrained end-to-end noisy speech emotion recognition method based on the teacher-student model and the method of removing speech enhancement to verify the effectiveness of the speech enhancement module;

[0051] Step 5-4: Discuss the specific value of the weight loss in the joint constraint and explore the optimal weight value.

[0052] Beneficial effects:

[0053] 1. The present invention uses the teacher-student model and draws on the idea of ​​knowledge distillation. Through multi-task training of the teacher model, pure and significant emotional features are extracted to guide the training process of the student model, thereby improving the accuracy and robustness of the model and enhancing the generalization ability.

[0054] 2. The present invention introduces a multi-level enhanced loss mechanism, which allows the student model to learn the key emotional information extracted by the teacher model from the clean speech in the middle layer, and makes the features learned by the student model from the noisy speech close to the features of the clean speech. The present invention combines multiple loss functions through multiple hyperparameters to achieve the effect of improving the model accuracy in a noisy environment.

[0055] 3. The present invention adopts the idea of ​​speech enhancement auxiliary task, combines the speech enhancement auxiliary task with multi-level enhancement loss, realizes a composite multi-level speech enhancement auxiliary structure, and improves the robustness of speech emotion recognition task in noisy environment.

[0056] 4. During the testing phase, the present invention only needs to focus on the core process in the student model that is specific to speech emotion recognition and classification, without relying on the auxiliary task branches designed in the teacher model or the student model, which significantly reduces the complexity of the model that needs to be processed during the testing process, thereby effectively improving the test execution rate without sacrificing accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a schematic diagram of the operation flow of the present invention.

[0058] Figure 2 This is a speech enhancement framework diagram in the student model of the present invention.

[0059] Figure 3 It is a multi-level enhancement loss framework diagram of the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings of the specification. It should be noted that, in order to avoid blurring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solution of the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0061] In addition, it should be noted that the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or apparatus.

[0062] like Figures 1 to 3 As shown, the present invention proposes a multi-task constrained end-to-end noisy speech emotion recognition method based on a teacher-student model, which applies the teacher-student model and the speech enhancement module to speech emotion recognition to extract deep features, has a certain robustness to noise, and can build a higher performance speech emotion recognition system, which specifically includes the following steps:

[0063] Step 1: Input the clean speech signal into the teacher model, which is composed of a CNN feature encoder consisting of 7 one-dimensional convolutional layers and a Wav2vec 2.0 model consisting of 12 Transformer layers. Then, through the fully connected layer, the letter vector and emotion soft label are obtained, and the teacher model network is fine-tuned under the joint constraints of speech recognition and emotion recognition.

[0064] Step 2: Input the noisy speech signal into the student model. The student model emotion prediction task is composed of a CNN feature encoder consisting of 7 one-dimensional convolutional layers and a Wav2vec 2.0 model consisting of 12 Transformer layers, and then pass through the fully connected layer to obtain the predicted label;

[0065] Step 3: Input the noisy speech signal into the student model. The student model speech enhancement task consists of a CNN feature encoder consisting of 7 one-dimensional convolutional layers and a speech enhancement structure consisting of 12 Transformer layers and 7 one-dimensional transposed convolutional layers, and outputs the predicted speech;

[0066] Step 4: Based on the proposed multiple joint loss functions, a composite constraint mechanism is formed to guide and regulate the training process of the student model and improve the accuracy of the model in a noisy environment;

[0067] Step 5: Perform a performance evaluation on the proposed multi-task constrained end-to-end noisy speech emotion recognition method based on the teacher-student model.

[0068] This experiment uses IEMOCAP, a multimodal database for the SER field, which contains 12 hours of recorded data divided into improvisation and script parts. The corpus contains nine emotions: anger, disgust, fear, frustration, sadness, happiness, excitement, surprise, and neutral. Due to the imbalance of the dataset, researchers usually choose the four most common emotions. Because excitement and happiness are similar and there are few happy samples, researchers sometimes replace happiness with excitement or combine the two to increase the amount of data. This experiment uses four emotional states (neutral, excitement, sadness, and anger). The noise in the experiment comes from the MUSAN dataset, which includes 11 real-world environmental noises. To simulate the noisy scene, this experiment constructs a noisy speech dataset by randomly mixing clean audio from IEMOCAP and noise from the MUSAN dataset.

[0069] The specific steps of step 1 include:

[0070] Step 1-1: The input noisy speech signal audio waveform X is passed through 7 one-dimensional convolutional layers to obtain the downsampled audio feature Z;

[0071] Step 1-2: The downsampled audio feature z output by the feature encoder is passed through 12 Transformer layers to obtain the context representation c;

[0072] Step 1-3: The context representation c is passed through the fully connected layer to obtain the letter vector and sentiment soft label The letter vectors are no longer involved in subsequent tasks.

[0073] The specific steps of step 2 include:

[0074] Step 2-1: The input clean speech signal audio waveform x' is passed through 7 one-dimensional convolutional layers to obtain the downsampled audio feature Z';

[0075] Step 2-2: The downsampled audio feature z' output by the feature encoder is passed through 12 Transformer layers to obtain the context representation c';

[0076] Step 2-3: The context representation c' is passed through the fully connected layer to obtain the sentiment label Participate in subsequent tasks.

[0077] The specific steps of step 3 include:

[0078] Step 3-1: The input clean speech signal audio waveform x' is passed through 7 one-dimensional convolutional layers to obtain the downsampled audio feature z';

[0079] Step 3-2: The downsampled audio feature z' output by the feature encoder is passed through 12 Transformer layers to obtain the context representation c';

[0080] Step 3-3: The context representation c' is passed through the speech reconstruction module to obtain the predicted speech for subsequent tasks.

[0081] In the above step 3-3 of the present invention, the specific method of implementing the speech reconstruction module comprises the following steps:

[0082] Step 3-3-1: Construct a speech reconstruction module, which consists of 7 transposed convolutional neural networks that match the dimensions of the CNN feature extractor;

[0083] Among them, the 7-layer TransConv parameters are shown in Table 1:

[0084] Table 1 Core parameters of speech reconstruction module

[0085] layer Convolution Kernel Input Channels Output Channel Step Length TransConv7 2 1024 512 2 TransConv6 2 512 512 2 TransConv5 3 1024 512 2 TransConv4 3 512 512 2 TransConv3 3 1024 512 2 TransConv2 3 512 512 2 TransConv1 10 1024 1 5

[0086] Step 3-3-2: Use the residual connection technology to connect the odd-layer network output of the CNN feature extractor with the corresponding layer output of the speech reconstruction module, which effectively promotes the flow of information within the network.

[0087] The specific steps of step 4 include:

[0088] Step 4-1: The calculation formula of distillation loss KL is expressed as:

[0089]

[0090] in, is the soft label of the i-th sample obtained by the teacher model, is the soft prediction of the i-th sample obtained by the student model, T is the distillation temperature coefficient, is a sentiment soft label, For sentiment labels, we get the distillation loss;

[0091] Step 4-2: Construct the sentiment label loss function as a multi-classification cross entropy loss function:

[0092]

[0093] Among them, M represents the number of samples, N represents the number of emotion categories, and p(x ij ) represents the predicted probability that the observed sample i belongs to category j, and the sentiment label loss is obtained;

[0094] Step 4-3: Construct a multi-level enhancement loss function:

[0095]

[0096] Where M is defined as the number of samples, is the output of the i-th layer of the teacher model, represents the feature vector of the output of the i-th layer of the student network model, that is, the predicted value of the i-th layer of the student model, and i represents the number of layers of the Transformer network of the teacher model and the student model;

[0097] The losses between the even-numbered layers of the teacher model and the student model Transformer network are calculated separately, for a total of 6 layers, and the formula is as follows:

[0098]

[0099] Get multi-level enhancement loss;

[0100] Step 4-4: Construct the scale-invariant signal-to-noise ratio loss function (SI-SNR):

[0101]

[0102] Among them, s targct represents the target signal (i.e., the original clean speech signal), e noisc represents the noise signal (i.e., the difference between the enhanced signal and the target signal), ||·|| 2 represents the L2 norm, which is a measure of signal energy; the speech enhancement loss is obtained;

[0103] Step 4-5: Combine the distillation loss, sentiment label loss, multi-level enhancement loss and speech enhancement loss to constrain the training of the student model:

[0104] L ALL =αL KL +(1-α)L CE +βL S +γL SI-SNR ,

[0105] Among them, α is the weight of distillation loss, β is the weight of multi-level enhancement loss, and γ is the weight of speech enhancement loss.

[0106] In the above step 5 of the present invention, a specific method for evaluating the performance of the proposed multi-task constrained end-to-end noisy speech emotion recognition method based on the teacher-student model includes the following steps:

[0107] Step 5-1: Conduct ablation experiments on a multi-task constrained end-to-end noisy speech emotion recognition method based on a teacher-student model and a method of removing the teacher model to verify the effectiveness of the teacher-student model;

[0108] Step 5-2: Conduct ablation experiments on a multi-task constrained end-to-end noisy speech emotion recognition method based on the teacher-student model and a method of removing multi-level enhanced connections to verify the effectiveness of the teacher-student model;

[0109] Step 5-3: Conduct ablation experiments on a multi-task constrained end-to-end noisy speech emotion recognition method based on a teacher-student model and a method for removing speech enhancement to verify the effectiveness of the speech enhancement module;

[0110] Step 5-4: Discuss the specific value of the weight loss in the joint constraint and explore the optimal weight value.

[0111] The simulation experiment and data of the present invention are as follows:

[0112] In this embodiment, the evaluation indicators used are WA and UA commonly used in the SER field. A sampling rate of 16khz is used for all signals. In the experiment, the noisy speech data set is randomly divided into 80% training data and 20% test data, and 5-fold cross validation is performed. In the process of pursuing model optimization, the entire training process is 200 epochs. The loss function used in the proposed MNTM training phase is the joint loss, in which, after the loss function hyperparameter optimization, α, β and γ are set to 0.7, 3 and -0.1 respectively. T is set to 5. The AdamW optimizer is used in the experiment. The batch size of the model is set to 2. This experiment is built based on the Pytorch framework, and the experimental operating environment is performed on the NVIDIA Tesla P100 SXM2 machine. All experiments were performed using CUDA version 10.2.

[0113] Ablation experiments were conducted on the verification of the teacher model, multi-level connection, and speech enhancement. The results are shown in Table 2:

[0114] Table 2 Ablation experiments of MNTM on IEM+MUS database

[0115]

[0116] Under the same signal-to-noise ratio environment, by comparing and analyzing the confusion matrices of the MNTM model and the MNTM-TM model, it can be clearly found that the introduction of the supervision mechanism of the teacher model significantly improves the MNTM model's resistance to noise interference and maintains a high classification accuracy. In addition, this chapter also conducts an in-depth analysis of the misclassification in the confusion matrix. By comparing the misclassification ratios between different emotion categories, it can be found that some emotion categories (such as nature) are more easily confused in a noisy environment. Therefore, when designing an emotional speech recognition system, special attention should be paid to these easily confused emotion categories, and corresponding strategies should be adopted to improve the recognition ability of the model.

[0117] By comparing the performance of the MNTM model and the MNTM-MECA model under various signal-to-noise ratio conditions, it can be found that when the multi-level enhanced connection mechanism is removed from the overall model framework, the overall performance of the model shows a downward trend. This phenomenon shows that the multi-level connection strategy at the intermediate level can provide more effective learning support for the student model, especially in assisting it to learn the intermediate features extracted by the teacher model, which is conducive to the student model to better capture the significant emotional information extracted by the teacher model from the clean speech in a noisy environment. In addition, the performance comparison between the MNTM-TM model and the MNTM-MECA model also reveals the role of the distillation loss in the teacher model, verifying the effective guiding role of the teacher model on the student model. At the same time, the introduction of the multi-level enhanced loss further enhances the student model's learning ability of the teacher model, verifying the necessity of constructing a multi-level enhanced loss. Compared with the MNTM-TM model, the performance of the MNTM-MECA model can be improved by up to about 2.7%.

[0118] Under five different signal-to-noise ratio conditions, the introduction of speech enhancement auxiliary tasks significantly improved model performance. In the signal-to-noise ratio range of -10dB to 10dB, the introduction of speech enhancement auxiliary tasks can improve the classification accuracy by up to about 1.6%. This phenomenon is consistent with the theory of multi-task learning in existing literature, that is, by adding auxiliary tasks, the performance of the model on the main task can be improved. In addition, the introduction of appropriate auxiliary tasks can enhance the model's learning ability for specific tasks, thereby further improving its overall performance.

[0119] In summary, this embodiment proposes a multi-task constrained end-to-end noisy speech emotion recognition method based on a teacher-student model, and proposes three methods that can improve the performance of the speech separation system from three aspects: teacher model, multi-level connection, and speech enhancement. The experimental results verify the rationality and effectiveness of these three methods. Compared with other methods, the method of this embodiment is improved in multiple measurement indicators, can overcome the influence of noise on speech emotion recognition to a certain extent, and has a good reference significance in practical applications.

[0120] The above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A multi-task constrained end-to-end noisy speech emotion recognition method based on teacher-student model, It is characterized in that The method comprises the following steps: Step 1: Input the clean speech signal into the teacher model, which is composed of a CNN feature encoder consisting of 7 one-dimensional convolutional layers and a Wav2vec 2.0 model consisting of 12 Transformer layers. Then, through the fully connected layer, the letter vector and emotion soft label are obtained, and the teacher model network is fine-tuned under the joint constraints of speech recognition and emotion recognition. Step 2: Input the noisy speech signal into the student model. The student model emotion prediction task is composed of a CNN feature encoder consisting of 7 one-dimensional convolutional layers and a Wav2vec 2.0 model consisting of 12 Transformer layers, and then pass through the fully connected layer to obtain the predicted label; Step 3: Input the noisy speech signal into the student model. The student model speech enhancement task consists of a CNN feature encoder consisting of 7 one-dimensional convolutional layers and a speech enhancement structure consisting of 12 Transformer layers and 7 one-dimensional transposed convolutional layers, and outputs the predicted speech; Step 4: Based on the proposed multiple joint loss functions, a composite constraint mechanism is formed to guide and regulate the training process of the student model and improve the accuracy of the model in a noisy environment; Step 5: Perform a performance evaluation on the proposed multi-task constrained end-to-end noisy speech emotion recognition method based on the teacher-student model.

2. According to the multi-task constrained end-to-end noisy speech emotion recognition method based on the teacher-student model in claim 1, It is characterized in that The specific steps of step 1 include: Step 1-1: The input noisy speech signal audio waveform x is passed through 7 one-dimensional convolutional layers to obtain the downsampled audio feature z; Step 1-2: The downsampled audio feature z output by the feature encoder is passed through 12 Transformer layers to obtain the context representation c; Step 1-3: The context representation c is passed through the fully connected layer to obtain the letter vector and sentiment soft label The letter vectors are no longer involved in subsequent tasks.

3. According to the multi-task constrained end-to-end noisy speech emotion recognition method based on teacher-student model in claim 1, It is characterized in that The specific steps of step 2 include: Step 2-1: The input clean speech signal audio waveform x' is passed through 7 one-dimensional convolutional layers to obtain the downsampled audio feature z; Step 2-2: The downsampled audio feature z' output by the feature encoder is passed through 12 Transformer layers to obtain the context representation c'; Step 2-3: The context representation C′ is passed through the fully connected layer to obtain the sentiment label Participate in subsequent tasks.

4. According to the multi-task constrained end-to-end noisy speech emotion recognition method based on teacher-student model in claim 1, It is characterized in that The specific steps of step 3 include: Step 3-1: The input clean speech signal audio waveform x′ is passed through 7 one-dimensional convolutional layers to obtain the downsampled audio feature z′; Step 3-2: The downsampled audio feature z′ output by the feature encoder is passed through 12 Transformer layers to obtain the context representation c′; Step 3-3: The context representation c′ is passed through the speech reconstruction module to obtain the predicted speech to participate in subsequent tasks.

5. According to the multi-task constrained end-to-end noisy speech emotion recognition method based on teacher-student model in claim 4, It is characterized in that In step 3-3, the specific steps of implementing the speech reconstruction module include: Step 3-3-1: Construct a speech reconstruction module, which consists of 7 transposed convolutional neural networks (TransConv) that match the dimensions of the CNN feature extractor; Step 3-3-2: Use the residual connection technology to connect the odd-layer network output of the CNN feature extractor with the corresponding layer output of the speech reconstruction module.

6. According to the multi-task constrained end-to-end noisy speech emotion recognition method based on teacher-student model in claim 1, It is characterized in that The specific steps of step 4 include: Step 4-1: The calculation formula of distillation loss KL is expressed as: in, is the soft label of the i-th sample obtained by the teacher model, is the soft prediction of the i-th sample obtained by the student model, T is the distillation temperature coefficient, is a sentiment soft label, For the sentiment label, we get the distillation loss; Step 4-2: Construct the sentiment label loss function as a multi-classification cross entropy loss function: Among them, M represents the number of samples, N represents the number of emotion categories, and p(x ij ) represents the predicted probability that the observed sample i belongs to category j, and the sentiment label loss is obtained; Step 4-3: Construct a multi-level enhancement loss function: Where M is defined as the number of samples, is the output of the i-th layer of the teacher model, represents the feature vector of the output of the i-th layer of the student network model, that is, the predicted value of the i-th layer of the student model, and i represents the number of layers of the Transformer network of the teacher model and the student model; The losses between the even-numbered layers of the teacher model and the student model Transformer network are calculated separately, for a total of 6 layers, and the formula is as follows: Get multi-level enhancement loss; Step 4-4: Construct the scale-invariant signal-to-noise ratio loss function (SI-SNR): Among them, s targct represents the target signal, e noisc represents the noise signal, ||·|| 2 represents the L2 norm, which is a measure of signal energy, and obtains the speech enhancement loss; Step 4-5: Combine the distillation loss, sentiment label loss, multi-level enhancement loss and speech enhancement loss to constrain the training of the student model: L ALL =αL KL +(1-α)L CE +βL S +γL SI-SNR , Among them, α is the weight of distillation loss, β is the weight of multi-level enhancement loss, and γ is the weight of speech enhancement loss.

7. According to the multi-task constrained end-to-end noisy speech emotion recognition method based on teacher-student model in claim 1, It is characterized in that In step 5, a specific method for evaluating the performance of the proposed multi-task constrained end-to-end noisy speech emotion recognition method based on a teacher-student model includes the following steps: Step 5-1: Conduct ablation experiments on the multi-task constrained end-to-end noisy speech emotion recognition method based on the teacher-student model and the method of removing the teacher model to verify the effectiveness of the teacher-student model; Step 5-2: Conduct ablation experiments on the multi-task constrained end-to-end noisy speech emotion recognition method based on the teacher-student model and the method of removing multi-level enhanced connections to verify the effectiveness of the teacher-student model; Step 5-3: Conduct ablation experiments on the multi-task constrained end-to-end noisy speech emotion recognition method based on the teacher-student model and the method of removing speech enhancement to verify the effectiveness of the speech enhancement module; Step 5-4: Discuss the specific value of the weight loss in the joint constraint and explore the optimal weight value.

Images