A multi-task constraint end-to-end noisy speech emotion recognition method based on teacher-student model

Abstract

The application discloses a kind of multi-task constraint end-to-end noisy speech emotion recognition methods based on teacher-student model, comprising: propose an end-to-end multi-task constraint network based on teacher-student model, draw lessons from the idea of knowledge distillation, after the multi-task training of teacher model is completed, the mature teacher model that has been trained is extracted to pure, significant emotional features, to guide the training process of student model;A multi-level enhancement loss mechanism is introduced, so that the student model can learn the key emotional information extracted from the clean speech by the teacher model in the middle layer;The idea of speech enhancement auxiliary task is adopted, and the speech enhancement auxiliary task is combined with multi-level enhancement loss;During the test phase, only the core process specific to speech emotion recognition classification in the student model needs to be focused on, effectively improving the execution rate of the test.The application realizes the speech emotion recognition task in noisy environment, improves the accuracy of emotion classification, and is suitable for intelligent customer service and mental health monitoring.

Description

Technical Field

[0001] This invention belongs to the field of speech emotion recognition technology, specifically involving a multi-task constrained end-to-end noisy speech emotion recognition method based on a teacher-student model. Background Technology

[0002] Currently, speech emotion recognition systems have demonstrated excellent performance under ideal environmental conditions. However, when these systems are applied to the real world, environmental noise interferes with the effective information in the speech signal, making it extremely difficult to detect significant speech emotion representations. Researchers have undertaken targeted efforts from multiple aspects, including preprocessing operations to improve or eliminate noise in the original speech signal, selecting features that are more robust to noise, and constructing more robust and stable models.

[0003] Preprocessing algorithms improve speech signal quality by reducing or eliminating the impact of noise. Existing methods include spectral subtraction, audio enhancement methods based on optimal modified logarithmic spectral amplitude estimators, and methods using filters for preprocessing, among others. Deep learning-based speech enhancement is also showing increasing advantages in recognizing emotions in noisy speech. For example, one study used a Long Short-Term Memory (LSTM) architecture to build an enhancement model and combined it with a post-processing method that estimates the LSTM structure mask using improved speech presence probabilities, thereby improving recognition accuracy. Chakraborty et al. investigated a feature compensation technique based on vector Taylor series expansions of noise Mel-frequency cepstral coefficients and improved this technique with vector Taylor series expansions containing auditory masking formulas, demonstrating significant performance advantages.

[0004] Feature selection aims to identify the most relevant features, extract features from a broader feature set, and reduce data dimensionality to reduce overfitting and increase interpretability. In research on speech emotion recognition in noisy environments, researchers have explored and proposed various feature extraction strategies, including noise adaptation, speaker adaptation, adaptation combining speaker and noise, and noise-specific feature selection. These methods are often combined with relevance-based feature subset selection and sequential floating forward search to improve model performance in noisy environments. Bandela et al. proposed a feature selection method based on low-level descriptor performance, using a Convolutional Neural Network (CNN) structure to test on 65 low-level descriptors in the ComParE2013 feature set, evaluating the noise robustness of different feature selection methods. Experimental results show that this method improves speech emotion recognition performance by 24.4% (wake-up), 23.9% (advantage), and 43.2% (valence) under 10dB noise.

[0005] To address noise interference, building more robust and robust models is one of the key methods to improve the overall model recognition rate. Triantafyllopoulos et al. studied training scalable deep learning architectures to enhance audio signals in low signal-to-noise ratio environments. Zhu Zhou et al. proposed a virtual amplification method and a robust multi-scene speech emotion recognition system, synthesizing a new database by modifying the environmental conditions of the audio and simulating experiments under different levels of Gaussian white noise, real-world noise, and reverberation. Tan et al. used semi-supervised learning techniques to recognize musical emotions and conducted student training experiments in noisy environments, contributing more inspiration and insights. Guimaraes et al. used noise and reverberation to enhance training data and combined course learning and multi-task learning methods to build a speech emotion recognition model based on HuberT. Experimental results showed that the model exhibited certain advantages in "outdoor" speech applications. Jaiswal et al. studied the impact of noise on human and machine perception of emotions and explored methods for integrating denoising feature spaces and speech enhancement algorithms with emotion recognition models. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings and deficiencies of existing technologies by proposing a multi-task constrained end-to-end noisy speech emotion recognition method based on a teacher-student model. This method utilizes the teacher model to extract significant emotion features through multi-task training and borrows knowledge distillation to guide student model training, improving accuracy and robustness. A multi-level enhancement loss mechanism is introduced, enabling the student model to learn key emotion information from the teacher model in intermediate layers, making noisy speech features approximate clean speech features. This, combined with a hyperparameter loss function, improves model accuracy in noisy environments. A composite, multi-level speech enhancement auxiliary structure is constructed by combining speech enhancement auxiliary tasks with multi-level enhancement loss, enhancing robustness in noisy environments. During the testing phase, only the core process of the student model is considered, without relying on the teacher model or auxiliary tasks, reducing testing complexity and increasing testing speed.

[0007] The technical solution adopted by this invention to solve its technical problem is: to provide a multi-task constrained end-to-end noisy speech emotion recognition method based on a teacher-student model, the method comprising the following steps.

[0008] Step 1: Input the clean speech signal into the teacher model. The teacher model consists of a cascaded CNN feature encoder composed of 7 one-dimensional convolutional layers and a Wav2vec 2.0 model composed of 12 Transformer layers. Then, through the fully connected layer, letter vectors and sentiment soft labels are obtained. The teacher model network is fine-tuned under the joint constraints of speech recognition and sentiment recognition.

[0009] Step 2: Input the noisy speech signal into the student model. The student model's sentiment prediction task consists of a cascaded CNN feature encoder composed of 7 one-dimensional convolutional layers and a Wav2vec 2.0 model composed of 12 Transformer layers. Then, the predicted label is obtained through a fully connected layer.

[0010] Step 3: Input the noisy speech signal into the student model. The speech enhancement task of the student model consists of a CNN feature encoder composed of 7 one-dimensional convolutional layers and a speech enhancement structure composed of 12 Transformer layers and 7 one-dimensional transposed convolutional layers, and outputs the predicted speech.

[0011] Step 4: Based on the proposed multiple joint loss function, a composite constraint mechanism is formed to guide and regulate the training process of student models and improve the accuracy of models in noisy environments.

[0012] Step 5: Evaluate the performance of the proposed end-to-end noisy speech emotion recognition method based on a teacher-student model with multi-task constraints.

[0013] Furthermore, the specific steps of step 1 are as follows:

[0014] Step 1-1: The input noisy speech signal audio waveform X is passed through 7 one-dimensional convolutional layers to obtain downsampled audio features Z;

[0015] Step 1-2: The downsampled audio features z output by the feature encoder are passed through 12 Transformer layers to obtain the context representation c;

[0016] Steps 1-3: The context representation c is passed through a fully connected layer to obtain letter vectors and sentiment soft tags y. i t The letter vectors are no longer involved in subsequent tasks.

[0017] Furthermore, the specific steps of step 2 are as follows:

[0018] Step 2-1: The input clean speech signal audio waveform x' is passed through 7 one-dimensional convolutional layers to obtain the downsampled audio features z';

[0019] Step 2-2: The downsampled audio feature z' output by the feature encoder is passed through 12 Transformer layers to obtain the context representation c';

[0020] Steps 2-3: The context representation c' is passed through a fully connected layer to obtain the sentiment label. Participate in subsequent tasks.

[0021] Furthermore, the specific steps of step 3 are as follows:

[0022] Step 3-1: The input clean speech signal audio waveform x' is passed through 7 one-dimensional convolutional layers to obtain the downsampled audio features z';

[0023] Step 3-2: The downsampled audio feature z' output by the feature encoder is passed through 12 Transformer layers to obtain the context representation c';

[0024] Step 3-3: The context representation c' is processed by the speech reconstruction module to obtain the predicted speech for subsequent tasks.

[0025] Furthermore, in step 3-3, the specific method for implementing the speech reconstruction module includes the following steps:

[0026] Step 3-3-1: Construct the speech reconstruction module, which consists of 7 transposed convolutional neural networks that match the dimensions of the CNN feature extractor;

[0027] Step 3-3-2: Using residual connection technology, the output of the odd-numbered layers of the CNN feature extractor is connected to the corresponding layer output of the speech reconstruction module, which effectively promotes the flow of information within the network.

[0028] Furthermore, the specific steps of step 4 are as follows:

[0029] Step 4-1: The formula for calculating distillation loss KL is expressed as follows:

[0030]

[0031] in, It is the soft label of the i-th sample obtained by the teacher model. It is a soft prediction of the i-th sample obtained from the student model, where T is the distillation temperature coefficient. As a soft label for emotions, For sentiment tags, distillation loss is obtained;

[0032] Step 4-2: Construct the sentiment label loss function as a multi-class cross-entropy loss function:

[0033]

[0034] Where M represents the number of samples, N represents the number of sentiment categories, and p(x) ij The value represents the predicted probability that observed sample i belongs to category j, and the sentiment label loss is obtained.

[0035] Step 4-3: Construct a multi-level augmentation loss function:

[0036]

[0037] Where M is defined as the number of samples, is the output of the i-th layer of the teacher model, represents the feature vector of the i-th layer of the student network model, that is, the predicted value of the i-th layer of the student model, and i represents the number of layers of the Transformer network of the teacher model and the student model.

[0038] The loss between even-numbered layers of the Transformer network for both the teacher and student models (a total of 6 layers) was calculated, as shown in the following formula:

[0039]

[0040] The loss is enhanced by multiple levels;

[0041] Step 4-4: Construct the scale-invariant signal-to-noise ratio loss function (SI-SNR):

[0042]

[0043] Among them, s targct Represents the target signal (i.e., the original clean speech signal), e noisc Represents the difference between the noise signal (i.e., the enhanced signal and the target signal), ||·|| 2 The L2 norm, a measure of signal energy, is used to obtain the speech enhancement loss.

[0044] Steps 4-5: Jointly constrain the training of the student model by applying the distillation loss, sentiment labeling loss, multi-level enhancement loss, and speech enhancement loss together.

[0045] L ALL =αL KL +(1-α)L CE +βL S +γL SI-SNR ,

[0046] Where α is the weight of the distillation loss, β is the weight of the multi-level enhancement loss, and γ is the weight of the speech enhancement loss.

[0047] Furthermore, in step 5, the specific steps for performance evaluation of the proposed multi-task constrained end-to-end noisy speech emotion recognition method based on a teacher-student model are as follows:

[0048] Step 5-1: Conduct ablation experiments on the multi-task constrained end-to-end noisy speech emotion recognition method based on the teacher-student model and the method of removing the teacher model to verify the effectiveness of the teacher-student model;

[0049] Step 5-2: Conduct ablation experiments on the multi-task constrained end-to-end noisy speech emotion recognition method based on the teacher-student model and the method of removing multi-level enhancement connections to verify the effectiveness of the teacher-student model;

[0050] Step 5-3: Conduct ablation experiments on the end-to-end noisy speech emotion recognition method based on the teacher-student model and the speech enhancement removal method to verify the effectiveness of the speech enhancement module;

[0051] Step 5-4: Discuss the specific values ​​of the weights in the weight loss of the joint constraints, and explore the optimal weight values.

[0052] Beneficial effects:

[0053] 1. This invention uses a teacher-student model and draws on the idea of ​​knowledge distillation. Through multi-task training of the teacher model, pure and significant emotional features are extracted to guide the training process of the student model, thereby improving the accuracy and robustness of the model and enhancing its generalization ability.

[0054] 2. This invention introduces a multi-level enhancement loss mechanism, which enables the student model to learn key emotional information extracted from clean speech by the teacher model in the intermediate layer, so that the features learned by the student model from noisy speech approximate the features of clean speech. This invention combines multiple loss functions through multiple hyperparameters, which can achieve the effect of improving the accuracy of the model in noisy environments.

[0055] 3. This invention adopts the idea of ​​speech enhancement auxiliary task, combining speech enhancement auxiliary task with multi-level enhancement loss to realize a composite multi-level speech enhancement auxiliary structure, which improves the robustness of speech emotion recognition task in noisy environments.

[0056] 4. During the testing phase, this invention only needs to focus on the core process specific to speech emotion recognition and classification in the student model, without relying on auxiliary task branches designed in the teacher model or student model. This significantly reduces the model complexity required during testing, thereby effectively improving the execution speed of testing without sacrificing accuracy. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the operation process of the present invention.

[0058] Figure 2 This is a diagram of the speech enhancement framework in the student model of the present invention.

[0059] Figure 3 This is a diagram illustrating the multi-level enhancement loss framework of the present invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be noted that, to avoid obscuring the invention with unnecessary details, only structures and / or processing steps closely related to the invention are shown in the drawings, while other details not directly related to the invention are omitted.

[0061] Additionally, it should be noted that the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0062] like Figures 1 to 3 As shown, this invention proposes a multi-task constrained end-to-end noisy speech emotion recognition method based on a teacher-student model. The teacher-student model and speech enhancement module are applied to speech emotion recognition to extract deep features. This method exhibits robustness to noise and can construct a higher-performance speech emotion recognition system. The specific steps include:

[0063] Step 1: Input the clean speech signal into the teacher model. The teacher model consists of a cascaded CNN feature encoder composed of 7 one-dimensional convolutional layers and a Wav2vec 2.0 model composed of 12 Transformer layers. Then, through the fully connected layer, letter vectors and sentiment soft labels are obtained. The teacher model network is fine-tuned under the joint constraints of speech recognition and sentiment recognition.

[0064] Step 2: Input the noisy speech signal into the student model. The student model's sentiment prediction task consists of a cascaded CNN feature encoder composed of 7 one-dimensional convolutional layers and a Wav2vec 2.0 model composed of 12 Transformer layers. Then, the predicted label is obtained through a fully connected layer.

[0065] Step 3: Input the noisy speech signal into the student model. The speech enhancement task of the student model consists of a CNN feature encoder composed of 7 one-dimensional convolutional layers and a speech enhancement structure composed of 12 Transformer layers and 7 one-dimensional transposed convolutional layers, and outputs the predicted speech.

[0066] Step 4: Based on the proposed multiple joint loss function, a composite constraint mechanism is formed to guide and regulate the training process of student models and improve the accuracy of models in noisy environments.

[0067] Step 5: Evaluate the performance of the proposed end-to-end noisy speech emotion recognition method based on a teacher-student model with multi-task constraints.

[0068] This experiment used IEMOCAP, a multimodal database for the SER domain, containing 12 hours of recorded data, divided into impromptu and scripted sections. The corpus includes nine emotions: anger, disgust, fear, frustration, sadness, happiness, excitement, surprise, and neutrality. Due to dataset imbalance, researchers typically choose the four most common emotions. Because excitement and happiness are similar and happiness samples are scarce, researchers sometimes substitute excitement for happiness or combine the two to increase the data volume. This experiment used four emotional states (neutral, excited, sad, and angry). The noise in the experiment came from the MUSAN dataset, which includes 11 types of real-world environmental noise. To simulate noisy scenarios, this experiment constructed a noisy speech dataset by randomly mixing clean audio from IEMOCAP with noise from the MUSAN dataset.

[0069] Step 1 includes the following steps:

[0070] Step 1-1: The input noisy speech signal audio waveform X is passed through 7 one-dimensional convolutional layers to obtain downsampled audio features Z;

[0071] Step 1-2: The downsampled audio features z output by the feature encoder are passed through 12 Transformer layers to obtain the context representation c;

[0072] Steps 1-3: The context representation c is processed through a fully connected layer to obtain letter vectors and sentiment soft tags. The letter vectors will no longer be involved in subsequent tasks.

[0073] Step 2 includes the following specific steps:

[0074] Step 2-1: The input clean speech signal audio waveform x' is passed through 7 one-dimensional convolutional layers to obtain the downsampled audio features Z';

[0075] Step 2-2: The downsampled audio feature z' output by the feature encoder is passed through 12 Transformer layers to obtain the context representation c';

[0076] Steps 2-3: The context representation c' is passed through a fully connected layer to obtain the sentiment label. Participate in subsequent tasks.

[0077] Step 3 includes the following specific steps:

[0078] Step 3-1: The input clean speech signal audio waveform x' is passed through 7 one-dimensional convolutional layers to obtain the downsampled audio features z';

[0079] Step 3-2: The downsampled audio feature z' output by the feature encoder is passed through 12 Transformer layers to obtain the context representation c';

[0080] Step 3-3: The context representation c' is processed by the speech reconstruction module to obtain the predicted speech for subsequent tasks.

[0081] In step 3-3 of the present invention, the specific method for implementing the speech reconstruction module includes the following steps:

[0082] Step 3-3-1: Construct the speech reconstruction module, which consists of 7 transposed convolutional neural networks that match the dimensions of the CNN feature extractor;

[0083] The TransConv parameters for the 7 layers are shown in Table 1:

[0084] Table 1 Core parameters of the speech reconstruction module

[0085] TransConv7 2 1024 512 2 TransConv6 2 512 512 2 TransConv5 3 1024 512 2 TransConv4 3 512 512 2 TransConv3 3 1024 512 2 TransConv2 3 512 512 2 TransConv1 10 1024 1 5

[0086] Step 3-3-2: Using residual connection technology, the output of the odd-numbered layers of the CNN feature extractor is connected to the corresponding layer output of the speech reconstruction module, which effectively promotes the flow of information within the network.

[0087] Step 4 includes the following specific steps:

[0088] Step 4-1: The formula for calculating distillation loss KL is expressed as follows:

[0089]

[0090] in, It is the soft label of the i-th sample obtained by the teacher model. It is a soft prediction of the i-th sample obtained from the student model, where T is the distillation temperature coefficient. As a soft label for emotions, For sentiment tags, distillation loss is obtained;

[0091] Step 4-2: Construct the sentiment label loss function as a multi-class cross-entropy loss function:

[0092]

[0093] Where M represents the number of samples, N represents the number of sentiment categories, and p(x) ij The value represents the predicted probability that observed sample i belongs to category j, and the sentiment label loss is obtained.

[0094] Step 4-3: Construct a multi-level augmentation loss function:

[0095]

[0096] Where M is defined as the number of samples, is the output of the i-th layer of the teacher model, represents the feature vector of the i-th layer of the student network model, that is, the predicted value of the i-th layer of the student model, and i represents the number of layers of the Transformer network of the teacher model and the student model.

[0097] The loss between even-numbered layers of the Transformer network for both the teacher and student models (a total of 6 layers) was calculated, as shown in the following formula:

[0098]

[0099] The loss is enhanced by multiple levels;

[0100] Step 4-4: Construct the scale-invariant signal-to-noise ratio loss function (SI-SNR):

[0101]

[0102] Among them, s targct Represents the target signal (i.e., the original clean speech signal), e noisc Represents the difference between the noise signal (i.e., the enhanced signal and the target signal), ||·|| 2 The L2 norm is used to represent the signal energy; this yields the speech enhancement loss.

[0103] Steps 4-5: Jointly constrain the training of the student model by applying the distillation loss, sentiment labeling loss, multi-level enhancement loss, and speech enhancement loss together.

[0104] L ALL =αL KL +(1-α)L CE +βL S +γL SI-SNR ,

[0105] Where α is the weight of the distillation loss, β is the weight of the multi-level enhancement loss, and γ is the weight of the speech enhancement loss.

[0106] In step 5 of the present invention, the specific method for performance evaluation of the proposed multi-task constrained end-to-end noisy speech emotion recognition method based on a teacher-student model includes the following steps:

[0107] Step 5-1: An ablation experiment is conducted between a multi-task constrained end-to-end noisy speech emotion recognition method based on a teacher-student model and a method that removes the teacher model to verify the effectiveness of the teacher-student model.

[0108] Step 5-2: An ablation experiment is conducted between a multi-task constrained end-to-end noisy speech emotion recognition method based on a teacher-student model and a method for removing multi-level enhancement connections to verify the effectiveness of the teacher-student model.

[0109] Step 5-3: An ablation experiment is conducted between a multi-task constrained end-to-end noisy speech emotion recognition method based on a teacher-student model and a speech enhancement removal method to verify the effectiveness of the speech enhancement module.

[0110] Step 5-4: Discuss the specific values ​​of the weights in the weight loss of the joint constraints, and explore the optimal weight values.

[0111] The simulation experiments and data of this invention are as follows:

[0112] In this embodiment, the evaluation metrics used are WA and UA, commonly used in the SER domain. A sampling rate of 16kHz was used for all signals. In the experiment, the noisy speech dataset was randomly divided into 80% training data and 20% test data, and 5-fold cross-validation was performed. The entire training process lasted 200 epochs in pursuit of model optimization. The loss function used in the training phase of the proposed MNTM is joint loss, where, after loss function hyperparameter optimization, α, β, and γ were set to 0.7, 3, and -0.1, respectively. T was set to 5. The AdamW optimizer was used in the experiment. The batch size of the model was set to 2. This experiment was built on the PyTorch framework and run on an NVIDIA Tesla P100 SXM2 machine. All experiments were conducted using CUDA version 10.2.

[0113] Ablation experiments were conducted to validate the teacher model, multi-level connectivity, and speech enhancement. The results are shown in Table 2.

[0114] Table 2 Ablation experiments of MNTM in the IEM+MUS database

[0115]

[0116] Under the same signal-to-noise ratio (SNR) environment, by comparing and analyzing the confusion matrices of the MNTM and MNTM-TM models, it is evident that the supervision mechanism of introducing a teacher model significantly improves the resistance of the MNTM model to noise interference and maintains high classification accuracy. Furthermore, this chapter provides an in-depth analysis of misclassification in the confusion matrix. By comparing the misclassification ratios among different emotion categories, it can be found that certain emotion categories (such as natural) are more easily confused in noisy environments. Therefore, when designing an emotion-based speech recognition system, special attention needs to be paid to these easily confused emotion categories, and corresponding strategies should be adopted to improve the model's recognition ability.

[0117] By comparing the performance of models MNTM and MNTM-MECA under various signal-to-noise ratios, it can be observed that the overall performance of the model declines when the multi-level enhancement connection mechanism is removed from the overall model framework. This phenomenon indicates that the intermediate-level multi-level connection strategy can provide more effective learning support for the student model, especially in assisting it to learn the intermediate features extracted by the teacher model. This is beneficial for the student model to better capture the significant emotional information extracted by the teacher model from clean speech in noisy environments. Furthermore, the performance comparison between models MNTM-TM and MNTM-MECA also reveals the role of distillation loss in the teacher model, verifying the effective guidance of the teacher model on the student model. Simultaneously, the introduction of multi-level enhancement loss further strengthens the student model's learning ability from the teacher model, validating the necessity of constructing multi-level enhancement loss. Compared to model MNTM-TM, model MNTM-MECA can improve performance by up to approximately 2.7%.

[0118] Under five different signal-to-noise ratio (SNR) conditions, the introduction of a speech enhancement auxiliary task significantly improved model performance. Within the SNR range of -10 dB to 10 dB, the introduction of the speech enhancement auxiliary task improved classification accuracy by up to approximately 1.6%. This phenomenon aligns with existing theories on multi-task learning, namely, that adding auxiliary tasks can enhance the model's performance on the primary task. Furthermore, the appropriate introduction of auxiliary tasks can enhance the model's learning ability for specific tasks, thereby further improving its overall performance.

[0119] In summary, this embodiment proposes a multi-task constrained end-to-end noisy speech emotion recognition method based on a teacher-student model. It offers three methods to improve the performance of the speech separation system from three aspects: teacher model, multi-level connectivity, and speech enhancement. Experimental results verify the rationality and effectiveness of these three methods. Compared with other methods, the method in this embodiment achieves improvements in multiple measurement indicators and can overcome the impact of noise on speech emotion recognition to a certain extent, making it a valuable reference for practical applications.

[0120] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multi-task constrained end-to-end noisy speech emotion recognition method based on a teacher-student model, characterized in that, The method includes the following steps: Step 1: Input the clean speech signal into the teacher model. The teacher model consists of a cascaded CNN feature encoder composed of 7 one-dimensional convolutional layers and a Wav2vec 2.0 model composed of 12 Transformer layers. Then, through the fully connected layer, letter vectors and sentiment soft labels are obtained. The teacher model network is fine-tuned under the joint constraints of speech recognition and sentiment recognition. Step 2: Input the noisy speech signal into the student model. The student model's sentiment prediction task consists of a cascaded CNN feature encoder composed of 7 one-dimensional convolutional layers and a Wav2vec 2.0 model composed of 12 Transformer layers. Then, the predicted label is obtained through a fully connected layer. Step 3: Input the noisy speech signal into the student model. The speech enhancement task of the student model consists of a CNN feature encoder composed of 7 one-dimensional convolutional layers and a speech enhancement structure composed of 12 Transformer layers and 7 one-dimensional transposed convolutional layers, and outputs the predicted speech. Step 4: Based on the proposed multiple joint loss function, a composite constraint mechanism is formed to guide and regulate the training process of student models and improve the accuracy of models in noisy environments. Step 5: Evaluate the performance of the proposed end-to-end noisy speech emotion recognition method based on a teacher-student model with multi-task constraints.

2. The multi-task constrained end-to-end noisy speech emotion recognition method based on a teacher-student model according to claim 1, characterized in that, The specific steps of step 4 include: Step 4-1: The formula for calculating distillation loss KL is expressed as follows: ， ， ， in, It is the soft label of the i-th sample obtained by the teacher model. It is a soft prediction of the i-th sample obtained from the student model. This is the distillation temperature coefficient. As a soft label for emotions, For sentiment tags, distillation loss is obtained; Step 4-2: Construct the sentiment label loss function as a multi-class cross-entropy loss function: ， in, Indicates the number of samples. Represents the number of sentiment categories. The predicted probability that observed sample i belongs to category j is expressed as the sentiment label loss. Step 4-3: Construct a multi-level augmentation loss function: ， Where M is defined as the number of samples, is the output of the i-th layer of the teacher model, represents the feature vector of the i-th layer of the student network model, that is, the predicted value of the i-th layer of the student model, and i represents the number of layers of the Transformer network of the teacher model and the student model. The loss between even-numbered layers of the Transformer network for both the teacher and student models (a total of 6 layers) was calculated, as shown in the following formula: ， The loss is enhanced by multiple levels; Step 4-4: Construct the scale-invariant signal-to-noise ratio loss function (SI-SNR): ， in, Indicates the target signal. Indicates noise signal, The L2 norm, a measure of signal energy, is used to obtain the speech enhancement loss. Steps 4-5: Jointly constrain the training of the student model by applying the distillation loss, sentiment labeling loss, multi-level enhancement loss, and speech enhancement loss together. ， in, As a weight for distillation losses, For the weights of multi-level enhanced loss, Weights for speech enhancement loss.

3. The multi-task constrained end-to-end noisy speech emotion recognition method based on a teacher-student model according to claim 1, characterized in that, Step 5, which involves evaluating the performance of the proposed multi-task constrained end-to-end noisy speech emotion recognition method based on a teacher-student model, includes the following steps: Step 5-1: Conduct ablation experiments on the multi-task constrained end-to-end noisy speech emotion recognition method based on the teacher-student model and the method of removing the teacher model to verify the effectiveness of the teacher-student model; Step 5-2: Conduct ablation experiments on the multi-task constrained end-to-end noisy speech emotion recognition method based on the teacher-student model and the method of removing multi-level enhancement connections to verify the effectiveness of the teacher-student model; Step 5-3: Conduct ablation experiments on the end-to-end noisy speech emotion recognition method based on the teacher-student model and the speech enhancement removal method to verify the effectiveness of the speech enhancement module; Step 5-4: Discuss the specific values ​​of the weights in the weight loss of the joint constraints, and explore the optimal weight values.

Images