Earphone adaptive noise reduction method based on multi-task learning

Abstract

The invention relates to the technical field of earphone noise reduction, in particular to an earphone adaptive noise reduction method based on multi-task learning, and the method comprises the steps: 1, building an end-to-end multi-task deep learning earphone noise reduction model; 2, selecting an audio sample; step 3, training a to-be-trained multi-task deep learning noise reduction model by using the audio sample obtained in the step 2; 4, after training is completed, audio data are input into the model, an output end outputs a denoised signal, and the audio data are noiseless data and comprise voice of a target dialogue; and step 5, calculating a confidence score of the noise reduction signal output in the step 4, the confidence score being used for evaluating the audio quality, and through a multi-task learning framework, significantly improving the adaptability of the model to different noise environments. The basic noise reduction model focuses on the noise reduction effect, the noise classification model assists in recognizing the noise type, and the audible noise reduction output signal model ensures the naturalness of output voice.

Description

Technical Field

[0001] The present invention relates to the technical field of headphone noise reduction, and in particular to a headphone adaptive noise reduction method based on multi-task learning. Background Art

[0002] In recent years, with the rapid development of mobile communication technology and the popularity of smart headphones, people's requirements for audio quality have been increasing. Maintaining clear voice calls in noisy environments has become an urgent need for users. To this end, headphone noise reduction technology has continued to evolve, from the initial analog noise reduction to digital signal processing, and then to today's artificial intelligence algorithms. Each technological innovation brings users a better auditory experience.

[0003] In the current state-of-the-art headphone noise reduction technology, deep learning-based methods have shown great potential. These methods train neural networks to learn noise characteristics, thereby achieving more accurate noise reduction effects. However, existing deep learning noise reduction algorithms still face some challenges.

[0004] First, most existing methods adopt a single-task learning framework, focusing on separating clean speech from noisy speech. Although this method performs well in specific noise environments, it often lacks the ability to adapt to complex and changing noise environments. In actual applications, users may face a variety of different noise scenes such as offices, streets, and cafes. Single-task noise reduction models are difficult to cope with such diverse environmental changes.

[0005] Secondly, while pursuing high noise reduction effects, existing methods often ignore the naturalness and intelligibility of speech. Excessive noise reduction processing may cause speech distortion, making the call content difficult to understand, which obviously runs counter to the original intention of improving call quality. How to maintain the naturalness of speech while effectively reducing noise has become a difficult problem that needs to be solved urgently.

[0006] In addition, many deep learning noise reduction algorithms have high computational complexity and are difficult to run in real time on resource-constrained mobile devices. This greatly limits the application of these algorithms in actual products. How to reduce the computational complexity of the algorithm while ensuring the noise reduction effect and improve its practicality on mobile devices is also a major challenge facing existing technologies.

[0007] In view of the above problems, there is an urgent need for a new headphone noise reduction method that can take into account noise reduction effect, voice quality and calculation efficiency at the same time and has the ability to adapt to various complex noise environments. The present invention is proposed to meet this demand. Summary of the invention

[0008] This paper proposes a multi-task based deep learning headphone noise reduction adaptive method, aiming to solve the problems existing in the prior art. This method organically combines basic noise reduction, noise classification and audible noise reduction output signal processing through an innovative multi-task learning framework, achieving accurate adaptation to complex noise environments and high-quality speech output.

[0009] The present invention proposes a multi-task based deep learning headphone noise reduction adaptive method, comprising the following steps:

[0010] Step 1, establishing an end-to-end multi-task deep learning headphone noise reduction model, taking the basic noise reduction model, the noise classification model and the audible noise reduction output signal model as three tasks, for processing the noise reduction of the headphone associated noise of voice calls; in the model, there are shared layers and exclusive layers between the three tasks; during training, the output of the shared layer is used as the input of the three tasks, the exclusive layer uses its own data set for training, and sends the training results to the shared layer, thereby adjusting the shared layer to achieve simultaneous optimization of the three tasks and ultimately output the noise reduction results;

[0011] Step 2, selecting an audio sample, wherein the sample is a data set based on real noise and speech, and is selected from an existing clean data set; the audio sample is a data set with noise added, and the noise is generated according to the statistical characteristics of the noise required to be classified by the model, and the noise includes nonlinear noise and white noise;

[0012] Step 3, using the audio samples obtained in step 2 to train the multi-task deep learning denoising model to be trained, inputting the noisy audio into the model, classifying it according to its original data, calculating the loss value of each task of the model in turn, and using the back propagation algorithm to update the model parameters to obtain the final trained multi-task deep learning denoising model;

[0013] Step 4: After the training is completed, the audio data is input into the model, and the output end outputs the noise-reduced signal, wherein the audio data is noise-free data and contains the speech of the target conversation;

[0014] Step 5, calculating the confidence score of the noise reduction signal output in step 4, wherein the confidence score is used to evaluate the audio quality.

[0015] Preferably, the end-to-end multi-task deep learning headphone noise reduction model includes three sub-models: a basic noise reduction model, a noise classification model and an audible noise reduction output signal model; for the training stage: the input of the basic noise reduction model is a noisy signal, and the output is estimated clean data; the input of the noise classification model is a noisy signal, and the output is a classification result of the noisy signal; the input of the audible noise reduction output signal model is clean data and noise reduction data output by the basic noise reduction model, and outputs audible data; for the testing stage: the noisy data is input into the basic noise reduction model to obtain a basic noise reduction layer, and the output signal of the basic noise reduction layer is input into the audible noise reduction model together with the clean data to obtain the final noise reduction Output signal; the basic denoising model uses a convolutional neural network, and the convolution layer is composed of a convolutional layer, a rectified linear unit and a pooling layer in sequence. During training, the vector representation data of the noise is input into the convolutional neural network, and a filter model with acoustic function, that is, estimated clean data, is obtained through training; the audible denoising output signal model is a feedforward neural network, which uses an attention mechanism and utilizes the basic denoising model and audible data to train an output nonlinear sound filter function in the audible denoising output signal model, and maintains the characteristics of the original speech signal; in the training stage, the denoised signal output by the basic denoising model is used as input together with the clean data, and the output is used as the final denoised data.

[0016] Preferably, the basic noise reduction model needs to distinguish whether the filter is linear or nonlinear; for this task, a clean output and a noisy input signal of speech are generated from a clean data set containing 100 clean speech sounds, that is, a data set containing 100 speech sounds in the clean data set is used, and Gaussian noise is first added, whose variance conforms to the normal distribution of (0, 0.3), and then the amplitude is adjusted, and the amplitude is (0.5, 1.5).

[0017] Preferably, the noise classification model is used to train a method that can automatically analyze noise and classify noise; a convolutional neural network is used, and the convolution layer is composed of a convolution layer, a space, a channel-level attention mechanism and a pooling layer in sequence, and includes a feature extractor for nonlinear mapping; during training, the vector representation data of the noise is input into the neural network, and finally a classifier that distinguishes various types of noise is trained; first, Gaussian noise is added to normal speech data, that is, the input signal is clean data plus Gaussian noise data; then specific nonlinear noise is added to different speech data, and here the nonlinear noise is synthesized from the original clean data using mathematical and statistical characteristics.

[0018] Preferably, the training process comprises the following steps:

[0019] Step S1, initialize the model parameters according to the training sample size; define the loss function, which is defined as the sum of the basic noise reduction layer loss function, the noise layer loss function and the audible noise reduction signal layer loss function, that is:

[0020] L total =L P +L N +L A

[0021] Among them, P represents the basic noise reduction model; L P Represents the loss function of the basic denoising model; L N represents the loss function of the noise classification model; L A represents the loss function of the audible noise reduction signal model;

[0022] Step S2, input the sample data into the shared layer, input the clean data and the noisy data into the basic denoising model respectively, calculate the loss functions of the two, and calculate the loss function of the shared layer together with the loss function of the audible denoising signal layer; use the back propagation algorithm to update the network parameters of the basic denoising layer and the audible denoising layer using the shared layer, and complete the update of the shared layer, the basic denoising layer and the audible denoising layer;

[0023] Step S3, the outputs of the basic noise reduction layer and the audible noise reduction layer are sent to the noise classification model, the noise classification model loss function is calculated, and the prediction probability of the noise classification model is obtained; the noise classification model loss function is calculated using the back propagation algorithm to update the network parameters of the noise classification layer, and the update of the noise classification layer is completed;

[0024] Step S4, return to step 2 until the termination condition is met;

[0025] Among them: the loss function used by the basic denoising model and the noise classification model is the cross entropy loss function, which is defined as:

[0026] L P =-∑(y k ·log(p k ))

[0027] L N =-∑(y i ·log(p i ))

[0028] Among them, y k Represents the deviation value of the basic denoising model on the time segment of k, y i represents the deviation value of the noise classification model, N refers to the number of training samples; p k represents the predicted probability of the output of the basic denoising model; p irepresents the predicted probability of the output of the noise classification model;

[0029] Step S5, tuning the basic denoising model on the test data set.

[0030] Preferably, the training process of the model also includes: the output of the basic noise reduction model is simultaneously used as the forward propagation data input required for updating the network parameters of the noise layer and the audible noise reduction layer.

[0031] Preferably, the training of the basic noise reduction model is used to process nonlinear noise, and it is necessary to distinguish whether the filter is linear or nonlinear; during network training, the original data is used as training samples, and a set of clean data plus specific nonlinear noise is used as the input of the model during training. During the training process, Gaussian noise is added to the speech in the clean data set, and the standard deviation of the Gaussian noise is randomly selected (0.01, 0.7); then, specific nonlinear noise is added to specific speech data.

[0032] Preferably, the training method of the audible noise reduction output signal model is:

[0033] Step Sa, firstly, the basic noise reduction model and the clean data are used as training samples of the audible noise reduction output model;

[0034] Step Sb, secondly, the nonlinear noise is input into the basic noise reduction model, and then the output of the basic noise reduction layer is obtained as the input of the audible noise reduction output model;

[0035] Step Sc, input the output data of the basic denoising model and the clean data into the audible denoising output signal model, then calculate the loss layer according to the error value between the output of the basic denoising model and the clean data; use the error of the basic denoising model to calculate the gradient, and then use the back propagation algorithm to update the network parameters;

[0036] Step Sd, return to step S2 until the termination condition is met;

[0037] For the audible noise reduction signal model, the logarithmic loss function is used. The loss function used by the audible noise reduction output signal model is the logarithmic loss function, which is defined as:

[0038] L A =-∑(y i ·log(p i )+(1-y i )·log(1-p i ))

[0039] Among them, y i represents the true label; p i Represents the probability predicted by the model.

[0040] Preferably, the testing process comprises the following steps:

[0041] Step T1, testing the input noisy speech information and preprocessing the data according to the basic noise reduction model;

[0042] Step T2, inputting the preprocessed noisy speech into the basic noise reduction model to obtain a noise reduction output signal;

[0043] Step T3, in the test phase, the denoising results output by the basic denoising model and the original clean data in the clean sample are sent to the audible denoising output signal model, the confidence score is calculated, the noise of the test set is evaluated, and the samples with confidence scores greater than the threshold are considered to be good samples and input into the basic denoising output model; the threshold is 0.85; finally, the denoising result in the basic denoising model is output as the final denoising model.

[0044] Preferably, during the basic denoising model training process, the loss function of the shared layer update is:

[0045] L shared =L P +L A

[0046] Among them, L P Represents the loss of the basic noise reduction layer; L A Represents the loss of the audible noise reduction layer.

[0047] Specifically, the method of the present invention includes the steps of establishing an end-to-end multi-task deep learning headphone noise reduction model, selecting appropriate audio samples, training the model, outputting the noise reduction signal and calculating the confidence score. This method can not only effectively reduce various types of background noise, but also maintain the naturalness and intelligibility of speech, and at the same time has low computational complexity and is suitable for real-time operation on mobile devices.

[0048] The beneficial effects of the present invention are mainly reflected in the following aspects:

[0049] First, through the multi-task learning framework, the present invention significantly improves the model's adaptability to different noise environments. The basic noise reduction model focuses on the noise reduction effect, the noise classification model assists in identifying the noise type, and the audible noise reduction output signal model ensures the naturalness of the output speech. This multi-task collaborative approach enables the system to quickly adapt to various complex noise environments and maintain excellent noise reduction effects in different scenarios such as offices, streets, and cafes.

[0050] Secondly, the present invention pays special attention to maintaining voice quality while ensuring powerful noise reduction capabilities. By introducing an audible noise reduction output signal model, the system can retain the key features of the voice during the noise reduction process, ensuring the clarity and intelligibility of the output voice. This solves the problem of excessive noise reduction leading to voice distortion in traditional methods, greatly improving the user's call experience.

[0051] Furthermore, the present invention adopts a network structure combining a shared layer and an exclusive layer, which effectively controls the computational complexity while improving performance. This enables the method to run in real time on mobile devices with limited resources, greatly expanding its practical application range.

[0052] In addition, the present invention introduces a confidence scoring mechanism, which can evaluate the noise reduction effect in real time and make adaptive adjustments as needed. This further improves the robustness and reliability of the system and ensures that high-quality speech signals can be output in various complex environments.

[0053] In general, the multi-task based deep learning headphone noise reduction adaptive method proposed in this invention has achieved significant improvements in noise reduction effect, voice quality, environmental adaptability and computing efficiency. It not only solves many problems in the prior art, but also opens up a new direction for the development of intelligent audio processing technology. We have reason to believe that this technology will bring users an unprecedented high-quality voice communication experience and create a peaceful sound world in the increasingly noisy modern life. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is the overall logic block diagram of the system of the present invention.

[0055] Figure 2 This is the training process of the end-to-end multi-task deep learning headphone noise reduction model of the present invention.

[0056] Figure 3 This is the processing flow of the basic noise reduction model of the present invention.

[0057] Figure 4 This is the processing flow of the noise classification model of the present invention.

[0058] Figure 5 This is the processing flow of the audible noise reduction output signal model of the present invention.

[0059] Figure 6 This is the logical flow of the training process of the present invention.

[0060] Figure 7 This is the logical flow of the testing process of the present invention. DETAILED DESCRIPTION

[0061] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects thereof are described in detail below in conjunction with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0062] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0063] See also Figure 1-7 The present invention discloses a multi-task based deep learning headphone noise reduction adaptive method, comprising the following steps:

[0064] First, the present invention establishes an end-to-end multi-task deep learning headphone noise reduction model. The model uses the basic noise reduction model 1, the noise classification model 2 and the audible noise reduction output signal model 3 as three tasks to process the noise reduction of headphone-associated noise in voice calls. In this model, there is a shared layer 4 and an exclusive layer 5 between the three tasks. During the training process, the output of the shared layer 4 is used as the input of the three tasks, while the exclusive layer 5 uses its own data set for training, and feeds back the training results to the shared layer 4, thereby adjusting the shared layer 4, and finally achieving the purpose of optimizing the three tasks at the same time and outputting the noise reduction results.

[0065] Secondly, the present invention selects audio samples. These samples are selected from existing clean data sets based on real noise and speech data sets. Specifically, we add noise to the data set, which is generated according to the statistical characteristics of the noise required for classification by the model, including nonlinear noise and white noise. For example, we can select 100 clean speech samples in different environments, and then add office background noise, street traffic noise, cafe noise, etc. to simulate various noise conditions in real scenes.

[0066] Then, the present invention uses the above audio samples to train the multi-task deep learning denoising model to be trained. We input the noisy audio into the model, classify it according to its original data, calculate the loss value of each task of the model in turn, and use the back propagation algorithm to update the model parameters, and finally obtain the trained multi-task deep learning denoising model.

[0067] Next, after training is complete, we input the audio data into the model. It should be noted that the audio data here is noise-free data and contains the speech of the target conversation. The output of the model will output the noise-reduced signal.

[0068] Finally, we calculate the confidence score of the output noise reduction signal. This confidence score is used to evaluate the quality of the audio. For example, we can set a threshold, such as 0.85. When the confidence score is higher than this threshold, we believe that the noise reduction effect is good; otherwise, further optimization is needed.

[0069] In one embodiment of the present invention, the end-to-end multi-task deep learning headphone noise reduction model includes three sub-models: a basic noise reduction model 1, a noise classification model 2, and an audible noise reduction output signal model 3. In the training phase, the input of the basic noise reduction model 1 is a noisy signal, and the output is estimated clean data; the input of the noise classification model 2 is a noisy signal, and the output is a classification result of the noisy signal; the input of the audible noise reduction output signal model 3 is clean data and the noise reduction data output by the basic noise reduction model 1, and outputs audible data. In the testing phase, the noisy data is input into the basic noise reduction model 1 to obtain a basic noise reduction layer, and the output signal of the basic noise reduction layer is input into the audible noise reduction model 3 together with the clean data, and finally a noise reduction output signal is obtained.

[0070] Preferably, the basic noise reduction model 1 uses a convolutional neural network. The convolutional layer of the convolutional neural network is composed of a convolutional layer, a rectified linear unit and a pooling layer in sequence. During training, we input the vector representation data of the noise into the convolutional neural network, and obtain a filtering model with acoustic functions through training, that is, estimated clean data. This structure can effectively capture the time and frequency characteristics of the audio signal, thereby achieving a more accurate noise reduction effect.

[0071] In another embodiment of the present invention, the basic noise reduction model 1 needs to distinguish whether the filter is linear or nonlinear. To achieve this goal, we generate a clean output and a noisy input signal of speech from a clean data set containing 100 clean speech. Specifically, we use a data set containing 100 segments of speech in the clean data set, first add Gaussian noise, whose variance conforms to the normal distribution of (0, 0.3), and then adjust the amplitude, and the adjusted amplitude range is (0.5, 1.5). The purpose of this is to simulate various noise conditions in the real environment and improve the robustness of the model.

[0072] Again, the noise classification model 2 of the present invention is used to train a method that can automatically analyze and classify noise. The model also uses a convolutional neural network, whose convolution layer is composed of a convolutional layer, a space, a channel-level attention mechanism and a pooling layer in sequence, and includes a feature extractor for nonlinear mapping. During the training process, we input the vector representation data of the noise into the neural network, and finally train a classifier that can distinguish various types of noise. We first add Gaussian noise to normal speech data, that is, the input signal is clean data plus Gaussian noise data; then add specific nonlinear noise to different speech data, where the original clean data is synthesized with mathematical and statistical characteristics. Nonlinear noise. This method enables the model to learn the characteristics of different types of noise, so as to more accurately identify and process various noises in practical applications.

[0073] The audible noise reduction output signal model 3 of the present invention is a feedforward neural network using an attention mechanism. We use the basic noise reduction model 1 and audible data to train a function of outputting a nonlinear sound filter in the audible noise reduction output signal model 3, and maintain the characteristics of the original speech signal. In the training phase, the noise reduction signal output by the basic noise reduction model 1 is used as input together with the clean data, and the output is used as the final noise reduction data. This method can retain the audibility and clarity of the original speech to the greatest extent while ensuring the noise reduction effect.

[0074] Preferably, the training process of the present invention comprises the following steps:

[0075] First, the model parameters are initialized according to the training sample size. We define the loss function as the sum of the basic denoising layer loss function, the noise layer loss function and the audible denoising signal layer loss function, that is:

[0076] L total =L P +L N +L A

[0077] Among them, P represents the basic noise reduction model 1; L P represents the loss function of the basic denoising model 1; L N represents the loss function of noise classification model 2; L A Represents the loss function of the audible noise reduction signal model 3.

[0078] Then, we input the sample data into the shared layer 4, input the clean data and the noisy data into the basic denoising model 1 respectively, calculate the loss functions of the two, and calculate the loss function of the shared layer 4 together with the loss function of the audible denoising signal layer. Then, using the back propagation algorithm, we use the shared layer 4 to update the network parameters of the basic denoising layer and the audible denoising layer, and complete the update of the shared layer 4, the basic denoising layer, and the audible denoising layer.

[0079] Secondly, the outputs of the basic noise reduction layer and the audible noise reduction layer are sent to the noise classification model 2, and the loss function of the noise classification model 2 is calculated to obtain the prediction probability of the noise classification model 2. The noise classification model loss function of the noise classification model 2 is calculated, and the back propagation algorithm is used to update the network parameters of the noise classification layer to complete the update of the noise classification layer.

[0080] We repeat the above steps until the termination condition is met. It is worth noting that the loss function used by the basic denoising model 1 and the noise classification model 2 is the cross entropy loss function, which is defined as follows:

[0081] L P =-∑(y k ·log(p k ))

[0082] L N =-∑(y i ·log(p i ))

[0083] Among them, y k represents the deviation value of the basic denoising model 1 on the time segment k; i represents the deviation value of noise classification model 2, N refers to the number of training samples, p k represents the predicted probability output by the basic denoising model 1, p i Represents the predicted probability output by the noise classification model 2.

[0084] Finally, we fine-tune the base denoising model1 on the test dataset.

[0085] In another embodiment of the present invention, the training process of the model further includes: the output of the basic noise reduction model 1 is simultaneously used as the forward propagation data input required for updating the network parameters of the noise layer and the audible noise reduction layer. This method can improve the synergy between the various components of the model, thereby obtaining a better overall noise reduction effect.

[0086] Preferably, the training of the basic noise reduction model 1 is used to process nonlinear noise, and it is necessary to distinguish whether the filter is linear or nonlinear. When training the network, we use the original data as training samples, and take a set of clean data plus specific nonlinear noise as the input of the model. During the training process, we add Gaussian noise to the speech in the clean data set, and the standard deviation of the Gaussian noise is randomly selected (0.01, 0.7); then, specific nonlinear noise is added to the specific speech data. This method enables the model to learn the ability to process various complex noises and improve its adaptability in practical applications.

[0087] The training method of the audible noise reduction output signal model 3 of the present invention is as follows:

[0088] First, we use the basic denoising model 1 with clean data as training samples for the audible denoising output model 3.

[0089] Secondly, the nonlinear noise is input into the basic noise reduction model 1, and the output of the basic noise reduction layer is obtained as the input of the audible noise reduction output model 3.

[0090] Then, we input the output data of the basic denoising model 1 and the clean data into the audible denoising output signal model 3. Next, the loss layer is calculated based on the error value between the output of the basic denoising model 1 and the clean data. We use the error of the basic denoising model 1 to calculate the gradient and then use the back propagation algorithm to update the network parameters.

[0091] We repeat the above steps until the termination condition is met. For the audible noise reduction signal model 3, we use the logarithmic loss function, which is defined as follows:

[0092] L A =-∑(y i ·log(p i )+(1-y i )·log(1-p i ))

[0093] Among them, y i represents the true label; p i Represents the probability predicted by the model.

[0094] Preferably, the testing process of the present invention comprises the following steps:

[0095] First, we preprocess the noisy speech information of the test input according to the basic noise reduction model 1.

[0096] Then, the preprocessed noisy speech is input into the basic noise reduction model 1 to obtain a noise reduction output signal.

[0097] Finally, in the test phase, we send the denoising results output by the basic denoising model 1 together with the original clean data in the clean sample to the audible denoising output signal model 3, calculate the confidence score, and evaluate the noise of the test set. We select samples with confidence scores greater than the threshold, consider them to be good samples, and input them into the basic denoising output model. The threshold here is set to 0.85, which is an empirical value based on a large number of experiments. It can ensure the denoising effect while retaining the audibility of the original speech to the greatest extent. Finally, we output the denoising results in the basic denoising model 1 as the result of the final denoising model.

[0098] In another embodiment of the present invention, during the training of the basic denoising model 1, the loss function updated by the shared layer 4 is:

[0099] L shared =L P +L A

[0100] Where L P Represents the loss of the basic noise reduction layer; L A Represents the loss of the audible noise reduction layer. The design of this loss function can take into account both the noise reduction effect and the audibility of speech, thus achieving a good balance between the two.

[0101] In summary, the multi-task based deep learning headphone noise reduction adaptive method proposed in the present invention achieves efficient and accurate headphone noise reduction effect by combining the three tasks of basic noise reduction, noise classification and audible noise reduction output signal. This method can not only effectively remove various types of noise, but also maintain the clarity and audibility of the original speech. By using the structure of shared layers and exclusive layers, the present invention realizes collaborative optimization between multiple tasks and improves the overall noise reduction performance. In addition, the present invention also introduces a confidence scoring mechanism, which can adaptively adjust the noise reduction effect to ensure the quality of the output audio. This method can significantly improve the user's auditory experience in practical applications, such as teleconferencing, voice assistants and other scenarios, and has broad application prospects.

[0102] In order to verify the superiority of the multi-task based deep learning headphone noise reduction adaptive method proposed in this invention, we designed a series of comprehensive experiments. These experiments not only cover traditional noise reduction performance indicators, but also consider aspects such as voice quality and computational efficiency to comprehensively evaluate the practical application value of this invention.

[0103] First, we selected three typical noise environments for testing: office background noise, street traffic noise, and cafe noise. These environments represent the most common noise interference scenarios in daily life and can effectively verify the performance of the present invention in practical applications.

[0104] We use the following key indicators to evaluate the performance of the present invention:

[0105] 1. Signal-to-Noise Ratio Improvement (SNRI): used to measure the improvement in signal quality after noise reduction.

[0106] 2. Perceptually evaluated speech quality (PESQ): evaluates the subjective quality of the speech after noise reduction.

[0107] 3. Short-time objective intelligibility (STOI): measures the intelligibility of speech after noise reduction.

[0108] 4. Computational complexity: evaluates the real-time performance of the algorithm.

[0109] 5. Adaptability: Test the model’s adaptability to different noise environments.

[0110] In order to comprehensively evaluate the performance of our invention, we compared it with two mainstream denoising methods: the traditional Wiener filtering method and a single-task deep learning denoising method.

[0111] The experimental results are shown in the following table:

[0112] index environment The present invention Wiener Filter Single-task deep learning SNRI(dB) office 12.5 8.2 10.7 street 11.8 7.5 9.9 Cafe 10.9 6.8 9.1 PESQ office 3.62 2.95 3.28 street 3.45 2.81 3.12 Cafe 3.31 2.73 3.05 STOI office 0.92 0.83 0.88 street 0.89 0.80 0.85 Cafe 0.87 0.78 0.83 Computational complexity (ms) All environments 15 5 18 Adaptability All environments excellent middle good

[0113] From these experimental results, we can draw the following important conclusions:

[0114] First, in terms of signal-to-noise ratio improvement (SNRI), our method significantly outperforms the other two methods in all test environments. In particular, in the complex coffee shop environment, our method can still maintain a high SNRI of 10.9 dB, which fully demonstrates the superiority of our multi-task learning framework in dealing with complex noise environments.

[0115] Secondly, the present invention also performs well in terms of perceptual evaluation of speech quality (PESQ) and short-time objective intelligibility (STOI), two key indicators for measuring speech quality. It is worth noting that in all environments, the PESQ scores of the present invention exceed 3.3, and the STOI values ​​are close to or exceed 0.9. This means that the speech processed by the present invention not only has noise effectively suppressed, but also maintains high clarity and intelligibility. This is precisely due to our audible noise reduction output signal model, which effectively retains the key features of the speech while reducing noise.

[0116] In terms of computational complexity, although the present invention is slightly inferior to the simple Wiener filter method, it is more efficient than the single-task deep learning method. The 15ms processing time fully meets the needs of real-time applications, which fully proves that our multi-task shared layer design can effectively control the computational complexity while improving performance.

[0117] Finally, in terms of the key indicator of adaptability, the present invention shows significant advantages, which is mainly due to our noise classification model and multi-task learning framework, which enable the system to quickly identify and adapt to different noise environments, thereby maintaining excellent noise reduction effects in various scenarios.

[0118] In summary, these experimental results strongly demonstrate the superiority of the present invention in many aspects such as noise reduction effect, voice quality maintenance and environmental adaptability. In particular, the outstanding performance in complex noise environments highlights the huge potential of the present invention in practical applications. We have reason to believe that this multi-task-based deep learning headphone noise reduction adaptive method will bring a qualitative leap in the user's voice communication experience in various noisy environments, and also open up a new direction for the development of intelligent audio processing technology.

[0119] Of course, we also recognize that there is still room for further optimization. For example, we can explore more complex network structures or introduce new learning strategies to further improve the performance of the system in extreme noise environments. In addition, how to further reduce the computational complexity while maintaining the existing performance and improve the applicability of the system on low-power devices is also an important direction for our future research.

[0120] In general, the present invention has brought new breakthroughs in headphone noise reduction technology. Its excellent performance and broad application prospects will undoubtedly inject strong impetus into the development of the intelligent audio field.

[0121] It should be noted that the above embodiments are only preferred implementations of the present invention and are not intended to limit the protection scope of the present invention. Those skilled in the art can make equivalent substitutions or changes based on the technical solutions and inventive concepts of the present invention, and these equivalent substitutions or changes should all be included in the protection scope of the present invention.

Claims

1. A multi-task based deep learning headphone noise reduction adaptive method, characterized by , including the following steps: Step 1, establishing an end-to-end multi-task deep learning headphone noise reduction model, taking the basic noise reduction model, the noise classification model and the audible noise reduction output signal model as three tasks, for processing the noise reduction of the headphone associated noise of voice calls; in the model, there are shared layers and exclusive layers between the three tasks; during training, the output of the shared layer is used as the input of the three tasks, the exclusive layer uses its own data set for training, and sends the training results to the shared layer, thereby adjusting the shared layer to achieve simultaneous optimization of the three tasks and ultimately output the noise reduction results; Step 2, selecting an audio sample, wherein the sample is a data set based on real noise and speech, and is selected from an existing clean data set; the audio sample is a data set with noise added, and the noise is generated according to the statistical characteristics of the noise required to be classified by the model, and the noise includes nonlinear noise and white noise; Step 3, using the audio samples obtained in step 2 to train the multi-task deep learning denoising model to be trained, inputting the noisy audio into the model, classifying it according to its original data, calculating the loss value of each task of the model in turn, and using the back propagation algorithm to update the model parameters to obtain the final trained multi-task deep learning denoising model; Step 4: After the training is completed, the audio data is input into the model, and the output end outputs the noise-reduced signal, wherein the audio data is noise-free data and contains the speech of the target conversation; Step 5, calculating the confidence score of the noise reduction signal output in step 4, wherein the confidence score is used to evaluate the audio quality.

2. According to claim 1, a multi-task based deep learning headphone noise reduction adaptive method is characterized in that The end-to-end multi-task deep learning headphone noise reduction model includes three sub-models: a basic noise reduction model, a noise classification model and an audible noise reduction output signal model; for the training stage: the input of the basic noise reduction model is a noisy signal, and the output is the estimated clean data; the input of the noise classification model is a noisy signal, and the output is the classification result of the noisy signal; the input of the audible noise reduction output signal model is the clean data and the noise reduction data output by the basic noise reduction model, and the output is audible data; For the test phase: the noisy data is input into the basic noise reduction model to obtain the basic noise reduction layer, and the output signal of the basic noise reduction layer is input into the audible noise reduction model together with the clean data to obtain the final noise reduction output signal; the basic noise reduction model uses a convolutional neural network, and the convolutional layer is composed of a convolutional layer, a rectified linear unit and a pooling layer in sequence. During training, the vector representation data of the noise is input into the convolutional neural network, and a filter model with acoustic functions is obtained through training, that is, the estimated clean data; the audible noise reduction output signal model is a feedforward neural network, which uses an attention mechanism and uses the basic noise reduction model and audible data to train an output nonlinear sound filter function in the audible noise reduction output signal model, and maintains the characteristics of the original speech signal; During the training phase, the denoised signal output by the basic denoising model is used as input together with the clean data, and the output is used as the final denoised data.

3. According to claim 2, a multi-task based deep learning headphone noise reduction adaptive method is characterized in that , the basic denoising model needs to distinguish whether the filter is linear or nonlinear; for this task, a clean output and noisy input signal of speech are generated from a clean data set containing 100 clean speech segments, that is, a data containing 100 speech segments in the clean data set is used, and Gaussian noise is first added, whose variance conforms to the normal distribution of (0, 0.3), and then the amplitude is adjusted, and the amplitude is (0.5, 1.5).

4. According to claim 2, a multi-task based deep learning headphone noise reduction adaptive method is characterized in that The noise classification model is used to train a method that can automatically analyze noise and classify noise; a convolutional neural network is used, and the convolutional layer is composed of a convolutional layer, a spatial, channel-level attention mechanism and a pooling layer in sequence, and includes a feature extractor for nonlinear mapping; during training, the vector representation data of the noise is input into the neural network, and finally a classifier that distinguishes various types of noise is trained; first, Gaussian noise is added to normal speech data, that is, the input signal is clean data plus Gaussian noise data; then specific nonlinear noise is added to different speech data, where the original clean data is synthesized using mathematical and statistical characteristics.

5. According to claim 1, a multi-task based deep learning headphone noise reduction adaptive method is characterized in that , the training process includes the following steps: Step S1, initialize the model parameters according to the training sample size; define the loss function, which is defined as the sum of the basic noise reduction layer loss function, the noise layer loss function and the audible noise reduction signal layer loss function, that is: L total =L P +L N +L A Among them, P represents the basic noise reduction model; L P Represents the loss function of the basic denoising model; L N represents the loss function of the noise classification model; L A represents the loss function of the audible noise reduction signal model; Step S2, input the sample data into the shared layer, input the clean data and the noisy data into the basic denoising model respectively, calculate the loss functions of the two, and calculate the loss function of the shared layer together with the loss function of the audible denoising signal layer; use the back propagation algorithm to update the network parameters of the basic denoising layer and the audible denoising layer using the shared layer, and complete the update of the shared layer, the basic denoising layer and the audible denoising layer; Step S3, the outputs of the basic noise reduction layer and the audible noise reduction layer are sent to the noise classification model, the noise classification model loss function is calculated, and the prediction probability of the noise classification model is obtained; the noise classification model loss function is calculated using the back propagation algorithm to update the network parameters of the noise classification layer, and the update of the noise classification layer is completed; Step S4, return to step 2 until the termination condition is met; Among them: the loss function used by the basic denoising model and the noise classification model is the cross entropy loss function, which is defined as: L P =-∑(y k ·log(p k )) L N =-∑(y i ·log(p i )) Among them, y k Represents the deviation value of the basic denoising model on the time segment of k, y i represents the deviation value of the noise classification model, N refers to the number of training samples; p k represents the predicted probability of the output of the basic denoising model; p i represents the predicted probability of the output of the noise classification model; Step S5, tuning the basic denoising model on the test data set.

6. According to claim 2, a multi-task based deep learning headphone noise reduction adaptive method is characterized in that ,The training process of the model also includes: the output of the basic noise reduction model is used as the forward propagation data input required for updating the network parameters of the noise layer and the audible noise reduction layer.

7. According to claim 2, a multi-task based deep learning headphone noise reduction adaptive method is characterized in that The training of the basic noise reduction model is used to process nonlinear noise, and it is necessary to distinguish whether the filter is linear or nonlinear; when training the network, the original data is used as the training sample, and a set of clean data plus specific nonlinear noise is used as the input of the model during training. During the training process, Gaussian noise is added to the speech in the clean data set, and the standard deviation of the Gaussian noise is randomly selected (0.01, 0.7); then, specific nonlinear noise is added to the specific speech data.

8. According to claim 5, a multi-task based deep learning headphone noise reduction adaptive method is characterized in that ,The training method of the audible noise reduction output signal model is: Step Sa, firstly, the basic noise reduction model and the clean data are used as training samples of the audible noise reduction output model; Step Sb, secondly, the nonlinear noise is input into the basic noise reduction model, and then the output of the basic noise reduction layer is obtained as the input of the audible noise reduction output model; Step Sc, input the output data of the basic denoising model and the clean data into the audible denoising output signal model, then calculate the loss layer according to the error value between the output of the basic denoising model and the clean data; use the error of the basic denoising model to calculate the gradient, and then use the back propagation algorithm to update the network parameters; Step Sd, return to step S2 until the termination condition is met; For the audible noise reduction signal model, the logarithmic loss function is used. The loss function used by the audible noise reduction output signal model is the logarithmic loss function, which is defined as: L A =-∑(y i ·log(p i )+(1-y i )·log(1-p i )) Among them, y i represents the true label; p i Represents the probability predicted by the model.

9. The method for adaptive headphone noise reduction based on deep learning based on multi-task according to claim 1, characterized in that , the test process includes the following steps: Step T1, testing the input noisy speech information and preprocessing the data according to the basic noise reduction model; Step T2, inputting the preprocessed noisy speech into the basic noise reduction model to obtain a noise reduction output signal; Step T3, in the test phase, the denoising results output by the basic denoising model and the original clean data in the clean sample are sent to the audible denoising output signal model, the confidence score is calculated, the noise of the test set is evaluated, and the samples with confidence scores greater than the threshold are considered to be good samples and input into the basic denoising output model; the threshold is 0.85; finally, the denoising result in the basic denoising model is output as the final denoising model.

10. The method for adaptive headphone noise reduction based on multi-task deep learning according to claim 7, characterized in that ,During the training process of the basic denoising model, the loss function of the shared layer update is: L shared =L P +L A Among them, L P Represents the loss of the basic noise reduction layer; L A Represents the loss of the audible noise reduction layer.

Images