Audio processing method and device, computer device, and program
By fusing two-level networks into a single model through stepwise training with depth clustering and mask estimation loss functions, the method addresses the computational complexity issue in speech enhancement, enhancing noise and reverberation removal efficiency.
Patent Information
- Application Number
- JP2024532312
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2022-05-07
- Filing Date
- 2023-03-31
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2043-03-31
AI Technical Summary
Existing speech enhancement technologies face challenges in efficiently reducing noise and reverberation in complex audio environments due to the high computational complexity of two-level networks, which are not suitable for practical applications.
A speech processing method that fuses two-level networks into a single model through stepwise training using depth clustering and mask estimation loss functions, reducing computational resources while effectively removing noise and reverberation.
This approach improves the performance of speech enhancement by efficiently removing noise and reverberation, optimizing the training process and reducing computational costs.
Smart Images

Figure 2024542658000001_ABST
Abstract
Description
[Technical field]
[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to a Chinese patent application bearing application number 202210495197.5, filed with the China Patent Office on May 7, 2022, the entire contents of which are incorporated herein by reference.
[0002] The present application relates to the field of voice recognition technology, and more particularly to a voice processing method and device, a storage medium, a computer device and a program product thereof. [Background technology]
[0003] The essence of speech enhancement is noise reduction of speech. In daily life, the speech collected by the microphone is usually "polluted" speech with different noises. The main purpose of speech enhancement is to restore the clean speech we want from these "polluted" noisy speeches, thereby effectively suppressing various interference signals and enhancing the target speech signal. This not only improves the quality of the speech, but also helps to improve the speech recognition performance.
[0004] The application fields of speech enhancement include video conferencing and speech recognition, etc., and it is a pre-processing module of many speech coding and recognition systems, and is usually classified into near-field speech enhancement and far-field speech enhancement. In a complex speech collection environment, noise and reverberation exist simultaneously, so existing speech enhancement uses a noise reduction and dereverberation method based on a two-level network, but due to the large computational complexity of this two-level network, speech enhancement cannot meet the performance requirements of practical applications. Summary of the Invention [Problem to be solved by the invention]
[0005] The embodiments of the present application aim to provide a voice processing method and an apparatus therefor, a storage medium, a computer device, and a program product, and to improve the performance of voice enhancement. [Means for solving the problem]
[0006] An embodiment of the present application provides a speech processing method, the method including: acquiring initial speech features of a speech speech; inputting the initial speech features into a pre-trained speech enhancement model to obtain target speech features output from the speech enhancement model, where the speech enhancement model is obtained by stepwise training based on a depth clustering loss function and a mask estimation loss function; and calculating a noise- and reverberant-removed target speech based on the target speech features.
[0007] An embodiment of the present application further provides a speech processing apparatus, the apparatus including: an acquisition module configured to acquire initial speech features of a speech speech; an enhancement module configured to input the initial speech features into a pre-trained speech enhancement model to obtain target speech features output from the speech enhancement model, the speech enhancement model being obtained by stepwise training based on a depth clustering loss function and a mask estimation loss function; and a computation model configured to calculate a noise- and reverberant-removed target speech based on the target speech features.
[0008] An embodiment of the present application further provides a computing device comprising a processor and a memory, the memory storing computer program instructions which, when invoked by the processor, execute the above audio processing method.
[0009] An embodiment of the present application further provides a computer-readable storage medium, in which program code is stored, the program code performing the above-mentioned audio processing method when executed by a processor.
[0010] An embodiment of the present application further provides a computer program product or a computer program, the computer program product or the computer program including computer instructions, the computer instructions being stored in a storage medium, a processor of a computing device reading the computer instructions from the storage medium, and the processor executing the instructions, thereby causing the computer to perform the steps of the above-mentioned audio processing method. Effect of the Invention
[0011] The embodiments of the present application gradually train a pre-configured speech enhancement model through two different loss functions, guide the model to efficiently remove noise and reverberation in speech features, enable noise reduction and dereverberation tasks, and achieve optimal training effects through a unique training process, thereby helping to improve the noise reduction and dereverberation capabilities of the speech enhancement model, and improving the performance of speech enhancement while reducing model computational resources. [Brief description of the drawings]
[0012] [Figure 1] FIG. 1 is a schematic diagram of a general noise reduction and dereverberation method according to an embodiment of the present application; [Diagram 2] FIG. 1 is a schematic diagram of the architecture of a speech processing system according to an embodiment of the present application. [Diagram 3] 1 is a flow chart of a speech processing method according to an embodiment of the present application; [Figure 4] FIG. 1 is a schematic diagram of an application scene of a voice processing method according to an embodiment of the present application. [Diagram 5] FIG. 2 is a schematic diagram of the architecture of a speech enhancement model according to an embodiment of the present application; [Figure 6] 4 is a flowchart of another audio processing method according to an embodiment of the present application. [Figure 7] 1 is a flow chart of audio feature extraction according to an embodiment of the present application; [Figure 8] FIG. 2 is a schematic diagram of the architecture of a given coordination network according to an embodiment of the present application; [Figure 9] FIG. 1 is a module block diagram of a voice processing device according to an embodiment of the present application. [Figure 10] FIG. 2 is a module block diagram of a computing device according to an embodiment of the present application. [Figure 11] FIG. 2 is a module block diagram of a computer-readable storage medium according to an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0013] In order to more clearly describe the technical solutions of the embodiments of the present application, the following briefly introduces the drawings used in the description of the embodiments. Obviously, the above drawings are only some embodiments of the present application, and those skilled in the art can also obtain other related drawings based on these drawings without any creative efforts.
[0014] In the following, the embodiments of the present application are described in detail. Examples of the embodiments are shown in the drawings, in which the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the drawings are exemplary and are intended only to explain the present application, and should not be construed as limiting the present application.
[0015] In order to make those skilled in the art better understand the technical solution of the present application, the following clearly and completely describes the technical solutions of the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, and not all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are included in the protection scope of the present application.
[0016] In daily life, the problem of performing voice communication under noise interference often occurs. For example, there are mobile phone use in cars and trains, ambient noise, and noisy far-end voice collected by microphones during multi-person video conferences, so it is necessary to use voice enhancement technology to extract as pure original voice as possible from noisy voice signals. Depending on the call scene, the call type made by a user using a client may include near-end call and far-end call, and for the participants of the call, the near-end is the position of the participant and the far-end is the position of the other participants in the remote conference. Each position has at least one microphone and one speaker. However, the near-end call of the client is only suitable for one person or a small number of close-distance calls, and the voice-video experience is general.
[0017] In order to improve user experience, the industry has been focusing on the research of far-end speech on large-screen communication devices. However, the far-end speech has a longer speech distance, a lower signal-to-noise ratio, and speech voice is usually accompanied by noise and reverberation, so it is necessary to use a high-performance far-field speech enhancement to perform noise reduction and dereverberation on the speech voice. The speech enhancement solutions in the related art usually adopt two models to perform noise reduction and dereverberation respectively. For noisy and reverberant speech, please refer to Figure 1. Figure 1 shows two commonly used technology solutions of noise reduction and dereverberation, including a method of dereverberation after noise reduction and a method of noise reduction after dereverberation.
[0018] For example, the microphone array is divided into different subsets, and each subset obtains the speech enhanced by each microphone through a first-level speech enhancement network, integrates the enhanced speech, and then obtains the final output through a second-level speech enhancement network. However, such a two-level network-based speech enhancement scheme requires a large amount of computation in the training process, which is not suitable for the performance requirements of the actual application of the product. Reducing the number of parameters of the network to reduce the amount of computation will result in a poor effect of the network in performing speech enhancement.
[0019] In order to solve the above problems, the applicant has researched and proposed a speech processing method provided by the embodiments of the present application, which can obtain initial speech features of a speech voice, input the initial speech features into a pre-trained speech enhancement model to obtain a target speech feature output from the speech enhancement model, and the speech enhancement model is obtained by stepwise training based on a depth clustering loss function and a mask estimation loss function. This merges two models (two-level networks) into one model to reduce the computational cost of the model training process. Based on the target speech features, calculate the target speech with noise and reverberation removed. In this way, model training is performed on the pre-set speech enhancement model through different loss functions, and the model is guided to efficiently remove noise and reverberation in the speech features, thereby improving the performance of speech enhancement while reducing the model computational resources.
[0020] First, the application scene of the voice processing method according to the present application will be described. Figure 2 is a schematic diagram of the architecture of a voice processing system. In some embodiments, the voice processing system 300 is applied to a remote video conference scene, and the voice processing system 300 may include a near-end client 310, a far-end client 330, and a server side 350. Here, the near-end client 310, the far-end client 330, and the server side 350 are communicatively connected via a network, and in one embodiment, the near-end client 310 and the far-end client 330 may be a large-screen terminal for video, and the server side 350 may be a cloud server.
[0021] Exemplarily, the far-end client 330 can collect initial speech with noise and reverberation emitted by participants and transmit the initial speech to the server side 350, and after receiving the initial speech, the server side 350 can use a pre-trained speech enhancement model to perform noise reduction and dereverberation on the initial speech to obtain an enhanced clean speech (target speech), and transmit the clean speech to the near-end client 310. In some embodiments, the speech enhancement model may be located in the near-end client 310 or the far-end client 330 according to the needs of actual application scenarios.
[0022] It should be noted that the above voice processing system 300 is merely an example, and the architecture and application scenarios of the voice processing system described in the embodiments of the present application are intended to more clearly explain the technical solutions of the embodiments of the present application, and are not intended to limit the technical solutions provided by the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of voice processing system architecture and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application can be similarly applied to similar technical problems.
[0023] Please refer to Fig. 3, which is a flowchart of a voice processing method according to an embodiment of the present application. In a specific embodiment, the voice processing method is applied to a voice processing device 500 shown in Fig. 9 and a computer device 600 (Fig. 10) in which the voice processing device 500 is arranged.
[0024] Taking a computer device as an example, the specific process of the embodiment of the present application will be described. Of course, the computer device applied to the embodiment of the present application may be a server or a terminal, etc. The server may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, big data and artificial intelligence platforms. The terminal may be, but is not limited to, a smartphone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc.
[0025] In the following, the process shown in FIG. 3 will be described in detail with reference to the application scenario shown in FIG. 4. FIG. 4 is a schematic diagram of an application scenario of the voice processing method according to the embodiment of the present application. In the application scenario, the voice processing method can be applied to a specific voice enhancement system, and the voice enhancement model 411 of the voice enhancement system can be deployed in a cloud server 410, and the cloud server 410 can be communicatively connected to the conference terminals (the first conference terminal 430 and the second conference terminal 450) of two venues, respectively. Here, the first conference terminal 430 and the second conference terminal 450 collect the voices (i.e., original call voices) of the participants in the respective venues, upload the collected voices to the cloud server 410, and the cloud server 410 completes the voice enhancement of the voices to obtain clean voices, and finally, the cloud server 410 sends the clean voices to the corresponding conference terminals for playback. The voice processing method may specifically include the following steps.
[0026] In step S110, the initial voice features of the call voice are obtained.
[0027] In an embodiment of the present application, a computer device can obtain initial speech features of a call voice that requires speech enhancement, where the initial speech features are acoustic features obtained based on a transformation of the call voice, such as, but not limited to, a logarithmic power spectrum (LPS) or mel-frequency cepstral coefficients (MFCC).
[0028] Unlike image data, voice data often cannot be directly input into a model, and there is no obvious feature change in the long-term domain, so it is difficult to learn the features of voice data. The time-domain data of voice usually consists of 16K sampling rate, that is, 16,000 sampling points per second. If the time-domain sampling points are directly input, the amount of training data is too large, making it difficult to train a practical effect. Therefore, in voice processing related tasks, voice data is usually converted into acoustic features to be used as the input or output of the model.
[0029] In one embodiment, after acquiring the call voice, frame division processing and windowing processing are performed on the call voice to obtain the initial voice features. For example, frame division processing and windowing processing are sequentially performed on all the call voices collected by the microphone to obtain the voice signal frames of the call voice, Fast Fourier Transformation (FFT) is performed on the voice signal frames to obtain the FFTed discrete power spectrum, and then logarithmic calculation is performed on the acquired discrete power spectrum to obtain the logarithmic power spectrum as the initial voice features. By performing frame division processing and windowing processing on the call voice, the call voice can be converted from a non-stationary signal in the time domain space to a stationary signal in the frequency domain space, making it easier to train the model.
[0030] The purpose of speech signal frame division is to divide several speech sampling points into one frame, within which the characteristics of the speech signal are considered stable. Usually, the length of one frame should be short enough to ensure that the intra signal is stable, so the length of one frame should be smaller than the length of one phoneme, and at normal sound speed, the duration of one phoneme is about 50 ms. In addition, to perform Fourier analysis, one frame must contain a sufficient vibration period, which is around 100 Hz for male voices and around 200 Hz for female voices, which is 10 ms and 5 ms in terms of periods. Therefore, the length of speech frame division is generally 10 to 40 ms.
[0031] After frame division, there will be discontinuity at the beginning and end of each frame, so the more frames are divided, the greater the error from the original signal. To solve this problem, windowing is used to make the framed signal continuous and each frame expresses the characteristics of a periodic function. For example, available window functions include rectangular window, Hamming window, and Hanning window.
[0032] In the video conference scene shown in Fig. 4, since there is a certain distance between the participants and the conference terminal, noise and reverberation occur in the participant's voice collected by the conference terminal. Therefore, the voice processing method provided by the embodiment of the present application can be used to perform voice enhancement processing on the participant's voice, thereby removing the noise and reverberation in the voice.
[0033] For example, the second conference terminal 450 collects the voice of the participant 420 in the venue, i.e., the call voice, through a microphone, and transmits the call voice to the cloud server 410 via a network. Then, after receiving the call voice, the cloud server 410 performs frame division processing, windowing processing, and Fourier transform on the call voice to obtain initial voice features.
[0034] In step S120, the initial speech features are input into a pre-trained speech enhancement model to obtain target speech features output from the speech enhancement model.
[0035] In actual application scenarios, the speech voice collected by the microphone array contains noise and reverberation at the same time, and considering a two-level network for noise reduction and dereverberation of speech voice, the parameter amounts of the two networks are large during training, so a large amount of computational resources must be consumed, and reducing the parameter amount of each network will also reduce the noise reduction and dereverberation performance of the model. Therefore, the two-level network can be fused into one network, and the parameter amount of the fused model is smaller than the parameter amount of the two networks, which can greatly reduce the computational amount of the training process and improve the speech enhancement performance of the model.
[0036] In the embodiment of the present application, the speech enhancement model can generate a target speech feature corresponding to a speech voice based on the input initial speech feature, i.e., a clean speech feature with noise and reverberation removed after speech enhancement. Referring to Figure 5, Figure 5 is a schematic diagram of the architecture of a speech enhancement model. The speech enhancement model may include a plurality of hidden layers, a depth clustering layer, a speech mask estimation layer, and a noise mask estimation layer.
[0037] Here, the depth clustering layer, the speech mask estimation layer and the noise mask estimation layer can be linear layers, and the inputs of these three layers are all from the output of the hidden layer. The hidden layer can calculate intermediate features based on the input initial speech features, and the intermediate features are intermediate values of the speech enhancement process.
[0038] For example, the depth clustering layer can be realized by normalization and tangent function (denoted as tanh), where the output of the hidden layer is first normalized to restrict the output of the hidden layer within a certain range, e.g., [0,1] or [-1,1], to facilitate subsequent processing, and then the tangent function value is calculated for the normalized result as the output of the depth clustering layer.
[0039] For example, the speech mask estimation layer and the noise mask estimation layer can both be realized by a softmax function.
[0040] The speech mask estimation layer can perform mask inference (MI) based on the intermediate features to obtain target speech features with noise and reverberation removed, the noise mask estimation layer can perform mask inference based on the intermediate features to obtain speech features with noise, and the depth clustering layer can perform deep clustering (DC) based on the acquired intermediate features to assist the speech mask estimation layer and the noise mask estimation layer in noise reduction and dereverberation. For example, the hidden layer can be a long short-term memory network (LSTM) or a variant such as a bidirectional long-short-term memory network (Bi-LSTM), because the speech features have a time series with short-term stability, which is consistent with the long short-term memory ability of LSTM. The hidden layer can also be other networks with memory-like properties, such as a gated recurrent unit (GRU).
[0041] In one embodiment, in the model training process, the model can be trained stepwise by a depth clustering loss function corresponding to the depth clustering layer and a mask estimation loss function corresponding to the speech mask estimation layer and the noise mask estimation layer, respectively. Illustratively, in a first step, a noise reduction model can be trained based on the depth clustering loss function and the mask estimation loss function, and the training is stopped when the noise reduction model converges. Here, the mask estimation loss function corresponding to the speech mask estimation layer uses the clean speech label without noise and with reverberation. In a second step, a dereverberation model is trained, and the noise reduction model trained in the first step is used as the dereverberation model, and the dereverberation model is trained based on the depth clustering loss function and the mask estimation loss function, and the training is stopped when the dereverberation model converges. Here, the mask estimation loss function corresponding to the speech mask estimation layer uses the clean speech label without noise and without reverberation, so that the finally obtained dereverberation model, i.e., the speech enhancement model, has the ability to perform noise reduction and dereverberation simultaneously.
[0042] It should be noted that the depth clustering layer of the speech enhancement model is a binary loss based on time-frequency point clustering, and due to the regularization characteristics of the depth clustering loss, in the training process of the related art, it is difficult to guide the speech mask estimation layer and the noise mask estimation layer to effectively remove noise and reverberation in the speech, and further, it is difficult to effectively improve the speech enhancement performance of the model. Meanwhile, through the step-by-step training method of the embodiment of the present application, the noise reduction task and the dereverberation task can achieve the optimal training effect in a unique training process, which helps to improve the noise reduction and dereverberation capabilities of the speech enhancement model.
[0043] As a result, the speech enhancement model obtained by the above training can obtain intermediate features by multi-layer LSTM, and the speech mask estimation layer can perform mask estimation based on the intermediate features to calculate the speech mask, i.e., the target speech feature. Illustratively, in the video conference scene shown in FIG. 4, the cloud server 410 can obtain the initial speech feature, and then input the initial speech feature into the speech enhancement model 411, and the speech mask estimation layer of the speech enhancement model 411 can perform mask estimation based on the intermediate features to calculate the speech mask, i.e., the target speech feature, and the intermediate feature is obtained by the multi-layer LSTM. In the application scene of speech enhancement, it is only necessary to restore the speech using the target speech feature output from the speech mask estimation, so that the amount of calculation in the speech enhancement process can be effectively reduced.
[0044] In step S130, a denoised and dereverberated target speech is calculated based on the target speech features.
[0045] In one embodiment, the target speech features obtained can be subjected to inverse feature transformation to calculate a target speech from which noise and reverberation have been removed. For example, the target speech features can be subjected to an inverse Fourier transform (IFT) to transform the target speech features from the frequency domain to the time domain, thereby obtaining a time domain speech after speech enhancement, i.e., a target speech. Exemplarily, in the video conference scene shown in FIG. 4, the cloud server 410 can obtain the target speech features output from the speech enhancement model 411, and then convert the target speech features, i.e., clean speech features, into a target speech by inverse Fourier transform, thereby obtaining a clean speech from which noise and reverberation have been removed. The cloud server 410 can transmit the clean speech to the first conference terminal 430, and play the noise- and reverberation-free speech of the participant 420 from the speaker of the first conference terminal 430.
[0046] In the embodiment of the present application, the initial speech features of the speech speech are obtained, and the initial speech features are input into a pre-trained speech enhancement model to obtain a target speech feature output from the speech enhancement model, and the speech enhancement model is obtained by stepwise training based on a depth clustering loss function and a mask estimation loss function, and a target speech with noise and reverberation removed can be calculated based on the target speech features. In this way, model training is performed on the pre-set speech enhancement model through different loss functions, and the model is guided to efficiently remove noise and reverberation in the speech features, so that the performance of speech enhancement can be improved while reducing the model computational resources.
[0047] The methods described in the above examples will be explained in more detail below with reference to the examples.
[0048] In the embodiment of the present application, a case where the voice processing device is specifically incorporated in a computer device will be described as an example.
[0049] Please refer to Fig. 6, which shows another speech processing method according to an embodiment of the present application, and in a specific embodiment, the speech processing method is applied to a predetermined enhancement network shown in Fig. 8. In the following, the process shown in Fig. 5 will be described in detail.
[0050] The embodiments of the present application incorporate artificial intelligence (AI). Artificial intelligence technology is the theory, method, technique, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use the knowledge to obtain optimal results. In other words, artificial intelligence is a comprehensive technology of computer science that seeks to understand the substance of intelligence and produce new intelligent machines that can react in a manner similar to human intelligence. Artificial intelligence studies the design principles and realization methods of various intelligent machines so that machines have the functions of perception, reasoning, and decision-making.
[0051] Artificial intelligence technology is a comprehensive discipline that includes a wide range of fields, including both hardware and software technology. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, and other technologies. Artificial intelligence software technology mainly includes several major directions, such as computer vision technology, voice processing technology, natural language processing technology, and machine learning / deep learning.
[0052] The technical solutions provided by the embodiments of the present application relate to technologies such as artificial intelligence speech technology, and the core technologies of speech technology include automatic speech recognition technology (ASR: Automatic Speech Recognition), speech synthesis technology (TTS: Text To Speech), and voiceprint recognition technology (VPR: Voiceprint Recognition). Enabling computers to hear, see, speak, and feel is the development direction of future human-machine interaction, among which voice has become one of the most popular human-machine interaction methods in the future.
[0053] The following will be described in detail with reference to the process shown in Figure 6 and the network architecture diagram shown in Figure 8. The voice processing method may specifically include the following steps:
[0054] In step S210, the computing device obtains a training sample set.
[0055] The voice processing method provided by the embodiment of the present application includes training a predetermined enhancement network, and it should be noted that the training of the predetermined enhancement network may be performed in advance based on an acquired training sample dataset, and then, every time voice enhancement needs to be performed on the initial voice features of the call voice, the voice enhancement model obtained by the training can be used to calculate target voice features with noise and reverberation removed, and there is no need to re-train the predetermined enhancement network every time voice enhancement is performed.
[0056] In some embodiments, the training sample set can be determined using the wsj0-2mix (Wall Street Journal) dataset, which includes a 30-hour speech training set and a 10-hour speech training set. The noisy and reverberant speech used for network training can be generated by randomly selecting speech from different speakers from the corresponding sets and mixing them with random relative signal-to-noise ratios (SNRs) between 0 dB and 10 dB.
[0057] In one embodiment, the step of the computing device obtaining a training sample set may include the following steps: (1) The computing device acquires a first audio sample. (2) The computing device performs speech feature extraction on the first sample speech to obtain a noisy speech feature. (3) The computing device obtains a second sample audio. (4) The computing device performs speech feature extraction on the second sample speech to obtain a first clean speech label and a second clean speech label. (5) The computing device determines depth clustering annotations based on the first sample audio and the second sample audio.
[0058] Here, the first sample sound is a sound including noise and reverberation collected based on a microphone, the second sample sound is a clean sound without noise and with reverberation, and the clean sound without noise and without reverberation, and the depth clustering annotation is the proportion of features of the first sample sound and the second sample sound at each time-frequency point.
[0059] For example, the computer device can directly collect the call voice including noise and reverberation by the microphone, for example, in a video conference, the speech of the participant collected by the microphone of the large screen conference terminal is taken as the first sample voice. In the actual training process, the engineer can directly obtain the first sample voice from the noise reduction training corpus that has already been built.
[0060] The computer device can perform voice feature extraction on the acquired first sample voice. Referring to FIG. 7, FIG. 7 is a flowchart of voice feature extraction, in which a call voice including noise and reverberation, i.e., a first sample voice, is collected by a microphone.
number
number
number
number
number
number
number
number
number
number
number
[0061] The computer device can obtain a reference clean speech from the noise reduction training corpus, and the clean speech can be the second sample speech. To facilitate stepwise training for a given enhancement network, a clean speech with no noise and reverberation and a clean speech with no noise and no reverberation can be obtained, and then speech feature extraction is performed on the clean speech with no noise and reverberation to obtain a first clean speech label, and speech feature extraction is performed on the clean speech with no noise and no reverberation to obtain a second clean speech label. In the calculation process, the noise speech label
number
number
number
[0062] In one embodiment, the computing device performs depth clustering annotation by comparing audio energy of the first sample audio and the second sample audio at each time-frequency point.
number
[0063] In step S220, the computing device obtains a predetermined enhancement network.
[0064] When a product related to speech enhancement technology is put into industrial use, in consideration of the very strict requirements for delay, i.e., real-time performance, it is necessary to reduce the amount of parameters of the speech enhancement model as much as possible, but as a result, the speech enhancement effect of the model is greatly reduced. Therefore, in the embodiment of the present application, it is proposed to fuse a two-level network into one network, so that the speech enhancement model can simultaneously perform noise reduction and dereverberation, and still improve the effect of speech enhancement without reducing the amount of parameters of the model.
[0065] Referring to FIG. 8, FIG. 8 is a schematic diagram of the architecture of a given enhancement network. The given enhancement network includes a hidden layer, a deep clustering layer, and a mask estimation layer. The given enhancement network is a network with lower weights shared and multiple outputs, where the deep clustering layer can assist the mask estimation of the voice mask estimation layer and the noise mask estimation layer, so that the voice mask estimation layer and the noise mask estimation layer can effectively distinguish noise and reverberation in the voice in the process of network training, and the hidden layer can utilize LSTM or Bi-LSTM. The hidden layer shown in FIG. 8 is LSTM, and the mask estimation layer includes a voice mask layer (Clean-MI) and a noise mask layer (Noise-MI).
[0066] The speech mask estimation layer can calculate a speech mask, i.e., a clean speech label, and the noise mask estimation layer can calculate a noise and reverberation mask, i.e., a noisy speech label. In addition, since the application process only needs to restore the speech using the mask output from the speech mask estimation, the computational complexity of the speech enhancement process is not increased, and the speech enhancement efficiency is improved.
[0067] In step S230, the computing device performs denoising training and dereverberation training on the predetermined enhancement network step by step through a training sample set until the predetermined enhancement network satisfies a predetermined condition, and obtains a trained target enhancement network as a speech enhancement model.
[0068] The target enhancement network obtained after the training is completed, i.e., the speech enhancement model, needs to simultaneously perform two enhancement tasks, namely, noise reduction and dereverberation, and if the two enhancement tasks are trained simultaneously, the training of a given enhancement network cannot achieve the optimal training effect. Therefore, a stepwise training method can be adopted to separately train the training processes of the two tasks.
[0069] Specifically, the embodiment of the present application provides two stepwise training methods, for example, denoising training can be performed first and then dereverberation training, or denoising training can be performed first, where the purpose of denoising training is to equip the network with the ability of noise reduction, and the purpose of dereverberation training is to equip the network with the ability of dereverberation, so that the two enhancement tasks can achieve optimal training effect in their own training process, and thus the performance of speech enhancement of the speech enhancement model can be improved.
[0070] In some embodiments, the step of the computing device incrementally performing denoising and dereverberation training on the predetermined enhancement network through a training sample set until the predetermined enhancement network satisfies a predetermined condition may include the steps of: (1) The computer device inputs the noisy speech features into a hidden layer and generates intermediate training features through the hidden layer. (2) The computer device inputs the intermediate training features into a deep clustering layer, and generates clustering training annotations through the deep clustering layer. (3) The computing device inputs the intermediate training features into a speech mask estimation layer, and generates clean speech training features through the speech mask estimation layer. (4) The computing device inputs the intermediate training features into a noise mask estimation layer, and generates noisy speech training features through the noise mask estimation layer. (5) The computer device constructs a target loss function based on the clean speech label, the noisy speech label, the depth clustering annotations, the clean speech training features, the noisy speech training features, and the clustering training annotations, and incrementally performs denoising training and dereverberation training on the given enhancement network based on the target loss function until the given enhancement network satisfies a given condition.
[0071] Here, the intermediate training feature is an intermediate value generated from a hidden layer of a given enhancement network, and is input to the depth clustering layer, the voice mask estimation layer and the noise mask estimation layer as a shared value, respectively, to achieve lower weight sharing and reduce the amount of parameters of the network. The voice mask estimation layer and the noise mask estimation layer are based on the intermediate training feature to generate clean voice training feature
number
number
number
[0072] In one embodiment, the computer device constructs a target loss function based on the clean speech label, the noisy speech label, the depth clustering annotations, the clean speech training features, the noisy speech training features, and the clustering training annotations, and the step of incrementally performing denoising training and dereverberation training on a given enhancement network based on the target loss function until the given enhancement network satisfies a predetermined condition may include the following steps:
[0073] (5.1) The computing device determines a first loss function based on the clustering training annotations and the deep clustering annotations.
[0074] Here, the first loss function is a depth clustering loss function. Exemplarily, the first loss function
number
number
number
[0075] (5.2) The computing device determines a second loss function based on the clean speech training features and the clean speech labels.
[0076] For the two stepwise training methods, two different second loss functions can be determined based on different clean speech labels.
[0077] In some embodiments, the computing device includes a clean voice training feature.
number
number
number
number
[0078] In some embodiments, the computing device includes a clean voice training feature.
number
number
number
number
[0079] (5.3) The computing device determines a third loss function based on the noisy speech training features and the noisy speech labels.
[0080] For example, the third loss function
number
number
number
[0081] Here, the second loss function
number
number
[0082] (5.4) The computer device constructs a target loss function for the predetermined enhancement network based on the first loss function, the second loss function, and the third loss function, and gradually performs denoising training and dereverberation training on the predetermined enhancement network based on the target loss function until the predetermined enhancement network satisfies a predetermined condition.
[0083] Exemplarily, the computer device calculates a first loss function
number
number
number
number
[0084]
number
[0085] Where:
number
number
[0086] Generally, noise refers to "unwanted sounds" under certain circumstances, such as loud human voices and various sudden sounds. Reverberation refers to the acoustic continuity phenomenon that still exists after the sound source in the room stops emitting. Considering that different application scenarios have different needs for speech enhancement, for example, in a multi-person venue, noise in the speech collected by the conference terminal is mainly removed, and in a professional recording location, reverberation in the speech collected by the recording equipment is mainly removed, different methods of step-by-step training can be carried out based on the actual scenario used for the final speech enhancement model.
[0087] In some embodiments, an application scene attribute can be obtained based on the actual scene used for the final speech enhancement model, and a corresponding distributed training policy can be determined based on the application scene attribute. A target loss function for a given enhancement network is constructed based on the first loss function, the second loss function, and the third loss function based on the distributed training policy, and denoising training and dereverberation training are stepwise performed on the given enhancement network based on the target loss function until the given enhancement network satisfies a predetermined condition.
[0088] Here, the applied scene attributes are used to represent an actual scene to which the speech enhancement model is applied, such as scene attributes with emphasis on noise reduction and scene attributes with emphasis on dereverberation. The distributed training policy includes a first distributed training policy and a second distributed training policy, and the first distributed training policy is used for a scene with emphasis on noise reduction to perform denoising training first and then dereverberation training. The second distributed training policy is used for a scene with emphasis on dereverberation to perform denoising training first and then dereverberation training.
[0089] In one embodiment, in an application scene for the purpose of noise removal, for example, in a video conference with multiple participants, in addition to the voice from the speaker collected by the conference terminal, the voice of other speakers is also included, and noise reduction processing needs to be performed on the call voice collected by the conference terminal, so that noise removal training can be performed before dereverberation training. The computer device can determine a target loss function of the predetermined enhancement network based on a first loss function, a second loss function, and a third loss function based on a first distributed training policy, and the second loss function is determined by the noise removal loss function. Next, the noise removal training is repeated for the predetermined enhancement network based on the target loss function until the predetermined enhancement network satisfies a predetermined condition, and a noise removal network is obtained, and the noise removal network only plays the role of noise reduction.
[0090] In some embodiments, the computer device can determine a target loss function of the denoising network based on the first loss function, the second loss function, and the third loss function, and the second loss function is determined by the dereverberation loss function. Then, the denoising training is repeated for the dereverberation network based on the target loss function until the denoising network satisfies a predetermined condition. In this way, by performing the separate denoising training first, the interference of the reverberation factor in the training process can be avoided, so that the generated target enhancement network has better noise reduction performance.
[0091] In another embodiment, in application scenes aimed at reverberation, for example, in a recording studio, the requirements for sound quality are high and it is particularly important to remove unnecessary reverberation, so dereverberation training can be performed first, followed by noise removal training. The computer device can determine a target loss function for a given enhancement network based on a first loss function, a second loss function, and a third loss function based on a second distributed training policy, and the second loss function is determined by the dereverberation loss function. Then, dereverberation training is repeated for the given enhancement network based on the target loss function until the given enhancement network meets a predetermined condition, to obtain a dereverberation network, and the dereverberation network only plays the role of dereverberation.
[0092] In some embodiments, the computer device can determine a target loss function of the dereverberation network based on a first loss function, a second loss function, and a third loss function, and the second loss function is determined by the denoising loss function. Then, the denoising training is repeated for the dereverberation network based on the target loss function until the dereverberation network satisfies a predetermined condition. In this way, by performing the separate dereverberation training first, the interference of noise factors in the training process can be avoided, so that the generated target enhancement network has better dereverberation performance.
[0093] For example, when noise is precisely defined, the concept of noise essentially includes reverberation, so that when there are no special requirements for the application scenario of the speech enhancement model, a given enhancement network can be trained for denoising first, and then trained for dereverberation, so that the ability of dereverberation can be learned on the basis of a good noise reduction network. In this way, both of the two training processes can achieve optimal training effect, and the speech enhancement performance of the speech enhancement model can be improved.
[0094] The predetermined condition may be that the total loss value of the target loss function is equal to or less than a predetermined value, that the total loss value of the target loss function does not change, or that the number of training times reaches a predetermined number. For example, an optimizer can be employed to optimize the target loss function, and the learning rate, the batch size during training, and the training period (epoch) can be set based on experimental experience.
[0095] It can be understood that, after multiple cycles of repeated training for the network to be trained (a given enhancement network / denoising network / dereverberation network) based on the training sample data set, where each cycle includes multiple repeated training, and by continuing to optimize the parameters of the network to be trained, the above total loss value will become smaller, and eventually become smaller to a fixed value or smaller than the above given value, in this case, it represents that the network to be trained has converged. Of course, it can also be determined that the given enhancement network / denoising network / dereverberation network has converged after the training times reach a given number.
[0096] In training a given enhancement network by multi-task learning, the network is trained using a combination of depth clustering loss and mask estimation loss, but the mask estimation loss is only used in the verification process of the target enhancement network, i.e., the speech enhancement model selection, and when the speech enhancement model is executed, the output of the mask estimation branch is used as the mask after speech enhancement, i.e., the target speech feature.
[0097] In step S240, the computing device obtains initial voice characteristics of the speech voice.
[0098] In step S250, the computing device inputs the initial speech features into a hidden layer, and generates intermediate features through the hidden layer.
[0099] In step S260, the computing device inputs the intermediate features into a speech mask estimation layer, generates clean speech features through the speech mask estimation layer, and sets the clean speech features as target speech features.
[0100] In one embodiment, after collecting a speech voice, the computer device can perform speech feature extraction on the speech voice, including frame division processing, windowing processing and Fourier transform, to obtain initial speech features; the computer device can input the initial speech features to a hidden layer of a speech enhancement network and generate intermediate features through the hidden layer; the computer device can input the intermediate features to a speech mask estimation layer and generate clean speech features through the speech mask estimation layer, and the clean speech features can be used as target speech features.
[0101] In step S270, the computing device performs feature inverse transformation on the target speech features to calculate a target speech with noise and reverberation removed.
[0102] In one embodiment, the computer device can obtain the target speech features, and then perform feature inverse transformation on the target speech features to transform the target speech features (mask) in the frequency domain space into the target speech in the time domain space. In some embodiments, the feature inverse transformation can be an inverse Fourier transform. In the embodiments of the present application, a training sample set is obtained, a predetermined enhancement network is obtained, and noise reduction training and dereverberation training are stepwise performed on the predetermined enhancement network through the training sample set until the predetermined enhancement network meets a predetermined condition, to obtain a trained target enhancement network as a speech enhancement model, input the initial speech features into a hidden layer, generate intermediate features through the hidden layer, input the intermediate features into a speech mask estimation layer, generate clean speech features through the speech mask estimation layer, use the clean speech features as the target speech features, perform feature inverse transformation on the target speech features, and calculate the target speech from which noise and reverberation have been removed. In this way, the increase in the amount of calculation in the speech enhancement process can be avoided and the speech enhancement efficiency can be improved by simply restoring the speech using the target speech features output from the speech mask estimation of the speech enhancement model.
[0103] 9, which is a module block diagram of a voice processing device 500 according to an embodiment of the present application. The voice processing device 500 includes an acquisition module 510, an enhancement module 520, and a calculation model 530, where the acquisition module 510 is configured to acquire initial voice features of a speech voice, the enhancement module 520 is configured to input the initial voice features into a pre-trained voice enhancement model to obtain target voice features output from the voice enhancement model, the voice enhancement model is obtained by stepwise training based on a depth clustering loss function and a mask estimation loss function, and the calculation model 530 is configured to calculate a target voice with noise and reverberation removed based on the target voice features.
[0104] In some embodiments, the speech processing device 500 may further include a sample acquisition module, a network acquisition module, and a model training module. The sample acquisition module is configured to acquire a training sample set, the training sample set including a noisy speech feature, a clean speech label, a noisy speech label, and a depth clustering annotation, the network acquisition module is configured to acquire a predetermined enhancement network, the predetermined enhancement network including a hidden layer, a depth clustering layer, and a mask estimation layer, and the network training module is configured to perform denoising training and dereverberation training on the predetermined enhancement network step by step through the training sample set until the predetermined enhancement network satisfies a predetermined condition, and obtain a trained target enhancement network as a speech enhancement model.
[0105] In some embodiments, the mask estimation layer includes a speech mask estimation layer and a noise mask estimation layer, and the network training module may include a concealment unit, a deep clustering unit, a speech estimation unit, a noise estimation unit, and a network training unit, wherein the concealment unit is configured to input noise speech features to a hidden layer and generate intermediate training features through the hidden layer, the deep clustering unit is configured to input the intermediate training features to a deep clustering layer and generate clustering training annotations through the deep clustering layer, and the speech estimation unit is configured to input the intermediate training features to the speech mask estimation layer and generate clustering training annotations through the deep clustering layer. and generating clean speech training features through a speech mask estimation layer, the noise estimation unit is configured to input the intermediate training features to a noise mask estimation layer and generate noisy speech training features through the noise mask estimation layer, and the network training unit is configured to construct a target loss function based on the clean speech labels, the noisy speech labels, the depth clustering annotations, the clean speech training features, the noisy speech training features, and the clustering training annotations, and to perform denoising training and dereverberation training on a predetermined enhancement network in a stepwise manner based on the target loss function until the predetermined enhancement network satisfies a predetermined condition.
[0106] In some embodiments, the network training unit comprises a first subunit, a second subunit, a third subunit, and a training subunit, where the first subunit is configured to determine a first loss function based on clustering training annotations and depth clustering annotations, the second subunit is configured to determine a second loss function based on clean speech training features and a clean speech label, and the third subunit is configured to determine a third loss function based on noisy speech training features and a noisy speech label, and the training subunit is configured to construct a target loss function for a given enhancement network based on the first loss function, the second loss function, and the third loss function, and to perform denoising training and dereverberation training on the given enhancement network stepwise based on the target loss function until the given enhancement network satisfies a given condition.
[0107] In some embodiments, the second subunit may be specifically configured to determine a denoising loss function based on the clean speech training features and the first clean speech label, the denoising loss function being the second loss function, where the first clean speech label is a speech label obtained based on the noiseless and reverberant speech.
[0108] In some embodiments, the second subunit may be further specifically configured to determine a dereverberation loss function based on the clean speech training features and a second clean speech label, the dereverberation loss function being the second loss function, where the second clean speech label is a speech label obtained based on noise-free and dereverberated speech.
[0109] In some embodiments, the training subunit may be specifically configured to perform the steps of: determining a target loss function for the given enhancement network based on the first loss function, the second loss function, and the third loss function; repeating denoising training on the given enhancement network based on the target loss function until the given enhancement network satisfies a predetermined condition, to obtain a denoising network, where the second loss function is determined by the denoising loss function; and determining a target loss function for the denoising network based on the first loss function, the dereverberation loss function, and the third loss function; and repeating denoising training on the denoising network based on the target loss function until the denoising network satisfies the predetermined condition, where the second loss function is determined by the dereverberation loss function.
[0110] In some embodiments, the training subunit may be specifically configured to perform the steps of: determining a target loss function for the given enhancement network based on the first loss function, the second loss function, and the third loss function; repeating dereverberation training on the given enhancement network based on the target loss function until the given enhancement network satisfies a predetermined condition, to obtain a dereverberation network, where the second loss function is determined by the dereverberation loss function; and determining a target loss function for the dereverberation network based on the first loss function, the second loss function, and the third loss function; and repeating denoising training on the dereverberation network based on the target loss function until the dereverberation network satisfies the predetermined condition, where the second loss function is determined by the denoising loss function.
[0111] In some embodiments, the sample acquisition module may be specifically configured to perform the steps of acquiring a first sample audio, where the first sample audio is a noisy audio collected based on a microphone; performing audio feature extraction on the first sample audio to obtain noisy audio features; acquiring a second sample audio, where the second sample audio includes a noise-free and reverberated clean audio and a noise-free and non-reverberated clean audio; performing audio feature extraction on the second sample audio to obtain a first clean audio label and a second clean audio label; and determining depth clustering annotations based on the first sample audio and the second sample audio.
[0112] In some embodiments, the speech enhancement model includes a hidden layer, a depth clustering layer, a speech mask estimation layer, and a noise mask estimation layer, and the enhancement module 520 may be specifically configured to input initial speech features into the hidden layer, generate intermediate features through the hidden layer, input the intermediate features into the speech mask estimation layer, generate clean speech features through the speech mask estimation layer, and make the clean speech features the target speech features; The computational model 530 may be specifically configured to perform feature inversion on the target speech features to compute a target speech with noise and reverberation removed.
[0113] Those skilled in the art can clearly understand that for convenience and conciseness of description, the specific operation processes of the above-mentioned devices and modules can refer to the corresponding processes in the above-mentioned method embodiments, and will not be described repeatedly here.
[0114] In some embodiments provided by the present application, the interconnections between modules may be electrical, mechanical, or of other form.
[0115] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each unit may exist physically separately, or two or more modules may be integrated into one module. The integrated module may be realized in the form of hardware or in the form of a software functional module.
[0116] In the technical solution provided by the present application, the initial speech features of the speech are obtained, and the initial speech features are input into a pre-trained speech enhancement model to obtain a target speech feature output from the speech enhancement model, and the speech enhancement model is obtained through stepwise training based on a depth clustering loss function and a mask estimation loss function, and a target speech with noise and reverberation removed can be calculated based on the target speech features. In this way, model training is performed on the pre-set speech enhancement model through different loss functions, and the model is guided to efficiently remove noise and reverberation in the speech, so that the performance of speech enhancement can be improved while reducing the model computational resources.
[0117] 10, the embodiment of the present application further provides a computer device 600, which includes a processor 610, a memory 620, a power supply 630, and an input unit 640, in which computer program instructions are stored in the memory 620, and the computer program instructions can execute steps of various methods provided by the above embodiment when called by the processor 610. Those skilled in the art can understand that the illustrated structure of the computer device does not constitute a limitation of the computer device, and the computer device may include more or less components than those illustrated, or may combine certain components, or have a different component arrangement.
[0118] The processor 610 may include one or more processing cores. The processor 610 connects each part of the entire battery management system using various interfaces and lines, executes instructions, programs, code sets or instruction sets stored in the memory 620, and calls data stored in the memory 620 to perform various functions of the entire battery management system, process data, and control the entire computer device by performing various functions of the computer device and processing data. In some embodiments, the processor 610 may be realized in the form of at least one hardware of a digital signal processor (DSP), a field programmable gate array (FPGA), and a programmable logic array (PLA). The processor 610 may integrate one or more combinations of a central processing unit (CPU), a graphics processing unit (GPU), and a modem, etc. Here, the CPU mainly processes the operating system, user interface, applications, etc., the GPU is responsible for rendering and drawing display contents, and the modem is used to handle wireless communication. It can be understood that the modem may not be integrated into the processor 610, but may be realized alone in only one communication chip.
[0119] The memory 620 may include a random access memory 620 (RAM) or a read-only memory 620 (Read-Only Memory). The memory 620 may be configured to store instructions, programs, codes, code sets, or instruction sets. The memory 620 may include a storage program area and a storage data area, where the storage program area can store instructions for implementing an operating system, instructions for implementing at least one function (e.g., touch function, audio playback function, image playback function, etc.), instructions for implementing the following various method embodiments, etc. The storage data area can also store data (e.g., phone book and audio / video data) created by the computer device during use. Correspondingly, the memory 620 may also include a memory controller for providing access to the memory 620 by the processor 610.
[0120] The power source 630 is logically connected to the processor 610 through a power management system, and can realize functions such as charging, discharging, and managing power consumption through the power management system. The power source 630 may further include optional components such as one or more DC or AC power sources, a recharging system, a power failure detection circuit, a power converter or inverter, and a power status indicator.
[0121] The input unit 640 is configured to receive entered numeric or textual information and generate keyboard, mouse, joystick, optical or trackball signal inputs associated with user settings and function control.
[0122] In addition, although not shown, the computer device 600 may further include a display unit, etc., which will not be described again here. Specifically, in the embodiment of the present application, the processor 610 in the computer device loads executable files corresponding to the processes of one or more applications into the memory 620 according to the following instructions, and the processor 610 executes the applications stored in the memory 620 to realize the steps of various methods provided by the above embodiment.
[0123] As shown in FIG. 11, an embodiment of the present application further provides a computer-readable storage medium 700, in which computer program instructions 710 are stored, and the computer program instructions 710 can be called by a processor to perform the method described in the above embodiment.
[0124] The computer-readable storage medium may be an electronic memory such as a flash memory, an electrically erasable programmable read-only memory (EEPROM), an EPROM, a hard disk, or a ROM. In some embodiments, the computer-readable storage medium includes a non-transitory computer-readable storage medium. The computer-readable storage medium 700 has storage space for program codes for performing the method steps of any of the above methods. These program codes may be read from or written into one or more computer program products. The program codes may be, for example, compressed in a suitable format.
[0125] According to one aspect of the present application, there is provided a computer program product or computer program, the computer program product or computer program comprising computer instructions, the computer instructions being stored in a computer readable storage medium, a processor of a computing device reading the computer instructions from the computer readable storage medium, and the processor executing the computer instructions, thereby causing the computing device to perform the methods provided in the various alternative embodiments provided by the above examples.
[0126] The above is only a preferred embodiment of the present application, and is not intended to limit the present application. As described above, the present application is disclosed in a preferred embodiment, but is not intended to limit the present application. Those skilled in the art can make equivalent changes or modifications using the technical content disclosed above without departing from the scope of the technical solution of the present application, but the changes and modifications made to the above embodiment based on the technical substance of the present application without departing from the content of the technical solution of the present application are still within the scope of the technical solution of the present application. [Explanation of symbols]
[0127] 300 Audio Processing System 310 Near-end client 330 Far End Client 350 Server side 410 Cloud Server 411 Speech Enhancement Model 420 participants 430 Conference Terminal No. 1 450 2nd Conference Terminal 500 Audio Processing Device 510 Acquisition Module 520 Emphasis Module 530 Calculation Module 600 Computer Equipment 610 Processor 620 Memory 630 Power supply 640 Input Units 700 Computer-readable storage medium 710 Computer Program Instructions
Claims
1. 1. A method for processing audio, comprising: obtaining initial speech characteristics of a call voice; inputting the initial speech features into a speech enhancement model to obtain target speech features output from the speech enhancement model, the speech enhancement model being obtained by stepwise training based on a depth clustering loss function and a mask estimation loss function; and calculating a denoised and dereverbed target speech based on the target speech features.
2. The audio processing method includes: The method further includes pre-training the speech enhancement model according to the following manner: Obtaining a training sample set, the training sample set including noisy speech features, clean speech labels, noisy speech labels, and depth clustering annotations; obtaining a given enhancement network; and performing denoising training and dereverberation training on the predetermined enhancement network through the training sample set step by step until the predetermined enhancement network satisfies a predetermined condition, and obtaining a target enhancement network after training as the speech enhancement model.
2. The audio processing method according to claim 1.
3. The predetermined enhancement network includes a hidden layer, a depth clustering layer, and a mask estimation layer, the mask estimation layer including a speech mask estimation layer and a noise mask estimation layer, and the step of incrementally performing denoising training and dereverberation training on the predetermined enhancement network through the training sample set until the predetermined enhancement network satisfies a predetermined condition includes: inputting the noisy speech features into the hidden layer and generating intermediate training features through the hidden layer; inputting the intermediate training features into the depth clustering layer and generating clustering training annotations through the depth clustering layer; inputting the intermediate training features into the speech mask estimation layer and generating clean speech training features through the speech mask estimation layer; inputting the intermediate training features into the noise mask estimation layer to generate noisy speech training features through the noise mask estimation layer; constructing a target loss function based on the clean speech label, the noisy speech label, the depth clustering annotations, the clean speech training features, the noisy speech training features, and the clustering training annotations; and incrementally performing denoising training and dereverberation training on the given enhancement network based on the target loss function until the given enhancement network satisfies a predetermined condition. The audio processing method according to claim 2 .
4. constructing a target loss function based on the clean speech label, the noisy speech label, the depth clustering annotations, the clean speech training features, the noisy speech training features, and the clustering training annotations, and incrementally performing denoising training and dereverberation training on the given enhancement network based on the target loss function until the given enhancement network satisfies a predetermined condition, determining a first loss function based on the clustering training annotations and the depth clustering annotations; determining a second loss function based on the clean speech training features and the clean speech labels; determining a third loss function based on the noisy speech training features and the noisy speech labels; constructing a target loss function for the given enhancement network based on the first loss function, the second loss function, and the third loss function; and incrementally performing denoising training and dereverberation training on the given enhancement network based on the target loss function until the given enhancement network satisfies a given condition. The audio processing method according to claim 3.
5. constructing a target loss function for the given emphasis network based on the first loss function, the second loss function, and the third loss function, performing weighted summation on the first loss function, the second loss function, and the third loss function based on weighting parameters corresponding to the first loss function, the second loss function, and the third loss function, respectively, to obtain a target loss function of the predetermined emphasis network; 5. The audio processing method according to claim 4.
6. The clean speech labels include first clean speech labels, and determining a second loss function based on the clean speech training features and the clean speech labels includes: determining a denoising loss function based on the clean speech training features and the first clean speech label; and selecting the denoising loss function as a second loss function, and the first clean speech label is a speech label obtained based on a noiseless and reverberant speech.
5. The audio processing method according to claim 4.
7. The clean speech labels include second clean speech labels, and determining a second loss function based on the clean speech training features and the clean speech labels includes: determining a dereverberation loss function based on the clean speech training features and the second clean speech label; and selecting the dereverberation loss function as a second loss function, the second clean speech label being a speech label obtained based on a noise-free and dereverberant speech.
5. The audio processing method according to claim 4.
8. constructing a target loss function for the given enhancement network based on the first loss function, the second loss function, and the third loss function; and incrementally performing denoising training and dereverberation training on the given enhancement network based on the target loss function until the given enhancement network satisfies a given condition, obtaining application scene attributes; determining a corresponding distributed training policy based on the application scene attributes; constructing a target loss function for the given enhancement network based on the first loss function, the second loss function, and the third loss function based on the distributed training policy; and incrementally performing denoising training and dereverberation training on the given enhancement network based on the target loss function until the given enhancement network satisfies a predetermined condition.
8. A method for processing audio according to claim 5, 6 or 7.
9. the distributed training policy includes a first distributed training policy, and the step of constructing a target loss function for the given enhancement network based on the first loss function, the second loss function, and the third loss function based on the distributed training policy, and incrementally performing denoising training and dereverberation training on the given enhancement network based on the target loss function until the given enhancement network satisfies a given condition, includes: if the distributed training policy is a first distributed training policy, determining a target loss function for the given enhancement network based on the first loss function, the second loss function, and the third loss function, and repeating denoising training on the given enhancement network based on the target loss function until the given enhancement network satisfies a predetermined condition to obtain a denoising network, wherein the second loss function is determined by a denoising loss function; determining a target loss function of the denoising network based on the first loss function, the second loss function, and the third loss function; and repeating dereverberation training on the denoising network based on the target loss function until the denoising network satisfies a predetermined condition, wherein the second loss function is determined by a dereverberation loss function.
9. The audio processing method according to claim 8.
10. the distributed training policy includes a second distributed training policy, and the step of constructing a target loss function for the given enhancement network based on the first loss function, the second loss function, and the third loss function based on the distributed training policy, and incrementally performing denoising training and dereverberation training on the given enhancement network based on the target loss function until the given enhancement network satisfies a given condition, includes: if the distributed training policy is a second distributed training policy, determining a target loss function for the given enhancement network based on the first loss function, the second loss function, and the third loss function, and repeating dereverberation training on the given enhancement network based on the target loss function until the given enhancement network satisfies a predetermined condition to obtain a dereverberation network, wherein the second loss function is determined by a dereverberation loss function; determining a target loss function of the dereverberation network based on the first loss function, the second loss function, and the third loss function; and repeating denoising training on the dereverberation network based on the target loss function until the dereverberation network satisfies a predetermined condition, wherein the second loss function is determined by the denoising loss function.
9. The audio processing method according to claim 8.
11. The step of obtaining the training sample set comprises: acquiring a first sample sound, the first sample sound being a sound including noise and reverberation collected based on a microphone; performing speech feature extraction on the first sample speech to obtain a noise speech feature; obtaining second sample speech, the second sample speech including clean speech with no noise and reverberation and clean speech with no noise and no reverberation; performing speech feature extraction on the second speech sample to obtain a first clean speech label and a second clean speech label; determining depth clustering annotations based on the first sample audio and the second sample audio. The audio processing method according to claim 2 .
12. The predetermined condition is: The total loss value of the target loss function is equal to or less than a predetermined value, the total loss value of the target loss function does not change, or the number of training sessions reaches a predetermined number. A method for processing audio according to any one of claims 3 to 11.
13. The speech enhancement model includes a hidden layer, a depth clustering layer, a speech mask estimation layer, and a noise mask estimation layer, and the step of inputting the initial speech features into a pre-trained speech enhancement model to obtain a target speech feature output from the speech enhancement model includes: inputting the initial speech features into the hidden layer and generating intermediate features through the hidden layer; inputting the intermediate features into the speech mask estimation layer, generating clean speech features through the speech mask estimation layer, and setting the clean speech features as target speech features; The step of calculating a denoised and dereverberated target speech based on the target speech features includes: performing a feature inverse transform on the target speech features to calculate a noise and reverberant free target speech; 2. The audio processing method according to claim 1.
14. 1. An audio processing device, comprising: An acquisition module configured to acquire initial speech characteristics of a speech signal; an enhancement module configured to input the initial speech features into a pre-trained speech enhancement model to obtain target speech features output from the speech enhancement model, the speech enhancement model being obtained by step-wise training based on a depth clustering loss function and a mask estimation loss function; a computational model for computing a target speech from which noise and reverberation have been removed based on the target speech features.
15. 1. A computer-readable storage medium, comprising: The computer-readable storage medium has stored thereon program code, the program code being called by a processor to perform the method according to any one of claims 1 to 13.
16. 1. A computer device comprising: Memory, one or more processors coupled to the memory; and one or more applications stored in said memory and configured to be executed by said one or more processors, said applications configured to perform the method of any one of claims 1 to 13.
17. A computer program product or computer program comprising: The computer program product or computer program comprises computer instructions, the computer instructions being stored in a storage medium, and a processor of a computing device reads and executes the computer instructions from the storage medium, thereby causing the computing device to perform the method of any one of claims 1 to 13.
Citation Information
Patent Citations
Speech dialogue method, apparatus and system
JP2019204074A
Audio signal processing system and method for converting an input audio signal
JP2021515277A
Methods and Systems for End-to-End Speech Separation with Unfolded Iterative Phase Reconstruction
US20190318754A1
