Noise reduction method for vocoder, vocoder, electronic device and storage medium thereof

Abstract

The present application discloses a denoising method for a vocoder and a vocoder, an electronic device and a storage medium thereof, wherein the method comprises: the vocoder obtains an acoustic feature to be synthesized and a noise label, wherein the noise label is related to the degree of denoising of the acoustic feature to be synthesized; the vocoder performs audio synthesis based on the acoustic feature to be synthesized and the noise label to obtain clean synthesized audio data; the vocoder uses the acoustic feature and the noise training label of the noisy training audio as input and the clean training audio as output to obtain training; wherein the noisy training audio is synthesized by the clean training audio and the noisy training audio, and the noise training label corresponds to the noise degree of the noisy training audio. In the above manner, the present application can control the degree of denoising of the acoustic feature to be synthesized.

Description

Technical Field

[0001] The present application relates to the field of speech technology, and in particular to a noise reduction method for a vocoder, a vocoder, an electronic device and a storage medium thereof. Background Art

[0002] General vocoders are usually non-universal vocoders, that is, they are trained with a single sound library and can only be used to synthesize the voice of the speaker in the sound library, and can only synthesize the language of the training sound library. For unseen speakers or unseen languages, the synthesis effect will be significantly reduced. In order to improve the versatility of vocoders, the demand for universal vocoders has emerged. In theory, a universal vocoder can synthesize audio with ideal effects for any language, any speaker, and the acoustic characteristics of any recording environment.

[0003] Since universal vocoders are highly versatile, the acoustic features used for synthesized audio in some scenarios usually contain non-human voices such as noise, reverberation or background sound, and the presence of non-human voices will greatly reduce the quality of the synthesized audio. After long-term research by the inventors of this application, it is found that most universal vocoders currently do not have a noise reduction function, and a few universal vocoders with a noise reduction function need to redesign the vocoder structure according to the noise reduction requirements, resulting in their solutions being non-versatile and unable to control the noise reduction range. Summary of the invention

[0004] In a first aspect, the present application provides a denoising method for a vocoder, the method comprising: the vocoder obtains an acoustic feature to be synthesized and a noise label, wherein the noise label is related to the degree of denoising of the acoustic feature to be synthesized; the vocoder performs audio synthesis based on the acoustic feature to be synthesized and the noise label to obtain clean synthesized audio data; the vocoder is trained with the acoustic feature of noisy training audio and the noise training label as input and the clean training audio as output; wherein the noisy training audio is synthesized from the clean training audio and the noisy training audio, and the noise training label corresponds to the noise degree of the noisy training audio.

[0005] A second aspect of the present application provides a vocoder, which includes: an acquisition module, used to acquire acoustic features to be synthesized and noise labels, wherein the noise labels are related to the noise reduction degree of the acoustic features to be synthesized; a noise reduction synthesis module, used to perform audio synthesis based on the acoustic features to be synthesized and the noise labels to obtain clean synthesized audio data; the vocoder is trained with noisy training audio and noise training labels as input and clean training audio as output; wherein the noisy training audio is synthesized from the clean training audio and the noisy training audio, and the noise training label corresponds to the noise degree of the noisy training audio.

[0006] A third aspect of the present application provides an electronic device, which includes a memory and a processor coupled to each other, the memory is used to store program data, and the processor is used to execute the program data to implement the aforementioned method.

[0007] A fourth aspect of the present application provides a computer-readable storage medium, in which program data is stored. When the program data is executed by a processor, it is used to implement the aforementioned method.

[0008] The beneficial effects of the present application are as follows: Different from the prior art, the vocoder of the present application obtains the acoustic features to be synthesized and the noise labels, wherein the noise labels are related to the noise reduction degree of the acoustic features to be synthesized, and then the vocoder performs audio synthesis based on the acoustic features to be synthesized and the noise labels to obtain clean synthesized audio data; wherein the vocoder is trained with the acoustic features and noise training labels of the noisy training audio as input and the clean training audio as output; wherein the noisy training audio is synthesized by the clean training audio and the noise training audio, and the noise training label corresponds to the noise degree of the noise training audio. Thus, by adding the corresponding noise training label to the noise degree of the noise training audio for training, a vocoder that can control the noise reduction degree can be obtained. In the application stage, the noise reduction degree of the acoustic features to be synthesized can be controlled by modifying the noise label, thereby controlling the background noise intensity of the synthesized audio to be suitable for different scenarios. In addition, there is no need to redesign the vocoder structure according to the noise reduction requirements, and it has wider versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in this application, the following briefly introduces the drawings required in the description of the embodiments. Obviously, the drawings described below are only some embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0010] Figure 1 It is a flow chart of an embodiment of a noise reduction method for a vocoder of the present application;

[0011] Figure 2 It is a schematic diagram of a process of synthesizing audio by a general vocoder in the related art;

[0012] Figure 3 It is a schematic diagram of the process of synthesizing audio by a vocoder with noise reduction function provided by the present application;

[0013] Figure 4 yes Figure 1 A schematic flow chart of an implementation method of step S12;

[0014] Figure 5 yes Figure 1A schematic flow chart of another implementation of step S12;

[0015] Figure 6 is a flow chart of an embodiment of a method for training a vocoder;

[0016] Figure 7 is a flow chart of another embodiment of a method for training a vocoder;

[0017] Figure 8 Another flow chart of another embodiment of the vocoder training method of the present application is as follows

[0018] Fig. 9 is a flow chart of another embodiment of a method for training a vocoder;

[0019] Fig.10 It is a schematic block diagram of the structure of an embodiment of a vocoder of the present application;

[0020] Fig.11 It is a schematic structural block diagram of an embodiment of an electronic device of the present application;

[0021] Fig.12 It is a schematic block diagram of the structure of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0022] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0023] The terms "first" and "second" in this application are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" can explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0024] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0025] For general vocoder tasks, signal processing-based solutions were usually used in the early days. This type of vocoder does not rely on training data and naturally has general synthesis capabilities, but the overall effect is average, and the naturalness and sound quality are relatively poor. The current mainstream technical solution is the neural network-based vocoder, which has significantly improved naturalness and sound quality compared to the former. Neural network general vocoders usually use a large amount of multi-person and multi-language sound library data. Some solutions will also combine speaker-related encoders, so that the trained vocoder can have general synthesis capabilities.

[0026] There are currently few types of solutions for the denoising task of general vocoders. One existing solution is based on a specific neural network vocoder structure HiNet, which modifies the structure of its predicted log-amplitude spectrum to enable it to have the ability to denoise and dereverberate. The HiNet vocoder takes acoustic features as input, predicts the log-amplitude spectrum and phase respectively through two independent networks, and then restores the audio through short-time Fourier synthesis. This solution redesigns the network that predicts the log-amplitude spectrum of the HiNet vocoder. For example, it uses noisy acoustic features to predict the reverberation wave and the noisy log-amplitude spectrum respectively, and then uses the above two to denoise and dereverberate the noisy log-amplitude spectrum. After that, a post-filter network is added to further improve the sound quality.

[0027] At present, most mainstream general vocoder solutions directly use sound library-level data for simple and direct training. If the input acoustic features are relatively complex, such as non-human voices such as ambient sound, background noise, reverberation, and breathing sounds, the synthesized audio effect will be significantly reduced, so the training data requirements are extremely high. However, the existing general vocoders with noise reduction have the following problems: First, they are usually limited to specific network structures and are not applicable to other vocoder structures, so their application range is relatively narrow; second, during training, in addition to general target audio, they also need to provide reverberation waves and noise logarithmic amplitude spectra as training targets. The data types are complex and the training is difficult; third, this solution can only remove background noise and reverberation, and cannot remove other non-human voices, such as breathing sounds; fourth, this solution cannot adjust the noise reduction amplitude and does not have the ability to control noise reduction.

[0028] Therefore, the present application proposes a denoising method for a vocoder, which can denoise the noisy features when the vocoder synthesizes audio based on the noisy features to improve the sound quality of the synthesized audio data. It can also be based on any existing mainstream vocoder structure, with a wide range of applications and low deployment costs. At the same time, the system training process is clear and simple, and the stability is high. In addition, it has the ability to control the noise reduction amplitude.

[0029] See also Figure 1 , Figure 1 It is a flowchart of an embodiment of a noise reduction method for a vocoder of the present application, wherein the executor of the present application is a vocoder.

[0030] like Figure 1 As shown, the method may include the following steps:

[0031] Step S11: the vocoder obtains the acoustic features to be synthesized and the noise label, wherein the noise label is related to the noise reduction degree of the acoustic features to be synthesized.

[0032] Among them, the acoustic features to be synthesized can be obtained by inputting the synthesized text into the acoustic model. The text to be synthesized is, for example, "Today's weather is really good". Inputting it into the acoustic model can be converted into the corresponding acoustic features to be synthesized, and then the vocoder can perform audio synthesis based on the acoustic features to be synthesized to obtain clean synthesized audio data. Since the acoustic features output by the acoustic model are usually noisy, if the noisy acoustic features are used for audio synthesis, the sound quality of the synthesized audio data will be affected. Optionally, the type of acoustic model can be selected according to actual needs.

[0033] Among them, the acoustic features may include but are not limited to: at least one of spectrum parameters such as spectrum and cepstrum. In addition, it may also include one or more of fundamental frequency, unvoiced sound and voiced sound. In the present embodiment, the acoustic features to be synthesized are illustrated by taking spectrum as an example, and specifically may be mel spectrogram. In other embodiments, the acoustic features to be synthesized may be cepstrum + fundamental frequency, and may also be combined with unvoiced sound and voiced sound. It is understandable that when applied, it is necessary to prepare acoustic features of the same category as input according to the acoustic features used when training the vocoder. For example, if the acoustic feature used in training is an 80-dimensional mel spectrum, then the 80-dimensional mel spectrum is also used as input when applied.

[0034] The noise label corresponds to the noise reduction degree of the acoustic feature to be synthesized. The noise label can be in the form of a numerical value, such as 0, 1, 2, 3, where 0 means no noise reduction at all, and the larger the numerical value, the greater the corresponding noise reduction degree. Of course, the noise label can also be other data with a certain order, such as letters, etc., which can be selected according to needs in actual applications and are not limited here.

[0035] The methods of obtaining noise labels are not limited to the following two: one is to determine the noise label according to the noise intensity in the acoustic feature to be synthesized. For example, the greater the noise intensity, the larger the noise label, and the greater the noise reduction degree of the corresponding acoustic feature to be synthesized; the other is manual addition, that is, the user can select the noise label for the acoustic feature to be synthesized according to the needs. For example, if the user wants to increase the noise reduction degree, he can select a larger noise label, and if he wants to reduce the noise reduction degree, he can select a smaller noise label.

[0036] In some embodiments, the noise label can be obtained by combining the above two methods. For example, the noise label can be determined according to the noise level in the acoustic feature to be synthesized, and then the user can understand the noise intensity in the acoustic feature to be synthesized according to the noise label, and then modify it based on the noise label. If the noise label is not specified, it will be defaulted to perform no noise reduction processing, so that the corresponding noise label is set, such as 0.

[0037] Step S12: The vocoder performs audio synthesis based on the acoustic features to be synthesized and the noise labels to obtain clean synthesized audio data; the vocoder takes the acoustic features of the noisy training audio and the noise training labels as input and the clean training audio as output, and is trained; wherein the noisy training audio is synthesized from the clean training audio and the noisy training audio, and the noise training label corresponds to the noise level of the noisy training audio.

[0038] In this embodiment, the vocoder obtains the acoustic features to be synthesized and the noise labels, wherein the noise labels are related to the noise reduction degree of the acoustic features to be synthesized, and then the vocoder performs audio synthesis based on the acoustic features to be synthesized and the noise labels to obtain clean synthesized audio data; wherein the vocoder is trained with the acoustic features and noise training labels of the noisy training audio as input and the clean training audio as output; wherein the noisy training audio is synthesized by the clean training audio and the noise training audio, and the noise training label corresponds to the noise degree of the noise training audio. Thus, by adding the corresponding noise training label to the noise degree of the noise training audio for training, a vocoder that can control the noise reduction degree can be obtained. In the application stage, the noise reduction degree of the acoustic features to be synthesized can be controlled by modifying the noise label, thereby controlling the background noise intensity of the synthesized audio to be suitable for different scenarios. In addition, there is no need to redesign the vocoder structure according to the noise reduction requirements, and it has wider versatility.

[0039] See also Figures 2 to 4 , Figure 2 It is a schematic diagram of the process of synthesizing audio by a general vocoder in the related art. Figure 3 is a schematic diagram of the process of synthesizing audio using a vocoder with noise reduction function provided by the present application, Figure 4 yes Figure 1 A schematic flow chart of an implementation of step S12 in FIG.

[0040] like Figure 2 As shown, in the related art, a general vocoder performs audio synthesis based on the acoustic features to be synthesized, wherein the target features obtained by processing the reference audio data by the speaker encoder can be combined to perform auxiliary synthesis to obtain clean synthesized audio data. For the description of the speaker encoder, please refer to the following embodiments.

[0041] like Figure 3 and Figure 4 As shown, in some embodiments, step S12 may include sub-steps S121 to S123:

[0042] Step S121: The vocoder converts the noise label into a noise feature.

[0043] Among them, the noise label can be input into the embedding layer (Embeddings), and then the noise feature (NoiseEmbedding) is obtained.

[0044] Step S122: Fusing the noise feature with the acoustic feature to be synthesized.

[0045] Among them, the noise features and acoustic features can be fused through Concat, that is, Figure 3 C in.

[0046] Step S123: Perform audio synthesis based on the fused features to obtain clean synthesized audio data.

[0047] For example, the vocoder includes a noise reduction universal vocoder, and the fused features can be input into the noise reduction universal vocoder so that the noise reduction universal vocoder performs audio synthesis based on the fused features to obtain clean synthesized audio data. Universal vocoders can play an important role in various scenarios such as personalized synthesis (sound cloning), cross-language synthesis (such as translation machines and other equipment), and multi-style synthesis. Among them, the noise reduction universal vocoder can be any universal vocoder trained by the training method provided in this application, and can be built on the basis of any existing mainstream vocoder structure with strong versatility and low deployment cost.

[0048] In some embodiments, the vocoder includes a self-noise reduction codec, and the self-noise reduction coder is used to encode the acoustic features to be synthesized before fusing the noise features with the acoustic features to be synthesized to obtain intermediate acoustic features, and then fuse the noise features with the intermediate acoustic features. Thus, by converting the acoustic features to be synthesized into intermediate acoustic features, and then fusing the intermediate acoustic features with the noise features, the intermediate acoustic features can filter out non-human voices such as noise and reverberation, and only retain the information corresponding to the clean acoustic features, which is equivalent to the noise reduction universal vocoder borrowing the encoder part (encoder) of the self-noise reduction coder, and performing a round of noise reduction on the input acoustic features to be synthesized in advance. Thus, the noise reduction capability of the vocoder can be further improved.

[0049] See also Figure 5 And continue to read Figure 3 , Figure 5 yes Figure 1 Schematic diagram of the flow chart of another implementation of step S12.

[0050] In some other embodiments, the vocoder further includes a speaker encoder. Figure 5 As shown, step S12 may include sub-steps S124 to S127, wherein there is no fixed order relationship between steps S124 to S127 and steps S121 to S123.

[0051] Step S124: The vocoder converts the noise label into a noise feature.

[0052] For this step, please refer to step S121, which will not be described in detail here.

[0053] Step S125: fusing the noise feature with the acoustic feature to be synthesized.

[0054] Among them, the noise features and acoustic features can be fused through Concat.

[0055] Step S126: The speaker encoder is used to process the reference audio data to obtain target features, and the reference audio data is obtained by collecting the target.

[0056] The reference audio data can be any content with a moderate length and is used to extract characteristic information such as the timbre of the target.

[0057] The target feature can reflect the sound characteristics of different targets. For example, the target feature can be a timbre feature, which is used to reflect the timbre of the target.

[0058] Step S127: Perform audio synthesis based on the fused features and the target features to obtain clean synthesized audio data.

[0059] Specifically, the fused features and target features can be input into a noise reduction universal vocoder, so that the noise reduction universal vocoder performs audio synthesis based on the fused features and target features to obtain clean synthesized audio data. Among them, the speaker encoder can uniquely encode the timbre and other characteristics of different speaking targets, that is, output the target features, so as to use the target features to assist in audio synthesis, and can well restore the timbre and other characteristics of the target in the audio.

[0060] The following is an introduction to the training method of the vocoder. The vocoder is trained with the acoustic features of the noisy training audio and the noise training label as input and the clean training audio as output; the noisy training audio is synthesized from the clean training audio and the noisy training audio, and the noise training label corresponds to the noise level of the noisy training audio.

[0061] See also Figure 6 , Figure 6 FIG. 4 is a flow chart of an embodiment of a method for training a vocoder.

[0062] First, the method of obtaining noisy training audio and noise training labels is introduced.

[0063] Step S21: Acquire clean training audio and noisy training audio.

[0064] (1) Obtain clean training audio

[0065] Among them, clean sound library data can be obtained from internal data, open source data, etc. as clean training audio. The clean training audio can involve multiple people, multiple languages, multiple scenes, multiple age groups, and multiple styles to ensure the diversity of data distribution, so that the vocoder trained with this data has better versatility.

[0066] In some embodiments, the clean training audio can be further energy regularized, and the acoustic features of the clean training audio can be extracted at the same time. In some scenarios, because different people have different volumes, the burden of model training will be increased, so the volume of the audio will be regularized to a preset proportion of the maximum volume (for example, 75%) to facilitate model training. In addition, the extracted acoustic features can also be uniformly normalized, such as calculating the mean and variance of all acoustic features, and subtracting the mean and variance as input during training to reduce the amount of calculation of the model. Afterwards, the clean training audio and its acoustic features together constitute the unnoised portion of the training data set.

[0067] (2) Obtaining noise training audio

[0068] Considering that actual noisy video data usually contains real environmental noise rather than white noise, the noise data used in model training should also be mainly real noise. For example, as much noise data as possible can be collected from professional noise websites, noise data sets, and artificial outdoor collection to ensure coverage of various scenarios and noise levels.

[0069] In some embodiments, the noise data may be further processed in advance to ensure the validity of the subsequent noise labels. For example, only a section of each noise file with relatively stable noise intensity is intercepted as noise training audio, which is usually several seconds long. The intercepted audio must not contain human voices to prevent mixing with human voices in clean training audio and affecting the effect of the vocoder.

[0070] The following is a noise addition process for clean training audio, wherein two exemplary noise addition methods are step S22 and step S23:

[0071] Step S22: adding noise to the clean training audio by randomly selecting the noise training audio to obtain the noisy training audio.

[0072] Specifically, a noisy training audio is randomly selected for each clean training audio to be noised.

[0073] Step S23: according to the preset noise adding intensity, the noise training audio is noise added to the clean training audio to obtain the noisy training audio.

[0074] The noise addition intensity can be adjusted by the signal-to-noise ratio of the clean training audio and the noisy training audio.

[0075] Specifically, noises of different intensities may be increased in proportion. For example, a preset number of clean training audios may be divided into a group, and then noises of different intensities may be increased in proportion to the preset number of clean training audios.

[0076] For example, four clean training audios can be divided into one group, the first one is not noised, and the next three are noised with signal-to-noise ratios of 20, 10, and 5 respectively. This method can obtain training audio data with different noise intensities and enhance the robustness of the model. Afterwards, the acoustic features of the noisy training audio can be extracted and used as the noisy part of the training data set. Among them, the noise intensity of the noisy training audio indirectly reflects the degree of noise in the corresponding acoustic features.

[0077] The noise reduction mentioned in this application refers to the removal of non-human voice parts such as noise, reverberation, and breathing in acoustic features, rather than the narrow sense of removing background noise. During training, the noise-added training audio generally refers to the mixing of various non-human voice parts.

[0078] Step S24: Generate a corresponding noise training label according to the noise intensity of the noise training audio.

[0079] Among them, the corresponding noise training label can be generated according to the signal-to-noise ratio of the clean training audio and the noisy training audio in the noisy training audio. Among them, if the corresponding noise labels are generated in sequence according to the signal-to-noise ratio from weak to strong, at this time, the stronger the signal-to-noise ratio, the smaller the corresponding noise training label. For example, the noise training labels of the above 4 noisy training audios are 0, 1, 2, and 3, respectively. Among them, 0 means no noise, and 3 means the strongest noise. The noise training label is also input as part of the training data set. Of course, in other implementations, the corresponding noise labels can also be generated in sequence according to the signal-to-noise ratio from strong to weak. At this time, the stronger the signal-to-noise ratio, the larger the corresponding noise training label.

[0080] The following is a noise adding training based on the training data set, which may specifically include step S25 and step S26:

[0081] Step S25: The vocoder uses the acoustic features of the clean training audio as input and the clean training audio as output, and trains to obtain a pre-trained model.

[0082] Among them, the pre-training model is obtained by pre-training the vocoder to ensure that the vocoder has basic synthesis capabilities and prevent noise from participating in training too early, which may lead to model convergence failure.

[0083] In some embodiments, when the vocoder includes a speaker encoder and a noise reduction universal vocoder, the speaker encoder has been trained, so that the target features output by the speaker encoder can be added during the noise training of the noise reduction universal vocoder to assist the acoustic features in synthesizing audio data with speaker features.

[0084] Specifically, the speaker encoder processes the reference training audio to obtain the target features of the corresponding target, so that the noise reduction universal vocoder can use the acoustic features and target features of the clean training audio as input and the clean training audio as output to train and obtain a pre-trained model. The reference training audio and the clean training audio are collected for the same target. The reference training audio can be the clean training audio or a part of the clean training audio, or it can be an audio different from the clean training audio. In this embodiment, for the convenience of training, the clean training audio can be directly selected as the reference training audio.

[0085] Step S26: The vocoder loads the pre-trained model, and uses the acoustic features of the noisy training audio and the noise training label as input and the clean training audio as output for training.

[0086] Specifically, the vocoder can convert the noise training label into a noise training feature, and then fuse the noise training feature with the acoustic feature of the noisy training audio, and then use the fused feature as the input of the vocoder. Here, please refer to steps S121 to S123 in the above application embodiment, which will not be repeated here.

[0087] In some embodiments, when the vocoder includes a speaker encoder and a noise reduction universal vocoder, the speaker encoder outputs the target features, and the noise reduction universal vocoder can be trained with the acoustic features of the noisy training audio, the noise training labels and the target features as inputs and the clean training audio as output.

[0088] With this training method, the model will be able to distinguish the noise part of the acoustic features and couple the noise intensity with the noise training label.

[0089] Furthermore, based on the noise-adding training method introduced above, this application also proposes two additional training methods. One is to perform distillation training when the denoising universal vocoder is an autoregressive denoising universal vocoder to solve the sound leakage problem caused by the characteristics of the autoregressive denoising universal vocoder itself; the other is to perform multi-task training in combination with the autoregressive denoising codec to further filter out the non-human voice part of the acoustic features and improve the sound quality of the final synthesized audio data. The above two methods can also be combined.

[0090] See also Figures 7 and 8 , Figure 7 is a flow chart of another embodiment of the vocoder training method of the present application, Figure 8 FIG. 4 is another flowchart of another embodiment of the vocoder training method of the present application.

[0091] In the present application, the vocoder may include any mainstream neural network vocoder, including non-autoregressive vocoders, such as HiFiGAN, Mel-GAN, etc.; and also autoregressive vocoders, such as WaveNet, WaveRNN, LPCNet, etc. When the vocoder includes an autoregressive denoising general vocoder, if the vocoder misidentifies a frame of the synthesized audio as noise from normal pronunciation and denoises it when performing audio synthesis after training, the misidentification error will be amplified in the subsequent process due to the characteristics of autoregression, resulting in sound leakage. For example, some unvoiced sounds and fricatives may be misjudged as noise because they have similarities with ambient noise in acoustic performance. This misjudgment, after being amplified by the autoregressive characteristics, may cause the entire phoneme to be denoised incorrectly, showing a sound leakage situation. Therefore, the present application is specifically designed for a general noise reduction vocoder system with an autoregressive denoising general vocoder model as the backbone, in order to solve the sound leakage problem.

[0092] like Figure 7As shown, after step S24 of the above embodiment, i.e., obtaining the training data set, steps S35 to S37 may also be included. Among them, when the vocoder includes an autoregressive denoising universal vocoder, that is, when the denoising universal vocoder is an autoregressive denoising universal vocoder (such as WaveNet, WaveRNN, LPCNet, etc.), the noise addition training method provided in this embodiment may be used for training to solve the sound leakage problem of the autoregressive denoising universal vocoder.

[0093] Step S35: The baseline vocoder is trained with the acoustic features of the clean training audio as input and the clean training audio as output.

[0094] First, a baseline universal vocoder without noise reduction capability can be trained using the acoustic features of clean training audio as input and clean training audio as output as a teacher model. The parameters of the trained teacher model will no longer be updated in subsequent training processes.

[0095] In some implementations, the baseline vocoder may also be trained by taking the acoustic features of the clean training audio and the target features as inputs in combination with the target features output by the speaker encoder.

[0096] Step S36: The baseline vocoder performs audio synthesis based on the acoustic features of the clean training audio to obtain clean synthesized audio.

[0097] The clean training audio is processed using the trained baseline vocoder to obtain clean synthesized audio, which is then used as the output of the autoregressive denoising universal vocoder.

[0098] Step S37: The autoregressive denoising universal vocoder is trained with the acoustic features of the noisy training audio and the noise training label as input and the clean synthetic audio as output.

[0099] Specifically, the autoregressive denoising universal vocoder can be trained using steps S25 and S26. The difference is that during training, the output of the autoregressive denoising universal vocoder is no longer based on the clean training audio to calculate the loss, but the clean synthetic audio output by the baseline vocoder is used as the output to calculate the loss, which is recorded as the third loss function (loss3).

[0100] In some embodiments, the autoregressive denoising universal vocoder and the baseline vocoder perform at least one loss calculation inside the vocoder. Specifically, the autoregressive denoising universal vocoder may include an encoder and a decoder. In one example, two hidden layer loss functions may be added inside the vocoder, wherein one hidden layer loss function is used to calculate the loss of the baseline vocoder and the autoregressive denoising universal vocoder at the encoder output, recorded as the first loss function (loss1), and the other hidden layer loss function is used to calculate the loss of the baseline vocoder and the autoregressive denoising universal vocoder before the decoder output, recorded as the second loss function (loss2). In this way, the distribution of the student model (autoregressive denoising universal vocoder) can be further brought closer to the teacher model (baseline vocoder).

[0101] It is understandable that the hidden layer loss function may be different for different vocoders, so the specific content of the hidden layer loss function and the location for calculating the loss function can be selected according to actual conditions, and this does not constitute a limitation on the present application.

[0102] In some implementations, the model structures of the baseline vocoder and the autoregressive denoising universal vocoder are the same, which reduces the difficulty of the student model learning the teacher model distribution on the one hand, and facilitates the increase of the hidden layer loss function on the other hand.

[0103] In some implementations, the autoregressive denoising universal vocoder may also be trained by combining the target features output by the speaker encoder and taking the acoustic features and the target features of the noisy training audio as inputs.

[0104] As mentioned above, this distillation training method has two advantages:

[0105] On the one hand, the clean synthetic audio output by the teacher model is obtained from the clean training audio, so the noise reduction ability of the student model can still be maintained.

[0106] On the other hand, the autoregressive denoising universal vocoder learns the clean synthetic audio generated by the teacher model, whose distribution is much simpler than the real audio (i.e., clean training audio). In other words, the modeling task of the autoregressive denoising universal vocoder has changed from complex noisy acoustic features -> complex clean training audio to complex noisy acoustic features -> simple clean synthetic audio, and the mapping relationship is much simpler. In the process of learning this relatively simple mapping relationship, the mapping ability of the model is enhanced. At the same time, since there is only the target sound (such as human voice) part in the clean synthetic audio, this means that the model can better learn the correspondence between the acoustic features of the target sound part and the audio, thereby implicitly improving the model's ability to distinguish the target sound and noise. Therefore, under this distillation scheme, the problem of easy sound leakage in the autoregressive denoising universal vocoder can be solved.

[0107] See also Fig. 9 , Fig. 9 It is a flowchart of another embodiment of the vocoder training method of the present application.

[0108] In order to further improve the noise reduction capability of the vocoder, a self-noise reduction codec can be designed based on the noise reduction universal vocoder. The noise reduction universal vocoder shares the encoder with the former, and the two are jointly trained, so that the noise reduction universal vocoder can borrow the noise reduction capability of the self-noise reduction codec to further improve the noise reduction effect.

[0109] like Fig. 9 As shown, in this embodiment, the vocoder includes a self-noise reduction codec and a noise reduction universal vocoder.

[0110] Specifically, the self-denoising codec takes the acoustic features of the noisy training audio as input and the acoustic features of the clean training audio as output, and the denoising universal vocoder takes the noisy training labels and the intermediate acoustic features output by the self-denoising codec as input and the clean training audio as output, and performs joint training to obtain the vocoder.

[0111] like Fig. 9 As shown in the figure, the self-denoising codec consists of an encoder and a decoder. The input of the self-denoising codec is the acoustic features of the noisy training audio, and the output is the acoustic features of the clean training audio. That is, the task of the self-denoising codec is to remove the noise from the acoustic features of the noisy training audio. Therefore, the intermediate acoustic features (hidden) output by the encoder in the self-denoising codec can filter out non-human voices such as noise and reverberation, and only retain the information corresponding to the clean acoustic features.

[0112] On this basis, the noise reduction universal vocoder replaces the acoustic features of the noisy training audio with the intermediate acoustic features as input, and the other parts remain unchanged, that is, the other parts are consistent with steps S25 and S26 in the above embodiment. It is equivalent to that the noise reduction universal vocoder borrows the encoder part (encoder) of the self-noise reduction encoder, and performs a round of noise reduction on the acoustic features of the input noisy training audio in advance. During training, the goal of the self-noise reduction encoder is clean acoustic features, and the goal of the noise reduction universal vocoder is clean audio, and the entire system performs multi-task training. After the training is completed, the decoder (decoder) part of the self-noise reduction codec can be discarded, and the remaining parts of the entire structure, namely the decoder part of the noise reduction codec and the noise reduction universal vocoder, are used as vocoders in actual applications.

[0113] Among them, when the denoising universal vocoder adopts the autoregressive denoising universal vocoder as the backbone, the noise addition training method of the above-mentioned first distillation method can be combined with the multi-task training method in this embodiment, that is, distillation training is performed under the condition of multi-task training, so as to obtain the best effect.

[0114] Specifically, the baseline vocoder is trained with the acoustic features of clean training audio as input and the clean training audio as output; the baseline vocoder performs audio synthesis based on the acoustic features of the clean training audio to obtain clean synthesized audio; the self-denoising codec takes the acoustic features of the noisy training audio as input and the acoustic features of the clean training audio as output, and the autoregressive denoising universal vocoder takes the noisy training labels and the intermediate acoustic features output by the self-denoising codec as input and the clean synthesized audio as output, and performs joint training to obtain the vocoder.

[0115] As mentioned above, this application provides a new noise-adding training method: by adding noise to clean training audio at different degrees and proportions, and adding corresponding noise training labels according to the degree of noise addition during training, a relatively stable noise reduction universal vocoder model is obtained, and the degree of noise reduction can be controlled by modifying the noise label during synthesis. And this method can be used on any mainstream vocoder, and is very versatile.

[0116] Furthermore, a distillation method under the background of noise-added training is provided: On some general vocoders for autoregressive noise reduction, the vocoders trained with noise-added audio have a certain probability of leaking sound. In this regard, this case adopts a method of distilling the baseline vocoder of clean training audio and adding a hidden layer loss function at the same time, effectively solving the leaking sound problem without losing the noise reduction effect.

[0117] Furthermore, a method for further improving the sound quality of the noise reduction system by using multi-task training is provided: by adding a self-noise reduction codec to the original vocoder structure, the model is trained on multiple tasks. The encoder part in the self-noise reduction codec is used to filter out noise and reverberation in the acoustic features, thereby further improving the sound quality of the synthesized audio data.

[0118] See also Fig.10 , Fig.10 It is a schematic block diagram of the structure of an embodiment of a vocoder of the present application.

[0119] In this embodiment, the vocoder 100 may include an acquisition module 110 and a noise reduction synthesis module 120, wherein the acquisition module 110 is used to acquire acoustic features to be synthesized and noise labels, wherein the noise labels are related to the noise reduction degree of the acoustic features to be synthesized; the noise reduction synthesis module 120 is used to perform audio synthesis based on the acoustic features to be synthesized and the noise labels to obtain clean synthesized audio data; the vocoder is trained by taking the acoustic features of the noisy training audio and the noise training labels as input and the clean training audio as output; wherein the noisy training audio is synthesized by the clean training audio and the noisy training audio, and the noise training label corresponds to the noise degree of the noisy training audio.

[0120] In some embodiments, the noise reduction synthesis module 120 includes a self-noise reduction codec and a noise reduction universal vocoder. The self-noise reduction codec takes the acoustic features of the noisy training audio as input and the acoustic features of the clean training audio as output, and the noise reduction universal vocoder takes the noisy training labels and the intermediate acoustic features output by the self-noise reduction codec as input and the clean training audio as output, and performs joint training to obtain the vocoder.

[0121] In some embodiments, when the noise reduction synthesis module 120 includes an autoregressive noise reduction universal vocoder, the baseline vocoder is trained using the acoustic features of clean training audio as input and the clean training audio as output; the baseline vocoder performs audio synthesis based on the acoustic features of the clean training audio to obtain clean synthesized audio; the autoregressive noise reduction universal vocoder is trained using the acoustic features of the noisy training audio and the noise training label as input and the clean synthesized audio as output.

[0122] In some embodiments, the autoregressive noise reduction universal vocoder and the baseline vocoder perform at least one loss calculation inside the vocoder.

[0123] In some embodiments, the model structure of the baseline vocoder and the autoregressive denoising universal vocoder is the same.

[0124] In some embodiments, the vocoder 100 uses the acoustic features of clean training audio as input and the clean training audio as output to train a pre-trained model; the vocoder 100 loads the pre-trained model and uses the acoustic features of noisy training audio and noise training labels as input and the clean training audio as output to train.

[0125] In some embodiments, the method for obtaining noisy training audio includes: obtaining clean training audio and noisy training audio; adding noise to the clean training audio using the noisy training audio in a random selection manner to obtain noisy training audio; and / or adding noise to the clean training audio using the noisy training audio according to a preset noise intensity to obtain noisy training audio.

[0126] In some implementations, the method for acquiring the noise training label includes: generating a corresponding noise training label according to the noise intensity of the noise-added training audio.

[0127] In some implementations, the noise reduction synthesis module 120 is further used to convert noise labels into noise features; fuse the noise features with the acoustic features to be synthesized; and perform audio synthesis based on the fused features to obtain clean synthesized audio data.

[0128] In some implementations, the noise reduction synthesis module 120 includes a self-noise reduction codec, which encodes the acoustic features to be synthesized to obtain intermediate acoustic features; and fuses the noise features with the intermediate acoustic features.

[0129] In some embodiments, the noise reduction synthesis module 120 also includes a speaker encoder, and the method also includes: the speaker encoder is used to process reference audio data to obtain target features, and the reference audio data is obtained by collecting the target; audio synthesis is performed based on the fused features and the target features to obtain clean synthesized audio data.

[0130] For descriptions of the above steps, please refer to the corresponding positions in the method embodiment, which will not be repeated here.

[0131] See also Fig.11 , Fig.11 It is a schematic structural block diagram of an embodiment of an electronic device of the present application.

[0132] like Fig.11 As shown, the electronic device 200 may include a memory 210 and a processor 220 coupled to each other, the memory 210 is used to store program data, and the processor 220 is used to execute the program data to implement the steps in any of the above method embodiments. Specifically, the electronic device 200 may include but is not limited to: a desktop computer, a laptop computer, a server, a mobile phone, a tablet computer, etc., which are not limited here.

[0133] Specifically, the processor 220 is used to control itself and the memory 210 to implement the steps in any of the above method embodiments. The processor 220 can also be called a CPU (Central Processing Unit). The processor 220 may be an integrated circuit chip with signal processing capabilities. The processor 220 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 220 can be implemented by multiple integrated circuit chips.

[0134] See also Fig.12 , Fig.12 It is a schematic block diagram of the structure of an embodiment of a computer-readable storage medium of the present application.

[0135] like Fig.12 As shown, the computer-readable storage medium 300 stores program data 310. When the program data 310 is executed by a processor, it is used to implement the steps in any of the above method embodiments.

[0136] The computer-readable storage medium 300 may be a medium that can store computer programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or may be a server that stores the computer program. The server may send the stored computer program to other devices for execution, or may execute the stored computer program itself.

[0137] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0138] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0139] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0140] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0141] The above descriptions are merely embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for reducing noise of a vocoder, characterized in that: include: The vocoder obtains the acoustic feature to be synthesized and the noise label, wherein the noise label corresponds to the noise reduction degree of the acoustic feature to be synthesized; The vocoder performs audio synthesis based on the acoustic features to be synthesized and the noise label to obtain clean synthesized audio data, wherein the vocoder performs audio synthesis based on the acoustic features to be synthesized and the noise label to obtain clean synthesized audio data, including: the vocoder converts the noise label into a noise feature; fuses the noise feature with the acoustic features to be synthesized; and performs audio synthesis based on the fused features to obtain clean synthesized audio data; The vocoder is trained with the acoustic features and noise training labels of the noisy training audio as input and the clean training audio as output; wherein the noisy training audio is synthesized by the clean training audio and the noisy training audio, and the noise training label corresponds to the noise level of the noisy training audio; the vocoder is trained with the acoustic features and noise training labels of the noisy training audio as input and the clean training audio as output, including: the vocoder is trained with the acoustic features of the clean training audio as input and the clean training audio as output to obtain a pre-trained model; the vocoder loads the pre-trained model and is trained with the acoustic features and noise training labels of the noisy training audio as input and the clean training audio as output.

2. The noise reduction method according to claim 1, characterized in that: The vocoder includes a self-noise reduction codec and a noise reduction universal vocoder, The vocoder takes the acoustic features of the noisy training audio and the noise training label as input and the clean training audio as output, and is trained to include: The self-denoising codec takes the acoustic features of the noisy training audio as input and the acoustic features of the clean training audio as output, and the denoising universal vocoder takes the noisy training labels and the intermediate acoustic features output by the self-denoising codec as input and the clean training audio as output, and performs joint training to obtain the vocoder.

3. The noise reduction method according to claim 1, characterized in that: When the vocoder comprises an autoregressive noise reduction universal vocoder, The vocoder takes the acoustic features of the noisy training audio and the noise training label as input and the clean training audio as output, and is trained to include: A baseline vocoder is trained using the acoustic features of the clean training audio as input and the clean training audio as output; The baseline vocoder performs audio synthesis based on the acoustic features of the clean training audio to obtain clean synthesized audio; The autoregressive denoising universal vocoder is trained by taking the acoustic features of the noisy training audio and the noise training label as input and the clean synthetic audio as output.

4. The noise reduction method according to claim 3, characterized in that: The autoregressive denoising universal vocoder takes the acoustic features of the noisy training audio and the noise training label as input and the clean synthesized audio as output, and is trained to include: The autoregressive noise reduction universal vocoder and the baseline vocoder perform at least one loss calculation inside the vocoder.

5. The noise reduction method according to claim 3, characterized in that: The model structure of the baseline vocoder and the autoregressive denoising universal vocoder is the same.

6. The noise reduction method according to claim 1, characterized in that: The method for obtaining the noise-added training audio includes: Acquire the clean training audio and the noisy training audio; In a random selection manner, adding noise to the clean training audio by using the noise training audio to obtain the noisy training audio; and / or, According to the preset noise adding intensity, the noise training audio is used to add noise to the clean training audio to obtain the noisy training audio.

7. The noise reduction method according to claim 1, characterized in that: The method for obtaining the noise training label includes: The corresponding noise training label is generated according to the noise intensity of the noise training audio.

8. The noise reduction method according to claim 1, characterized in that: The vocoder includes a self-noise reduction codec, Before fusing the noise feature with the acoustic feature to be synthesized, the method further includes: The self-noise reduction codec encodes the acoustic features to be synthesized to obtain intermediate acoustic features; The fusing process of the noise feature with the acoustic feature to be synthesized includes: The noise feature is fused with the intermediate acoustic feature.

9. The noise reduction method according to claim 1, characterized in that: The vocoder further comprises a speaker encoder, and the method further comprises: The speaker encoder is used to process reference audio data to obtain target features, and the reference audio data is obtained by collecting the target; The audio synthesis is performed based on the fused features to obtain clean synthesized audio data, including: Audio synthesis is performed based on the fused features and the target features to obtain clean synthesized audio data.

10. A vocoder, characterized in that: include: An acquisition module, used to acquire the acoustic features to be synthesized and the noise labels, wherein the noise labels correspond to the noise reduction degree of the acoustic features to be synthesized; The noise reduction synthesis module is used to perform audio synthesis based on the acoustic features to be synthesized and the noise labels to obtain clean synthesized audio data, wherein the vocoder performs audio synthesis based on the acoustic features to be synthesized and the noise labels to obtain clean synthesized audio data, including: the vocoder converts the noise labels into noise features; fuses the noise features with the acoustic features to be synthesized; and performs audio synthesis based on the fused features to obtain clean synthesized audio data; The vocoder is trained with the acoustic features and noise training labels of the noisy training audio as input and the clean training audio as output; wherein the noisy training audio is synthesized by the clean training audio and the noisy training audio, and the noise training label corresponds to the noise level of the noisy training audio; the vocoder is trained with the acoustic features and noise training labels of the noisy training audio as input and the clean training audio as output, including: the vocoder is trained with the acoustic features of the clean training audio as input and the clean training audio as output to obtain a pre-trained model; the vocoder loads the pre-trained model and is trained with the acoustic features and noise training labels of the noisy training audio as input and the clean training audio as output.

11. An electronic device, characterized in that: The electronic device comprises a memory and a processor coupled to each other, the memory is used to store program data, and the processor is used to execute the program data to implement the method according to any one of claims 1-9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program data, and when the program data is executed by a processor, it is used to implement the method according to any one of claims 1 to 9.

Images