A method and apparatus for vocoder training and speech synthesis
Patent Information
- Application Number
- CN202310624574.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-29
- Publication Date
- 2026-09-01
- Estimated Expiration
- 2043-05-29
AI Technical Summary
这些方法虽然能在一定程度上提高了模型的推理速度，但是在缺乏硬件资源的场景中，仍然很难做到快速语音合成
[0037] (1) A method is provided to effectively avoid a significant reduction in synthesized sound quality while reducing the number of model parameters.
Smart Images
Figure CN116631371B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of deep learning technology, and more particularly to the field of speech technology, specifically a method and apparatus for vocoder training and speech synthesis. Background Technology
[0002] TTS stands for Text-To-Speech, a technology that transforms input text into fluent, natural-sounding audio. It's primarily used in interactive applications such as car navigation, voice assistants, electronic dictionaries, and audiobook reading.
[0003] Currently, in the field of speech synthesis, fast speech synthesis refers to synthesizing speech in a very short time, typically requiring real-time performance at the level of hundreds of milliseconds. This necessitates algorithms and system design that are both highly efficient and have low latency to ensure rapid response times. However, due to cost constraints, most embedded devices lack hardware accelerators such as GPUs or TPUs to accelerate model inference, and therefore can only use CPUs for inference. Synthesizing a single long audio segment using a CPU often takes several seconds, resulting in significant inference latency.
[0004] The common speech synthesis process mainly consists of two steps. The first step is to use an acoustic model to transform the input text into corresponding acoustic features. The second step is to use a neural vocoder to convert the corresponding acoustic features into corresponding audio. In these two steps, the majority of the inference time is concentrated on the neural vocoder. To accelerate the audio inference process, knowledge from the field of signal processing is generally used to reduce the computational load of the neural vocoder, thereby improving the inference speed.
[0005] LPCNet uses linear prediction coefficients to predict the next waveform, Multiband-Melgan divides the waveform into multiple subbands for inference, and iSTFTNet converts the waveform into a complex spectrum for inference. While these methods can improve the inference speed of the model to some extent, they are still difficult to achieve fast speech synthesis in scenarios with limited hardware resources. Summary of the Invention
[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and apparatus for vocoder training and speech synthesis.
[0007] To achieve the above objectives, the technical solution of the present invention is as follows:
[0008] In a first aspect, the present invention provides a vocoder training method, comprising:
[0009] Teacher model training steps to train a teacher model, which is used for vocoder training;
[0010] The vocoder training process involves training the vocoder based on the teacher model and the network structure of the vocoder to obtain the vocoder.
[0011] Furthermore, the teacher model includes an encoder, a separator, a basis matrix transformation, and an overlapping layer; the encoder is used to encode noisy speech into hidden features, the separator predicts a mask through the hidden features, and then the mask is multiplied with the hidden features to separate the features of the predicted clean speech from the hidden features through the mask. After obtaining the features of the predicted clean speech, the features are transformed through the basis matrix to change the dimension of the features, and finally the predicted clean speech is obtained through the overlapping layer.
[0012] The network structure of the vocoder includes an upsampling module, a transformation module, a basis matrix transformation, and an overlapping additive layer. The upsampling module is used to expand the acoustic features to a high-dimensional space, the transformation module is used to transition the features in the high-dimensional space to temporal features, and the basis matrix transformation is used to change the dimension of the temporal features so as to obtain the predicted clean speech in the overlapping additive layer.
[0013] Furthermore, the training of the vocoder based on the network structure of the teacher model and the vocoder includes:
[0014] The network structure based on the teacher model and vocoder is jointly trained through knowledge distillation and generative adversarial networks, with the training loss being:
[0015]
[0016] Among them, L KD1 It measures the difference between the features of two basis matrices before transformation, L KD2 It measures the difference between two predicted clean speech or the difference between the outputs of the teacher model and the vocoder network structures. The loss is calculated using the short-time Fourier transform loss, as shown in the following formula:
[0017]
[0018] Where f(x) refers to the output of the vocoder, and wav refers to the output of the teacher model;
[0019] L dis The value measured is the difference between the discriminator's output and 1. The loss calculation uses the mean squared error loss.
[0020]
[0021] Here, dis(f(x)) is the output of the discriminator with input f(x). The closer the result is to 1, the more realistic the output of the vocoder is, and the closer it is to 0, the less realistic the output of the vocoder is.
[0022] Furthermore, the training loss L of the vocoder is... KD2 and L dis Revised to:
[0023]
[0024]
[0025] The vocoder no longer predicts the audio directly. Instead, the audio prediction is generated by subtracting the output of the vocoder from f(0), where f(0) is the result obtained after the zero vector is input into the vocoder.
[0026] Furthermore, the method also includes a reasoning optimization step:
[0027] Pre-calculate f(0); during pre-calculation, input a zero vector sequence that meets the length requirement, save the output, and during formal inference, truncate the long sequence f(0) into three parts according to the length of the actual f(0) corresponding to f(x), and concatenate the two parts related to the value of f(0) to obtain the value of the actual f(0).
[0028] Furthermore, the inference optimization step also includes:
[0029] The parameters and activation values of the vocoder are quantized and converted into integers or fixed-point numbers to reduce the computational load and memory bandwidth of floating-point operations.
[0030] Furthermore, the receptive field of the vocoder is calculated in advance before vocoder training, so that all audio in the training set is muted before vocoder training.
[0031] Secondly, the present invention provides a speech synthesis method, comprising:
[0032] Acquire the acoustic features to be processed;
[0033] The acoustic features are input into a vocoder trained by the method described above, and the synthesized speech is output.
[0034] Thirdly, the present invention provides a vocoder training apparatus, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the methods described above.
[0035] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the methods described above.
[0036] Compared with the prior art, the advantages of this invention are as follows:
[0037] (1) A method is provided to effectively avoid a significant reduction in synthesized sound quality while reducing the number of model parameters.
[0038] (2) By pre-calculating some values, the amount of computation required during actual model inference is reduced. The optimized inference speed is faster than the model mentioned in Part 1.
[0039] (3) A solution is provided for the contamination of the synthesized audio end after vocoder quantization. Attached Figure Description
[0040] Figure 1 The following is the overall flow of the vocoder training method provided in the embodiments of the present invention;
[0041] Figure 2 A schematic diagram of the components of the teacher model;
[0042] Figure 3 This is a schematic diagram of the network structure of a vocoder.
[0043] Figure 4 This is a diagram illustrating the training process of a vocoder network.
[0044] Figure 5 The spectrum diagram shows the noise level with single-frequency noise.
[0045] Figure 6 The spectrum diagram is shown without single-frequency noise.
[0046] Figure 7 Here is an example of a model receptive field with an input length of 7;
[0047] Figure 8 Here is an example of a model receptive field with an input length of 10;
[0048] Figure 9 A diagram illustrating the pre-calculation of f(0), truncation, and splicing operations;
[0049] Figure 10 This is a flowchart of a speech synthesis method provided in an embodiment of the present invention;
[0050] Figure 11 This is a schematic diagram of the composition of the vocoder training device provided in an embodiment of the present invention. Detailed Implementation
[0051] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.
[0052] Example 1:
[0053] The vocoder training method provided in this embodiment mainly consists of three steps, as follows: Figure 1 As shown, the process involves three steps: teacher model training, vocoder training, and inference optimization. Among these, teacher model training and vocoder training are the foundational steps, while inference optimization is a step that further optimizes the vocoder.
[0054] In the teacher model training step, the teacher model uses ConvTasNet. ConvTasNet is a speech separation model, and its role here is to separate clean speech from noisy speech. Figure 2 As shown, it mainly consists of the following four parts: encoder, separator, base matrix transformation and overlap and add (OLA) layer.
[0055] The main function of the encoder is to encode noisy speech into hidden features. The separator predicts a mask using the hidden features, and then the mask is multiplied with the hidden features. The purpose is to separate the features of the predicted clean speech from the hidden features through the mask. After obtaining the features of the predicted clean speech, the features are transformed by the basis matrix to change the dimension of the features. Finally, the predicted clean speech is obtained by passing the overlapping and summing layers.
[0056] After training is complete, the weights of the teacher model are frozen and used for training the vocoder.
[0057] In the vocoder training step, the vocoder is trained based on the teacher model and the vocoder's network structure to obtain the vocoder. The vocoder's network structure references Basis-Melgan, such as... Figure 3 As shown, the architecture includes an upsampling module, a transformation module, a basis matrix transformation, and an overlapping layer. The upsampling module expands the acoustic features to a high-dimensional space. The transformation module is responsible for transitioning the high-dimensional features to temporal features. The basis matrix transformation changes the dimension of the features to facilitate the transformation into clean predicted speech at the OLA layer. To reduce the number of parameters in the model, most of the convolutions in the original network are replaced with depthwise separable convolutions.
[0058] Training is conducted jointly through knowledge distillation and generative adversarial networks. Figure 4 This refers to the training process of the network.
[0059] The training loss is:
[0060]
[0061] α1 and α2 are hyperparameters that need to be set before training; α1 is used to adjust the hyperparameters obtained through the feature loss L. KD1 The effect on the student model, α2, was used to adjust the loss L through the discriminator. dis The impact on the student model. L KD1 It measures the difference between the features of two basis matrices before transformation, and the loss is calculated using the mean squared error loss (MSE Loss); L KD2 It measures the difference between two predicted clean speech or the difference between the outputs of the teacher model and the vocoder network structures. The loss is calculated using the Short Time Fourier Transform Loss (STFT Loss), as shown in the formula below, where f(x) refers to the output of the vocoder and wav refers to the output of the teacher model.
[0062]
[0063] L dis The value measured is the difference between the discriminator's output and 1. The loss calculation uses the mean squared error loss.
[0064]
[0065] Here, dis(f(x)) is the output of the discriminator with input f(x). The closer the result is to 1, the more realistic the output of the vocoder is, and the closer it is to 0, the less realistic the output of the vocoder is.
[0066] Since the vocoder lacks speech modeling capabilities in the early stages of training, α1 is set to 1 and α2 to 0. Therefore, distillation is used to aid and accelerate model training. After a certain number of iterations, α1 is set to 0 and α2 to 1, and the realism of the synthesized audio is improved through generative adversarial network training.
[0067] It is important to note that during the training of vocoders based on audio decomposition (based on OLA, iSTFT, or multiband), the synthesized audio is prone to single-frequency noise, which is a very obvious horizontal line on a single frequency, affecting the listening experience. Figure 5 There is single-frequency noise and Figure 6 Two examples of no single-frequency noise.
[0068] Synthesized audio is prone to single-frequency noise. Common solutions include training more steps or adding a discriminator to mitigate this phenomenon. However, as the number of vocoder parameters decreases, the model's fitting ability gradually weakens. Simply increasing training time becomes difficult to address this issue. Therefore, this paper proposes a novel solution: reducing the vocoder training loss L... KD2 and L dis Revised to:
[0069]
[0070]
[0071] The main difference from common solutions is that the vocoder here no longer directly predicts the audio. Instead, the audio is generated by subtracting the vocoder's output from f(0), where f(0) is the result obtained after inputting the zero vector into the vocoder. Experiments show that predicting the difference is easier to train than predicting the true value.
[0072] However, the model trained using the above method has a major drawback: the inference time doubles. This is because after inputting an acoustic feature, the vocoder needs to perform two consecutive inferences to obtain f(x) and f(0), and then subtract f(x) from f(0) to obtain the predicted audio.
[0073] Based on this, after the vocoder training is complete, an inference optimization step can be performed. To improve inference speed, f(0) needs to be calculated in advance. However, in reality, the input zero vector is a variable-length sequence with the same length as the input feature x. Therefore, it is necessary to find the output pattern of the zero-bias vector. Since the vocoder used here mainly uses convolution and transposed convolution, it only has local perception capabilities and not global perception. Due to the limited receptive field of the network, each point output by the network is only related to certain frames of its input acoustic features and is independent of other frames outside the receptive field. As follows... Figure 7 As shown, the dashed lines indicate which values affect the current value. Here, the model has an input length of 7 and an output length of 9. Output 7 is only related to inputs 5-7; changes to other inputs will not affect the output of output 7. Similarly, output 9 is only related to input 7 and is independent of other inputs.
[0074] When the input length increases, such as changing the input length from 7 to 10, then output 9 will change from being related only to input 7 to being related to inputs 7-9.
[0075] Furthermore, when the input sequence is a sequence of all zeros, then... Figure 7 Input number 7 and Figure 8The 10th input is essentially the same, both being a sequence of all zeros at the very end. Due to the limited receptive field of the model, Figure 7 The 9th output corresponding to input number 7 ( Figure 7 The two dashed lines on the middle number 9 will meet Figure 8 The 10th input corresponds to the 12th output ( Figure 8 The two dotted lines on number 12 are exactly the same. Based on this conclusion, we only need to... Figure 8 By concatenating outputs 1-7 with outputs 11 and 12, it can be combined with... Figure 7 The outputs of numbers 1-9 are completely identical.
[0076] Therefore, during pre-calculation, only a sufficiently long sequence of zero vectors needs to be input, and the output saved. During formal inference, the long sequence f(0) is truncated into three parts A, B, and C according to the length of the actual f(0) corresponding to f(x), and A and C are concatenated to immediately obtain the value of the actual f(0), such as... Figure 9 As shown. The truncation range depends on the receptive field size of the vocoder. Finally, the predicted audio is obtained by subtracting the output of f(x) from the actual f(0). Thus, by using this method for vocoder training and inference optimization, the model parameters can be reduced as much as possible and the inference speed of the model can be significantly improved with only a slight decrease in sound quality.
[0077] Furthermore, after model training is complete, the parameters and activation values of the neural vocoder model can be quantized. Quantization converts these values into integers or fixed-point numbers, reducing the computational cost and memory bandwidth required for floating-point operations, thus further improving inference speed. However, after quantization, variable-length inputs are converted to fixed-length inputs. Fixed-length inputs are padded with numerous zeros at the end of the original input. (This can be compared...) Figure 7 and Figure 8 It can be seen that inconsistent input lengths lead to inconsistent output results. Since fixed-length inputs only contaminate the input end, and based on the receptive field of the vocoder, the output will also only be contaminated at the end.
[0078] To reduce the impact of fixed-length input on the output, the receptive field of the vocoder can be calculated before training, i.e., the farthest contamination area that each input point can affect (e.g., the farthest influence range of each input point of the vocoder used here is 1440 points of the output audio, which is approximately 1440 / 16000 Hz = 0.09 s of audio at a sampling rate of 16000 Hz). Then, a 90-millisecond silence can be added to the end of all audio in the training set before model training. After fixed-length inference, the last 90 milliseconds of audio in the synthesized audio can be truncated. This ensures that the final audio will not be affected by fixed-length inference, because the area affected by fixed-length inference will only be generated in the last 1440 points of the audio, and these 1440 points (90 milliseconds of audio) have been truncated.
[0079] In summary, the vocoder training method provided in this embodiment has the following technical advantages compared with the prior art:
[0080] (1) A method is provided to effectively avoid a significant reduction in synthesized sound quality while reducing the number of model parameters.
[0081] (2) By pre-calculating some values, the amount of computation required during actual model inference is reduced. The optimized inference speed is faster than the model mentioned in Part 1.
[0082] (3) A solution is provided for the contamination of the synthesized audio end after vocoder quantization.
[0083] Example 2:
[0084] Please see Figure 10 This illustration shows the flow of an embodiment of the speech synthesis method provided in this disclosure. The speech synthesis method may include the following steps:
[0085] Acquire the acoustic features to be processed;
[0086] The acoustic features to be processed are input into a vocoder, which outputs synthesized speech.
[0087] The vocoder can be generated using the method described in Example 1. For the specific generation process, please refer to the relevant description in Example 1, which will not be repeated here.
[0088] It should be noted that the speech synthesis method in this embodiment can be used to test the vocoder generated in Embodiment 1 above. The vocoder can then be continuously optimized based on the test results. This method can also be a practical application of the vocoders generated in the above embodiments. Using the vocoders generated in the above embodiments for speech synthesis helps improve the sound quality of the synthesized speech, such as removing vibrato and muffled sounds.
[0089] Example 3:
[0090] See Figure 11 As shown, the vocoder training apparatus provided in this embodiment includes a processor 111, a memory 112, and a computer program 113, such as a vocoder training program, stored in the memory 112 and executable on the processor 111. When the processor 111 executes the computer program 113, it implements the steps of Embodiment 1 described above, for example... Figure 1 The steps are shown.
[0091] For example, the computer program 113 may be divided into one or more modules / units, which are stored in the memory 112 and executed by the processor 111 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 113 in the vocoder training device.
[0092] The vocoder training device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The vocoder training device may include, but is not limited to, a processor 111 and a memory 112. Those skilled in the art will understand that... Figure 11 This is merely an example of a vocoder training device and does not constitute a limitation on the vocoder training device. It may include more or fewer components than shown, or combine certain components, or different components. For example, the vocoder training device may also include input / output devices, network access devices, buses, etc.
[0093] The processor 111 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.
[0094] The memory 112 can be an internal storage unit of the vocoder training device, such as the hard disk or memory of the vocoder training device. The memory 112 can also be an external storage device of the vocoder training device, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the vocoder training device. Furthermore, the memory 112 can include both internal storage units and external storage devices of the vocoder training device. The memory 112 is used to store the computer program and other programs and data required by the vocoder training device. The memory 112 can also be used to temporarily store data that has been output or will be output.
[0095] Example 4:
[0096] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described in Embodiment 1.
[0097] The computer-readable medium shown can be any means that can contain, store, communicate, propagate, or transmit a program for use in or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, for example, by optically scanning the paper or other medium, then editing, interpreting, or otherwise processing it as necessary to obtain the program electronically, and then storing it in computer memory.
[0098] The above embodiments are merely illustrative of the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made based on the essence of the content of the present invention should be covered within the scope of protection of the present invention.
Claims
1. A vocoder training method, characterized in that, include: Teacher model training steps to train a teacher model, which is used for vocoder training; The vocoder training steps involve training the vocoder based on the teacher model and the network structure of the vocoder to obtain the vocoder. The training loss L of the vocoder is modified as KD2 and L dis is modified as: The vocoder no longer predicts the audio directly. Instead, the audio is generated by subtracting the output of the vocoder from f(0), where f(0) is the result obtained after the zero vector is input into the vocoder. wav refers to the output of the teacher model; The training of the vocoder based on the network structure of the teacher model and the vocoder includes: The network structure based on the teacher model and vocoder is jointly trained through knowledge distillation and generative adversarial networks, with the training loss being: where a1 and a2 are hyperparameters to be set before training, L KD1 The difference between the features of the two base matrices before transformation is measured, L KD2 The difference between the two predicted clean speeches or the difference between the outputs of the network structures of the teacher model and the vocoder is measured, and the loss is calculated using the short-time Fourier transform loss, as shown in the following formula: Where f(x) refers to the output of the vocoder, and wav refers to the output of the teacher model; L dis The value measured is the difference between the discriminator's output and 1. The loss calculation uses the mean squared error loss. Where dis(f(x)) is the output of the discriminator with input f(x). The closer the result is to 1, the more realistic the output of the vocoder is, and the closer it is to 0, the less realistic the output of the vocoder is. Pre-calculate f(0); during pre-calculation, input a zero vector sequence that meets the length requirement, save the output, and during formal inference, cut the long sequence f(0) into three parts according to the length of the actual f(0) corresponding to f(x), and concatenate the two parts related to the value of f(0) to obtain the value of the actual f(0); Before training the vocoder, the receptive field of the vocoder is calculated in advance. A certain length of silence is added to the end of all audio in the training set before training the vocoder. After fixed-length inference, the audio corresponding to the last length in the synthesized audio is truncated.
2. The vocoder training method as described in claim 1, characterized in that, The teacher model includes an encoder, a separator, a basis matrix transformation, and an overlapping layer. The encoder encodes noisy speech into hidden features. The separator predicts a mask using the hidden features. The mask is then multiplied with the hidden features to separate the features of the predicted clean speech from the hidden features. After obtaining the features of the predicted clean speech, the features are transformed using the basis matrix to change their dimension. Finally, the predicted clean speech is obtained by passing the overlapping layer. The network structure of the vocoder includes an upsampling module, a transformation module, a basis matrix transformation, and an overlapping additive layer. The upsampling module is used to expand the acoustic features to a high-dimensional space, the transformation module is used to transition the features in the high-dimensional space to temporal features, and the basis matrix transformation is used to change the dimension of the temporal features so as to obtain the predicted clean speech in the overlapping additive layer.
3. The vocoder training method as described in claim 1, characterized in that, Also includes: The parameters and activation values of the vocoder are quantized and converted into integers or fixed-point numbers to reduce the computational load and memory bandwidth of floating-point operations.
4. A speech synthesis method, characterized in that, include: Acquire the acoustic features to be processed; The acoustic features are input into a vocoder trained by the method described in any one of claims 1-3, and synthesized speech is output.
5. A vocoder training apparatus, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 3.
6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 3.