Method for constructing vocoder, speech processing method and related device

Abstract

The application provides a vocoder construction method, a speech processing method and related equipment, comprising training a generator with a sample set, determining the trained generator; the generator and a discriminator trained with the generator are used as a vocoder; the training process comprises processing a first output audio corresponding to a first feature vector and a second output audio corresponding to a second feature vector respectively, determining a first time domain loss, a first frequency domain loss, a second time domain loss and a second frequency domain loss; processing a third output audio corresponding to the second feature vector, determining a third frequency domain loss; then calculating the total loss of the generator; and training the network of the generator based on the total loss. In the scheme, when the generator is trained, the total loss of the generator is calculated by calculating the time domain loss and the frequency domain loss of multiple resolutions; and the convergence speed of the vocoder training is accelerated by the above method, so as to improve the language processing quality of the vocoder.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to a method for constructing a vocoder, a speech processing method, and related equipment. Background Technology

[0002] The backend model for speech synthesis is implemented using an acoustic model and a vocoder. The acoustic vocoder converts the semantic information generated at the front end into audio features, and the vocoder converts the audio features into audio signals. A GAN-based vocoder consists of a generator and a discriminator. The generator converts the audio features into audio signals, and the discriminator determines how close the generator's output is to reality.

[0003] Currently, the generator training layer does not have an additional loss function, which makes the model converge slowly and requires more parameters to achieve good results, resulting in low generator quality. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method for constructing a vocoder, a speech processing method, and related equipment to solve the problems of slow convergence speed and poor speech processing effect of vocoders in the prior art.

[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] The first aspect of the present invention illustrates a method for constructing a vocoder, the method comprising:

[0007] Obtain a sample set, which refers to the audio features obtained by initializing audio data collected within a preset time period;

[0008] The generator is trained using a sample set to determine the trained generator.

[0009] The trained generator and the discriminator trained with the generator are combined to form a vocoder;

[0010] The specific process of training the generator is as follows:

[0011] If the number of upsampling layers and residual modules is 1, the first output audio corresponding to the first feature vector and the second output audio corresponding to the second feature vector are processed respectively to determine the first time domain loss, the first frequency domain loss, the second time domain loss and the second frequency domain loss. The first feature vector is obtained by the generator's upsampling layer processing the sample set, and the second feature vector is obtained by the generator's residual module processing the first feature vector. The number of upsampling layers and residual modules is equal.

[0012] The third output audio corresponding to the second feature vector is processed to determine the third frequency domain loss;

[0013] The generator's total loss is calculated based on the first time-domain loss, the first frequency-domain loss, the second time-domain loss, the second frequency-domain loss, and the third frequency-domain loss, and the generator's network is trained using the total loss.

[0014] When it is determined that the third frequency domain loss meets the preset requirements, the network of the generator obtained by training with the total loss is determined to be the generator that has been trained.

[0015] Optionally, the step of processing the first output audio corresponding to the first feature vector and the second output audio corresponding to the second feature vector to determine the first time-domain loss, the first frequency-domain loss, the second time-domain loss, and the second frequency-domain loss includes:

[0016] The upsampling layer processes the sample set to obtain the first feature vector;

[0017] The residual module processes the first feature vector to obtain the second feature vector;

[0018] The first feature vector is processed using the first convolutional layer plus activation function Tanh layer of the generator to determine the first output audio corresponding to the first feature vector;

[0019] The second feature vector is processed using the second convolutional + activation function Tanh layer of the generator to determine the second output audio corresponding to the second feature vector;

[0020] The first output audio is processed in both the frequency domain and the time domain to determine the first time domain loss and the first frequency domain loss.

[0021] The second output audio is processed in both the frequency domain and the time domain to determine the second time domain loss and the second frequency domain loss.

[0022] Optional, also includes:

[0023] If the number of upsampling layers and residual modules is N, the i-th upsampling layer processes the second feature vector output by the (i-1)-th residual module to obtain the first feature vector. Where i equals 1, the i-th upsampling layer processes the sample set to obtain the first feature vector, N is a positive integer greater than or equal to 1, and i is less than or equal to N.

[0024] The i-th residual module processes the first feature vector output by the i-th upsampling layer to obtain the second feature vector;

[0025] The first feature vector is processed using the first convolutional layer plus activation function Tanh layer of the generator to determine the first output audio corresponding to the first feature vector;

[0026] The second feature vector is processed using the second convolutional + activation function Tanh layer of the generator to determine the second output audio corresponding to the second feature vector;

[0027] The first output audio is processed in both the frequency domain and the time domain to determine the first time domain loss and the first frequency domain loss.

[0028] The second output audio is processed in both the frequency domain and the time domain to determine the second time domain loss and the second frequency domain loss.

[0029] Optionally, the step of processing the third output audio corresponding to the second feature vector to determine the third frequency domain loss includes:

[0030] The second feature vector is processed using the third convolutional layer with activation function Tanh of the generator to determine the third output audio corresponding to the second feature vector;

[0031] Perform a short-time Fourier transform on the third output audio to determine the third frequency domain loss.

[0032] Optionally, if the number of upsampling layers and residual modules is N, if i is not equal to 1, the i-th upsampling layer processes the feature vector output by the (i-1)-th residual module to obtain the first feature vector, where if i is equal to 1, the i-th upsampling layer processes the sample set to obtain the first feature vector, N is a positive integer greater than or equal to 1, and i is less than or equal to N.

[0033] The i-th residual module processes the first feature vector output by the i-th upsampling layer to obtain the second feature vector;

[0034] The second feature vector obtained by the residual module with the i-th value equal to N is processed by the third convolution + activation function Tanh layer of the generator to determine the third output audio corresponding to the second feature vector;

[0035] Perform a short-time Fourier transform on the third output audio to determine the third frequency domain loss.

[0036] Optionally, determining the trained generator based on the third frequency domain loss includes:

[0037] When the curve drawn by the third frequency domain loss and the recorded historical third frequency domain loss reaches the preset expectation, the network of the generator obtained by training with the total loss is determined to be the trained generator, and the number of historical third frequency domain losses needs to reach a preset value.

[0038] If the curve plotted by the third frequency domain loss and the recorded historical third frequency domain loss does not meet the preset expectation, the process returns to the step of processing the first output audio corresponding to the first feature vector and the second output audio corresponding to the second feature vector to determine the first time domain loss, the first frequency domain loss, the second time domain loss and the second frequency domain loss.

[0039] A second aspect of this invention discloses a speech processing method applicable to a vocoder constructed using the training method for the vocoder shown in the first aspect of this invention, the method comprising:

[0040] The audio features are input into the vocoder, which are obtained by converting the acquired raw audio by the acoustic vocoder.

[0041] Based on the vocoder, the audio features are processed to output the target audio.

[0042] A third aspect of the present invention discloses an apparatus for constructing a vocoder, the apparatus comprising:

[0043] The acquisition unit is used to acquire a sample set, which refers to the audio features obtained by initializing the audio data collected within a preset time period.

[0044] A training unit is used to train the generator using a sample set and determine the trained generator.

[0045] A generation unit is used to combine the trained generator and the discriminator trained with the generator into a vocoder.

[0046] The training unit is specifically configured to: if the number of upsampling layers and residual modules is one, process the first output audio corresponding to the first feature vector and the second output audio corresponding to the second feature vector, respectively, to determine a first time-domain loss, a first frequency-domain loss, a second time-domain loss, and a second frequency-domain loss. The first feature vector is obtained by the generator's upsampling layer processing the sample set, and the second feature vector is obtained by the generator's residual module processing the first feature vector. The number of upsampling layers and residual modules is equal. Process the third output audio corresponding to the second feature vector to determine a third frequency-domain loss. Calculate the generator's total loss based on the first time-domain loss, the first frequency-domain loss, the second time-domain loss, the second frequency-domain loss, and the frequency-domain loss, and use the total loss to train the generator's network. When the third frequency-domain loss meets a preset requirement, determine the generator network trained with the total loss as the trained generator.

[0047] A fourth aspect of the present invention discloses a speech processing apparatus, comprising a vocoder constructed using the vocoder training method shown in the first aspect of the present invention, the apparatus comprising:

[0048] An input module is used to input audio features into the vocoder, wherein the audio features are obtained by the acoustic vocoder converting the acquired raw audio.

[0049] The vocoder is used to perform speech processing on the audio features and output the target audio.

[0050] A fifth aspect of the present invention discloses an electronic device for running a program, wherein the program, when running, executes a method for constructing a vocoder as shown in a first aspect of the present invention, or executes a speech processing method as shown in a second aspect of the present invention.

[0051] A sixth aspect of the present invention discloses a computer storage medium comprising a stored program, wherein, when the program is executed, the device on which the storage medium is located executes a vocoder construction method as shown in the first aspect of the present invention, or executes a speech processing method as shown in the second aspect of the present invention.

[0052] Based on the above embodiments of the present invention, a method for constructing a vocoder, a speech processing method, and related devices are provided. The method includes: acquiring a sample set; training a generator using the sample set to determine the trained generator; and combining the trained generator and a discriminator trained with the generator to form a vocoder. Specifically, the process of training the generator is as follows: if the number of upsampling layers and residual modules is one, processing the first output audio corresponding to the first feature vector and the second output audio corresponding to the second feature vector, respectively, to determine a first time-domain loss, a first frequency-domain loss, a second time-domain loss, and a second frequency-domain loss. The first feature vector is obtained by the generator's upsampling layer processing the sample set, and the second feature vector is obtained by the generator's residual module processing the first feature vector. The number of upsampling layers and residual modules are equal. Processing the third output audio corresponding to the second feature vector to determine a third frequency-domain loss; calculating the generator's total loss based on the first time-domain loss, the first frequency-domain loss, the second time-domain loss, the second frequency-domain loss, and the frequency-domain loss; training the generator's network based on the total loss, thereby confirming the trained generator. In this scheme, when training the generator, multi-resolution time-domain loss and frequency-domain loss are calculated, that is, a multi-level resolution loss function is introduced into the generator, and then the total loss of the generator is calculated. The above method can accelerate the convergence speed of vocoder training and improve the language processing quality of the vocoder. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0054] Figure 1 This is a flowchart illustrating a method for constructing a vocoder according to an embodiment of the present invention;

[0055] Figure 2 This is a flowchart illustrating the specific training process of the generator according to an embodiment of the present invention;

[0056] Figure 3 This is a schematic diagram of the generator architecture shown in an embodiment of the present invention;

[0057] Figure 4 This is a schematic diagram illustrating the process of training a specific generator according to an embodiment of the present invention;

[0058] Figure 5 This is a schematic flowchart illustrating a speech processing method according to an embodiment of the present invention;

[0059] Figure 6 This is a schematic diagram of a vocoder construction device according to an embodiment of the present invention;

[0060] Figure 7 This is a schematic diagram of the structure of a voice processing device according to an embodiment of the present invention. Detailed Implementation

[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0063] It should be noted that the descriptions involving "first," "second," etc., in this invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0064] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0065] As the background technology indicates, speech processing is implemented using acoustic models and vocoders. Acoustic vocoders convert semantic information generated by the front end into audio features, and vocoders convert these audio features into audio signals. GAN-based vocoders consist of a generator and a discriminator. The generator converts audio features into audio signals, and the discriminator determines how closely the generator's output approximates reality.

[0066] Currently, the generator training layer lacks an additional loss function, resulting in slow model convergence and requiring more parameters for training to achieve good results, thus leading to low generator quality. Furthermore, existing vocoders can only generate audio at one sampling rate. When synthesizing low-sampling-rate audio, it is necessary to downsample the synthesized high-sampling-rate audio or retrain a low-sampling-rate vocoder.

[0067] In this embodiment of the invention, during generator training, multi-resolution temporal and frequency domain losses are calculated, i.e., a multi-level resolution loss function is introduced into the generator to calculate the total loss of the generator. Although a convolutional + tanh layer is added during vocoder training to calculate the loss, the time consumed in synthesizing audio at the same sampling rate during vocoder inference remains unchanged. This method accelerates the convergence speed of vocoder training, thereby improving the language processing quality of the vocoder. Furthermore, the vocoder can generate audio at multiple sampling rates without retraining, reducing inference time.

[0068] See Figure 1 The diagram below illustrates a flowchart of a method for constructing a vocoder according to an embodiment of the present invention. The method includes:

[0069] Step S101: Obtain the sample set.

[0070] In step S101, the sample set is audio features in Mel spectrum format. The sample set refers to the audio features obtained by initializing audio data collected within a preset time period.

[0071] In the specific implementation step S101, training-ready audio data is collected within a preset time period, and audio features of the audio data are extracted; then, the audio features are converted into corresponding Mel Spectograms and used as a sample set.

[0072] Step S102: Train the generator using the sample set to determine the trained generator.

[0073] It should be noted that the specific implementation step S102, i.e. the process of specifically training the generator, is as follows: Figure 2 As shown.

[0074] In the embodiments of this application, based on Figure 2 The diagram illustrates the specific training process of the generator. Correspondingly, this application also discloses a schematic diagram of the generator's architecture, such as... Figure 3 As shown.

[0075] If the number of upsampling layers and residual modules is one, the specific training process of the generator includes the following steps:

[0076] Step S201: Process the first output audio corresponding to the first feature vector and the second output audio corresponding to the second feature vector respectively to determine the first time domain loss, the first frequency domain loss, the second time domain loss and the second frequency domain loss.

[0077] In step S201, the first feature vector is obtained by the generator's upsampling layer processing the sample set, and the second feature vector is obtained by the generator's residual stack processing the first feature vector. The number of upsampling layers and residual stacks are equal.

[0078] It should be noted that the Upsampling Layer corresponds to the Residual Stack in the same layer.

[0079] Optionally, before processing the sample set in the upsampling layer, it is necessary to utilize... Figure 3 The first convolutional layer, ConvLayer, performs convolution processing on the sample set and outputs feature data, namely audio features.

[0080] It should be noted that the feature data is vector data.

[0081] The specific implementation process of step S201 includes the following steps:

[0082] Step S11: The upsampling layer processes the sample set to obtain the first feature vector.

[0083] In the specific implementation step S11, the Upsampling Layer collects the audio features obtained by the previous convolutional layer ConvLayer, namely the first feature vector, and sends it to the first convolutional + activation function Tanh layer a1.

[0084] Optionally, the Upsampling Layer sends the acquired first feature vector to the corresponding residual stack.

[0085] Step S12: The residual module processes the first feature vector to obtain the second feature vector.

[0086] In the specific implementation step S12, the residual stack module encodes the first feature vector and sends the encoded second feature vector to the second convolution + activation function Tanh layer a2.

[0087] Step S13: Process the first feature vector using the first convolutional Conv+ activation function Tanh layer of the generator to determine the first output audio corresponding to the first feature vector.

[0088] Step S14: Process the second feature vector using the second convolutional + activation function Tanh layer a2 of the generator to determine the second output audio corresponding to the second feature vector.

[0089] In the specific implementation of steps S13 and S14, the convolution Conv in the first convolution Conv+ activation function Tanh layer reduces the feature dimension of the first feature vector to 1, and changes the numerical range of the feature to -1 to 1 through the Tanh layer, so as to output the first audio with the numerical range of the feature dimension in (-1,1); the convolution Conv in the second convolution+ activation function Tanh layer a2 reduces the feature dimension of the second feature vector to 1, and changes the numerical range of the feature to -1 to 1 through the Tanh layer, so as to output the second audio with the numerical range of the feature dimension in (-1,1).

[0090] It should be noted that both the first and second output audio are low-sampling-rate audio.

[0091] Step S14: Perform frequency domain processing and time domain processing on the first output audio respectively to determine the first time domain loss and the first frequency domain loss.

[0092] It should be noted that the first time-domain loss consists of energy loss, time loss, and phase loss.

[0093] The specific content of S14 is as follows: First, substitute the first output audio and the downsampled real audio Target WavDownsampling into formula (1) to calculate the frequency domain loss, and obtain the first frequency domain loss L. FD Next, the first output audio and the downsampled real audio (Target Wav Downsampling) are subjected to time domain loss calculation to obtain the first time domain loss.

[0094] Specifically, the time domain loss is calculated by combining the first output audio and the downsampled real audio (Target Wav Downsampling). The implementation process for obtaining the first time domain loss is as follows:

[0095] Substitute the first output audio and the real audio acquired by TargetWav Downsampling into formula (2) to calculate the energy loss and obtain the energy loss; substitute the first output audio and the real audio acquired by TargetWav Downsampling into formula (3) to calculate the time loss and obtain the time loss; substitute the first output audio and the real audio acquired by TargetWav Downsampling into formula (4) to calculate the phase loss and obtain the phase loss; substitute the energy loss obtained by formula (2), the time loss obtained by formula (3), and the phase loss obtained by formula (4) into formula (5) to calculate the sum of the energy loss, time loss, and phase loss as the first time domain loss.

[0096] Formula (1):

[0097]

[0098] Among them, L FD For the first frequency domain loss, This represents the output audio of the i-th convolutional + Tanh layer, i.e., the first output audio, y i Represents actual audio, L stft To calculate the frequency domain loss between the first output audio generated by the generator and the real audio, Let i be the expected value used to calculate the frequency domain loss. Furthermore, since the sum of the number of upsampling layers and residual stacks is L, i is 1.

[0099] It should be noted that L is a positive integer greater than or equal to 2.

[0100] Formula (2):

[0101]

[0102] Among them, L e For energy loss, This represents the output audio of the i-th convolutional + Tanh layer, i.e., the first output audio, y i Represents real audio. Let i be the expected value used to calculate the energy loss. Furthermore, since the sum of the number of Upsampling Layers and Residual Stacks is L, i is 1.

[0103] Formula (3):

[0104]

[0105] Among them, L t Due to time loss, This represents the output audio of the i-th convolutional + Tanh layer, i.e., the first output audio, y i Represents real audio. Let i be the expected value used to calculate the time loss. Furthermore, since the sum of the number of Upsampling Layers and Residual Stacks is L, i is 1.

[0106] Formula (4):

[0107]

[0108] Among them, L P For phase loss, This represents the output audio of the i-th convolutional + Tanh layer, i.e., the first output audio, y i Represents real audio. To calculate the expected value of the phase loss, for First-order difference, Δy i For y i The first-order difference. Furthermore, since the sum of the number of upsampling layers and residual stacks is L, i is 1.

[0109] Formula (5):

[0110] L TD =L e +L t +L p (5)

[0111] Among them, L TD For the first time-domain loss, L P For phase loss, L t For time loss, L e This is due to energy loss.

[0112] Step S15: Perform frequency domain processing and time domain processing on the second output audio respectively to determine the second time domain loss and the second frequency domain loss.

[0113] It should be noted that the second time-domain loss consists of energy loss, time loss, and phase loss.

[0114] The specific content of S15 is as follows: First, substitute the second output audio and the real audio collected by Target WavDownsampling into the above formula (1) to calculate the frequency domain loss, and obtain the second frequency domain loss L. FD Next, the second output audio and the downsampled real audio acquired by TargetWavDownsampling are subjected to time domain loss calculation to obtain the second time domain loss.

[0115] Specifically, the second output audio and the real audio acquired by Target Wav Downsampling are used to calculate the time domain loss. The implementation process of obtaining the second time domain loss is as follows: Substitute the second output audio and the real audio acquired by Target Wav Downsampling into the above formula (2) to calculate the energy loss and obtain the energy loss; substitute the second output audio and the real audio acquired by Target Wav Downsampling into the above formula (3) to calculate the time loss and obtain the time loss; substitute the second output audio and the real audio acquired by Target Wav Downsampling into the above formula (4) to calculate the phase loss and obtain the phase loss; substitute the energy loss obtained by formula (2), the time loss obtained by formula (3), and the phase loss obtained by formula (4) into the above formula (5) for calculation, and use the sum of the energy loss, time loss, and phase loss as the second time domain loss.

[0116] Step S202: Process the third output audio corresponding to the second feature vector to determine the third frequency domain loss.

[0117] The specific implementation process of step S202 includes the following steps:

[0118] Step S21: Process the second feature vector using the third convolutional + activation function Tanh layer of the generator to determine the third output audio corresponding to the second feature vector.

[0119] In the specific implementation step S21, the convolution Conv in the third convolution Conv+ activation function Tanh layer reduces the feature dimension of the second feature vector to 1, and changes the numerical range of the feature to -1 to 1 through the Tanh layer, so as to output the third output audio with the numerical range of the feature dimension in (-1,1).

[0120] It should be noted that the third output audio is high-sampling-rate audio.

[0121] Step S22: Perform a short-time Fourier transform on the third output audio to determine the third frequency domain loss.

[0122] In the specific implementation step S22, a short-time Fourier transform is performed on the third output audio and the real audio collected by Target Wav Downsampling, and the frequency domain loss between the third output audio and the real audio is calculated, that is, the third frequency domain loss.

[0123] Step S203: Calculate the total loss of the generator based on the first time-domain loss, the first frequency-domain loss, the second time-domain loss, the second frequency-domain loss, and the third frequency-domain loss, so as to train the generator network using the total loss.

[0124] In the specific implementation step S203, the first time domain loss, the first frequency domain loss, the second time domain loss, the second frequency domain loss and the third frequency domain loss are substituted into formula (6) to obtain the total loss of the generator. Then, the network parameters of the generator are adjusted using the total loss, that is, the generator network is retrained based on the total loss.

[0125] It should be noted that the network parameters of the generator refer to the parameters of the convolutional layer, upsampling layer, residual module, and other convolutional layers in the convolutional + activation function Tanh layer.

[0126] Among them, the convolution + activation function Tanh layer refers to the first convolution + activation function Tanh layer, the second convolution + activation function Tanh layer, and the third convolution + activation function Tanh layer.

[0127] It should be noted that if the number of the first time-domain loss, the first frequency-domain loss, the second time-domain loss, and the second frequency-domain loss is one, the first time-domain loss, the first frequency-domain loss, the second time-domain loss, the second frequency-domain loss, and the third frequency-domain loss are directly substituted into formula (6) to obtain the total loss of the generator; if the number of the first time-domain loss, the first frequency-domain loss, the second time-domain loss, and the second frequency-domain loss is N, the sum of the N first time-domain losses, the sum of the N first frequency-domain losses, the sum of the N second time-domain losses, and the sum of the N second frequency-domain losses are first calculated; and the sum of the N first time-domain losses, the sum of the N first frequency-domain losses, the sum of the N second time-domain losses, the sum of the N second frequency-domain losses, and the third frequency-domain loss are substituted into formula (6) to obtain the total loss of the generator.

[0128] Formula (6):

[0129]

[0130] Among them, L G The total loss is given by λ1, λ2, λ3, and λ4, which are preset values. c represents the audio feature, x represents the actual audio, and D... k This represents the output of the last layer of the k-th discriminator. This represents the output of the i-th layer of the k-th discriminator. L stft The third frequency domain loss, L, is the frequency domain loss between the output audio generated by the generator and the real audio. fm The feature map loss measures the loss of each layer of feature maps after the generated audio and the real audio have passed through the discriminator. (D) k (G(c)-1) 2 It is an adversarial loss, used to measure how close the generated audio is to 1 after passing through a discriminator.

[0131] Optionally, it also includes: determining whether the third output audio RawWaveform meets expectations; if so, training the generator network based on the total loss calculated at this time, and directly executing step S204.

[0132] It should be noted that the expectations here are set based on multiple test settings or actual situations.

[0133] Step S204: Determine whether the third frequency domain loss meets the preset requirements. If the third frequency domain loss meets the preset requirements, execute step S205. If the third frequency domain loss does not meet the preset requirements, return to execute step S201 again.

[0134] Optionally, it also includes: recording the third frequency domain loss.

[0135] The preset requirement refers to whether the curve drawn by the third frequency domain loss and the recorded historical third frequency domain loss meets the preset expectation.

[0136] It should be noted that the specific implementation process of step S204 includes the following steps:

[0137] Step S31: Determine whether the curve drawn by the third frequency domain loss and the recorded historical third frequency domain loss meets the preset expectation. If yes, execute step S205; otherwise, return to execute step S201 again.

[0138] In the specific implementation of step S31, the curves corresponding to the preset number of third frequency domain losses are plotted. If the curves become flat, the total loss corresponding to the last recorded third frequency domain loss is determined. The network parameters adjusted by the total loss are used as the final network parameters, and the generator corresponding to the network parameters is used as the trained generator.

[0139] Step S205: Determine the network of the generator obtained by training with the total loss as the generator that has been trained.

[0140] In the specific implementation of step S205, when the curve plotted by the total loss of a preset number of records reaches the preset expectation, the network of the current generator is determined to be the generator that has been trained.

[0141] Step S103: Combine the trained generator and the discriminator trained with the generator to form a vocoder.

[0142] In the specific implementation of step S103, during the generator training process, the discriminator is trained based on the third output audio of the generator and the real audio, and the trained generator and the discriminator trained with the generator are combined to form a vocoder.

[0143] In this embodiment of the invention, during generator training, multi-resolution temporal and frequency domain losses are calculated, i.e., a multi-level resolution loss function is introduced into the generator to calculate the total loss of the generator. Although a convolutional + tanh layer is added during vocoder training to calculate the loss, the time consumed in synthesizing audio at the same sampling rate during vocoder inference remains unchanged. This method accelerates the convergence speed of vocoder training, thereby improving the language processing quality of the vocoder. Furthermore, the vocoder can generate audio at multiple sampling rates without retraining, reducing inference time.

[0144] If the number of upsampling layers and residual stacks is N, the specific training process of the generator is as follows: Figure 4 As shown.

[0145] Step S401: The i-th upsampling layer processes the second feature vector output by the (i-1)-th residual module to obtain the first feature vector.

[0146] In step S401, if i equals 1, the i-th upsampling layer processes the sample set to obtain the first feature vector, N is a positive integer greater than or equal to 1, i is less than or equal to N, and N is L / 2.

[0147] In the specific implementation of step S401, when the i-th element is 1, the i-th upsampling layer collects the low-sample-rate audio features obtained by the previous convolutional layer Conv layer, i.e., the first feature vector, and sends it to the first convolutional + activation function Tanh layer a1, thereby executing steps S404 to S406, and sending it to the same layer, i.e., the i-th residual module Residual stack; when the i-th element is a positive integer greater than 1, the i-th upsampling layer processes the second feature vector output by the (i-1)-th residual module Residual stack to obtain the first feature vector, and sends it to the first convolutional + activation function Tanh layer a1 and the i-th residual module Residual stack.

[0148] It should be noted that the number of the first feature vectors is N.

[0149] Step S402: The i-th residual module processes the first feature vector output by the i-th upsampling layer to obtain the second feature vector.

[0150] In the specific implementation step S402, the i-th residual module Residual stack encodes the first feature vector obtained by the i-th upsampling layer in the same layer, and sends the encoded second feature vector to the second convolution + activation function Tanh layer a2.

[0151] Step S403: Determine whether the i-th feature vector is equal to N. If it is not equal, send the second feature vector to the (i+1)-th upsampling layer and return to execute step S401. If it is equal, execute steps S407 to S408.

[0152] In the specific implementation of step S403, it is determined whether the value of i+1 is greater than the value of N. If it is less than or equal to N, the second feature vector is sent to the (i+1)th upsampling layer, and steps S404 to S406 are executed. If it is greater than N, that is, i is equal to N, the second feature vector is sent to the third convolutional + activation function Tanh layer b, and steps S407 to S408 are executed.

[0153] Step S404: Process the first feature vector using the first convolutional + activation function Tanh layer of the generator to determine the first output audio corresponding to the first feature vector.

[0154] Step S405: Process the second feature vector using the second convolutional + activation function Tanh layer of the generator to determine the second output audio corresponding to the second feature vector.

[0155] Step S406: Perform frequency domain processing and time domain processing on the first output audio respectively to determine the first time domain loss and the first frequency domain loss.

[0156] Step S407: Perform frequency domain processing and time domain processing on the second output audio respectively to determine the second time domain loss and the second frequency domain loss.

[0157] It should be noted that the specific implementation processes of steps S404 to S407 are the same as those of steps S13 to S15 described above, and can be referred to each other.

[0158] Step S408: The second feature vector obtained by the residual module with the i-th value equal to N is processed by the third convolution + activation function Tanh layer of the generator to determine the third output audio corresponding to the second feature vector.

[0159] Step S409: Perform a short-time Fourier transform on the third output audio to determine the third frequency domain loss.

[0160] The specific implementation process of steps S408 to S409 is the same as that of steps S21 to S22 above, and they can be referred to each other.

[0161] Step S410: Calculate the total loss of the generator based on the first time-domain loss, the first frequency-domain loss, the second time-domain loss, the second frequency-domain loss, and the third frequency-domain loss, so as to train the generator network using the total loss.

[0162] Step S411: Determine whether the third frequency domain loss meets the preset requirements. If the third frequency domain loss meets the preset requirements, execute step S412. If the third frequency domain loss does not meet the preset requirements, return to execute step S401 again.

[0163] Step S412: Determine the network of the generator obtained by training with the total loss as the generator that has been trained.

[0164] It should be noted that the specific implementation process of steps S410 to S412 is the same as that of steps S201 to S205 above, and they can be referred to each other.

[0165] In this embodiment of the invention, when training the generator, if the number of upsampling layers and residual modules is N, multiple time-domain losses and frequency-domain losses are calculated, i.e., multi-level resolution loss functions are introduced into the generator, and then the total loss of the generator is calculated. Although a convolutional + tanh layer is added during vocoder training to calculate the loss, the time consumed in synthesizing audio with the same sampling rate during vocoder inference remains unchanged. This method can accelerate the convergence speed of vocoder training, thereby improving the language processing quality of the vocoder. Furthermore, the vocoder can generate audio with multiple sampling rates without retraining the vocoder, reducing inference time.

[0166] Based on the vocoder disclosed in the above embodiments of the present invention, the present invention also discloses a corresponding speech processing method, applicable to the vocoder shown in the above embodiments, such as... Figure 5 As shown, the method includes the following steps:

[0167] Step S501: Input the audio features into the vocoder.

[0168] In step S501, the audio features are obtained by converting the acquired raw audio using an acoustic vocoder.

[0169] In the specific implementation step S501, the original audio output by the user is acquired; the acquired original audio is converted using an acoustic vocoder to obtain audio features, and the audio features are then input into the vocoder.

[0170] Step S502: Perform speech processing on the audio features based on the vocoder and output the target audio.

[0171] In the specific implementation step S502, the audio features are processed by the vocoder constructed above to output the target audio.

[0172] In this embodiment of the invention, audio features are input into the vocoder, and speech processing is performed on the audio features based on the vocoder to output the target audio. The vocoder trained as described above can generate audio at various sampling rates without the need for retraining, thus reducing inference time.

[0173] Based on the vocoder construction method shown in the above embodiments of the present invention, the present invention also discloses a structural schematic diagram of a vocoder construction device, as follows: Figure 6 As shown, the device includes:

[0174] The acquisition unit 601 is used to acquire a sample set, which refers to the audio features obtained by initializing the audio data collected within a preset time period.

[0175] Training unit 602 is used to train the generator using a sample set and determine the trained generator.

[0176] The generation unit 603 is used to combine the trained generator and the discriminator trained with the generator into a vocoder.

[0177] The training unit 602 is specifically configured to: process the first output audio corresponding to the first feature vector and the second output audio corresponding to the second feature vector, respectively, to determine a first time-domain loss, a first frequency-domain loss, a second time-domain loss, and a second frequency-domain loss, wherein the first feature vector is obtained by the generator's upsampling layer processing the sample set, and the second feature vector is obtained by the generator's residual module processing the first feature vector, and the number of the upsampling layer and the residual module are equal; process the third output audio corresponding to the second feature vector to determine a third frequency-domain loss; calculate the generator's total loss based on the first time-domain loss, the first frequency-domain loss, the second time-domain loss, the second frequency-domain loss, and the frequency-domain loss, and use the total loss to train the generator's network; and when the third frequency-domain loss meets a preset requirement, determine the generator network trained by the total loss as the trained generator.

[0178] The specific principles and execution processes of each unit in the vocoder construction apparatus disclosed in the above embodiments of the present invention are the same as those of the vocoder construction method shown in the above embodiments of the present invention. Please refer to the corresponding parts of the vocoder construction method disclosed in the above embodiments of the present invention, and they will not be repeated here.

[0179] In this embodiment of the invention, during generator training, multi-resolution temporal and frequency domain losses are calculated, i.e., a multi-level resolution loss function is introduced into the generator to calculate the total loss of the generator. Although a convolutional + tanh layer is added during vocoder training to calculate the loss, the time consumed in synthesizing audio at the same sampling rate during vocoder inference remains unchanged. This method accelerates the convergence speed of vocoder training, thereby improving the language processing quality of the vocoder. Furthermore, the vocoder can generate audio at multiple sampling rates without retraining, reducing inference time.

[0180] Optionally, based on the vocoder construction method shown in the above embodiments of the present invention, the training unit 602 that processes the first output audio corresponding to the first feature vector and the second output audio corresponding to the second feature vector to determine the first time-domain loss, the first frequency-domain loss, the second time-domain loss, and the second frequency-domain loss is specifically used for:

[0181] An upsampling layer processes the sample set to obtain a first feature vector; a residual module processes the first feature vector to obtain a second feature vector; the generator's first convolutional + activation function Tanh layer processes the first feature vector to determine the first output audio corresponding to the first feature vector; the generator's second convolutional + activation function Tanh layer processes the second feature vector to determine the second output audio corresponding to the second feature vector; the first output audio is processed in both the frequency domain and the time domain to determine a first time domain loss and a first frequency domain loss; the second output audio is processed in both the frequency domain and the time domain to determine a second time domain loss and a second frequency domain loss.

[0182] Optionally, based on the vocoder construction method shown in the above embodiments of the present invention, the training unit 602 is further configured to:

[0183] If the number of upsampling layers and residual modules is N, the i-th upsampling layer processes the second feature vector output by the (i-1)-th residual module to obtain the first feature vector, where i equals 1, the i-th upsampling layer processes the sample set to obtain the first feature vector, N is a positive integer greater than or equal to 1, and i is less than or equal to N; the i-th residual module processes the first feature vector output by the i-th upsampling layer to obtain the second feature vector; the first feature vector is processed using the generator's first convolution + activation function Tanh layer to determine the first output audio corresponding to the first feature vector; the second feature vector is processed using the generator's second convolution + activation function Tanh layer to determine the second output audio corresponding to the second feature vector; the first output audio is processed in both the frequency domain and time domain to determine the first time domain loss and the first frequency domain loss; the second output audio is processed in both the frequency domain and time domain to determine the second time domain loss and the second frequency domain loss.

[0184] Optionally, based on the vocoder construction method shown in the above embodiments of the present invention, the third output audio corresponding to the second feature vector is processed to determine the training unit 602 for the third frequency domain loss, specifically used for:

[0185] The second feature vector is processed using the third convolutional layer with activation function Tanh of the generator to determine the third output audio corresponding to the second feature vector; a short-time Fourier transform is performed on the third output audio to determine the third frequency domain loss.

[0186] Optionally, based on the vocoder construction method shown in the above embodiments of the present invention, the training unit 602 is further configured to:

[0187] If the number of upsampling layers and residual modules is N, if i is not equal to 1, the i-th upsampling layer processes the feature vector output by the (i-1)-th residual module to obtain the first feature vector. If i is equal to 1, the i-th upsampling layer processes the sample set to obtain the first feature vector. N is a positive integer greater than or equal to 1, and i is less than or equal to N.

[0188] The i-th residual module processes the first feature vector output by the i-th upsampling layer to obtain the second feature vector;

[0189] The second feature vector obtained by the residual module with the i-th value equal to N is processed by the third convolution + activation function Tanh layer of the generator to determine the third output audio corresponding to the second feature vector;

[0190] Perform a short-time Fourier transform on the third output audio to determine the third frequency domain loss.

[0191] Optionally, the training unit 602, which determines the completed training of the generator based on the third frequency domain loss, is specifically used for:

[0192] When the curve drawn by the third frequency domain loss and the recorded historical third frequency domain loss reaches the preset expectation, the network of the generator obtained by training with the total loss is determined to be the trained generator, and the number of historical third frequency domain losses needs to reach a preset value.

[0193] When it is determined that the curve drawn by the third frequency domain loss and the recorded historical third frequency domain loss does not meet the preset expectation, the process returns to process the first output audio corresponding to the first feature vector and the second output audio corresponding to the second feature vector respectively, and determines the first time domain loss, the first frequency domain loss, the second time domain loss and the second frequency domain loss.

[0194] Based on the speech processing method shown in the above embodiments of the present invention, the present invention also discloses a structural schematic diagram of a speech processing device, as follows: Figure 7 As shown, the device includes:

[0195] The input module 701 is used to input audio features into the vocoder, wherein the audio features are obtained by the acoustic vocoder converting the acquired raw audio.

[0196] The vocoder 702 is used to perform speech processing on the audio features and output the target audio.

[0197] The specific principles and execution processes of each unit in the speech processing device disclosed in the above embodiments of the present invention are the same as those of the speech processing method shown in the above embodiments of the present invention. Please refer to the corresponding parts of the speech processing method disclosed in the above embodiments of the present invention, and they will not be repeated here.

[0198] In this embodiment of the invention, audio features are input into the vocoder, and speech processing is performed on the audio features based on the vocoder to output the target audio. The vocoder trained as described above can generate audio at various sampling rates without the need for retraining, thus reducing inference time.

[0199] This invention also discloses an electronic device for running database stored procedures, wherein the execution of the database stored procedures is as described above. Figure 1 , Figure 2 and Figure 4 The publicly available method for constructing a vocoder, or, as described above. Figure 5 The speech processing method described in [the document].

[0200] This invention also discloses a computer storage medium, the storage medium including a stored database procedure, wherein, during the execution of the stored database procedure, the device where the storage medium is located is controlled to execute the above-described procedure. Figure 1 , Figure 2 and Figure 4 The publicly available method for constructing a vocoder, or, as described above. Figure 5 The speech processing method described in [the document].

[0201] In the context of this disclosure, a computer storage medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0202] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0203] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0204] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for constructing a vocoder, characterized in that, The method includes: Obtain a sample set, which refers to the audio features obtained by initializing audio data collected within a preset time period; The generator is trained using a sample set to determine the trained generator. The trained generator and the discriminator trained with the generator are combined to form a vocoder; The specific process of training the generator is as follows: If the number of upsampling layers and residual modules is one, the first output audio corresponding to the first feature vector and the second output audio corresponding to the second feature vector are processed respectively to determine the first time-domain loss, the first frequency-domain loss, the second time-domain loss, and the second frequency-domain loss. The first feature vector is obtained by the generator's upsampling layer processing the sample set, and the second feature vector is obtained by the generator's residual module processing the first feature vector. The number of upsampling layers and residual modules is equal. The first output audio corresponding to the first feature vector is obtained by processing the first feature vector using the generator's first convolution + activation function Tanh layer. The second output audio corresponding to the second feature vector is obtained by processing the second feature vector using the generator's second convolution + activation function Tanh layer. The first time-domain loss and the first frequency-domain loss are obtained by processing the first output audio in the frequency domain and time domain respectively, and the second time-domain loss and the second frequency-domain loss are obtained by processing the second output audio in the frequency domain and time domain respectively. The third output audio corresponding to the second feature vector is processed to determine the third frequency domain loss; the third output audio corresponding to the second feature vector is obtained by processing the second feature vector using the third convolution + activation function Tanh layer of the generator; the third frequency domain loss is obtained by performing a short-time Fourier transform on the third output audio. The generator's total loss is calculated based on the first time-domain loss, the first frequency-domain loss, the second time-domain loss, the second frequency-domain loss, and the third frequency-domain loss, and the generator's network is trained using the total loss. When the third frequency domain loss is determined to meet the preset requirements, the network of the generator obtained by training with the total loss is determined to be the generator that has been trained; the preset requirements are whether the curve drawn by the third frequency domain loss and the recorded historical third frequency domain loss reaches the preset expectation.

2. The method according to claim 1, characterized in that, Also includes: If the number of upsampling layers and residual modules is N, the i-th upsampling layer processes the second feature vector output by the (i-1)-th residual module to obtain the first feature vector. Where i equals 1, the i-th upsampling layer processes the sample set to obtain the first feature vector, N is a positive integer greater than or equal to 1, and i is less than or equal to N. The i-th residual module processes the first feature vector output by the i-th upsampling layer to obtain the second feature vector; The first feature vector is processed using the first convolutional layer plus activation function Tanh layer of the generator to determine the first output audio corresponding to the first feature vector; The second feature vector is processed using the second convolutional + activation function Tanh layer of the generator to determine the second output audio corresponding to the second feature vector; The first output audio is processed in both the frequency domain and the time domain to determine the first time domain loss and the first frequency domain loss. The second output audio is processed in both the frequency domain and the time domain to determine the second time domain loss and the second frequency domain loss.

3. The method according to claim 1, characterized in that, Also includes: If the number of upsampling layers and residual modules is N, if i is not equal to 1, the i-th upsampling layer processes the feature vector output by the (i-1)-th residual module to obtain the first feature vector. If i is equal to 1, the i-th upsampling layer processes the sample set to obtain the first feature vector. N is a positive integer greater than or equal to 1, and i is less than or equal to N. The i-th residual module processes the first feature vector output by the i-th upsampling layer to obtain the second feature vector; The second feature vector obtained by the residual module with the i-th value equal to N is processed by the third convolution + activation function Tanh layer of the generator to determine the third output audio corresponding to the second feature vector; Perform a short-time Fourier transform on the third output audio to determine the third frequency domain loss.

4. The method according to claim 1, characterized in that, Determining that the third frequency domain loss meets preset requirements includes: When the curve drawn by the third frequency domain loss and the recorded historical third frequency domain loss reaches the preset expectation, the network of the generator obtained by training with the total loss is determined to be the trained generator, and the number of historical third frequency domain losses needs to reach a preset value. If the curve plotted by the third frequency domain loss and the recorded historical third frequency domain loss does not meet the preset expectation, the process returns to the step of processing the first output audio corresponding to the first feature vector and the second output audio corresponding to the second feature vector to determine the first time domain loss, the first frequency domain loss, the second time domain loss and the second frequency domain loss.

5. A speech processing method, characterized in that, A vocoder applicable to a training method for a vocoder according to claim 1, the method comprising: The audio features are input into the vocoder, which are obtained by converting the acquired raw audio by the acoustic vocoder. Based on the vocoder, the audio features are processed to output the target audio.

6. A device for constructing a vocoder, characterized in that, The device includes: The acquisition unit is used to acquire a sample set, which is an audio feature obtained by initializing audio data collected within a preset time period. A training unit is used to train the generator using a sample set and determine the trained generator. A generation unit is used to combine the trained generator and the discriminator trained with the generator into a vocoder. The training unit is specifically configured to: when the number of upsampling layers and residual modules is one, process the first output audio corresponding to the first feature vector and the second output audio corresponding to the second feature vector, respectively, to determine the first time-domain loss, the first frequency-domain loss, the second time-domain loss, and the second frequency-domain loss. The first feature vector is obtained by the generator's upsampling layer processing the sample set, and the second feature vector is obtained by the generator's residual module processing the first feature vector. The number of upsampling layers and residual modules is equal. The first output audio corresponding to the first feature vector is obtained by processing the first feature vector using the generator's first convolution + activation function Tanh layer. The second output audio corresponding to the second feature vector is obtained by processing the second feature vector using the generator's second convolution + activation function Tanh layer. The first time-domain loss and the first frequency-domain loss are obtained by processing the first output audio in the frequency domain and time domain, respectively. The second time-domain loss and the second frequency-domain loss are obtained by processing the second output audio in the frequency domain and time domain, respectively. The third output audio corresponding to the second feature vector is processed to determine the third frequency domain loss. The third output audio corresponding to the second feature vector is obtained by processing the second feature vector using the third convolutional + activation function Tanh layer of the generator. The third frequency domain loss is obtained by performing a short-time Fourier transform on the third output audio. The total loss of the generator is calculated based on the first time domain loss, the first frequency domain loss, the second time domain loss, the second frequency domain loss, and the third frequency domain loss, and the generator network is trained using the total loss. When the third frequency domain loss meets a preset requirement, the generator network trained with the total loss is determined to be the trained generator. The preset requirement is whether the curve drawn by the third frequency domain loss and the recorded historical third frequency domain loss meets a preset expectation.

7. A voice processing device, characterized in that, The apparatus includes a vocoder constructed according to the training method of the vocoder of claim 1, the apparatus comprising: An input module is used to input audio features into the vocoder, wherein the audio features are obtained by the acoustic vocoder converting the acquired raw audio. The vocoder is used to perform speech processing on the audio features and output the target audio.

8. An electronic device, characterized in that, The electronic device is used to run a program, wherein the program executes the method for constructing a vocoder as described in any one of claims 1-4, or executes the speech processing method as described in claim 5.

9. A computer storage medium, characterized in that, The storage medium includes a stored program, wherein, when the program is executed, it controls the device where the storage medium is located to perform the method for constructing a vocoder as described in any one of claims 1-4, or to perform the speech processing method as described in claim 5.

Images