Audio generator and method for generating an audio signal and training an audio generator

Abstract

Techniques for generating audio signals and training an audio generator are disclosed. An audio generator (10) can generate an audio signal (16) from target data (12) representing the audio signal (16) and an input signal (14), comprising: a first processing block (40, 50, 50a-50h) receiving first data (15, 59a) derived from the input signal (14) and outputting first output data (69); a second processing block (45) receiving the first output data (69) or data derived from the first output data (69) as second data. The first processing block (50) comprises: a conditional set of learnable layers (71, 72, 73) configured to process the target data (12) to obtain conditional feature parameters (74, 75); and a style element (77) configured to apply the conditional feature parameters (74, 75) to the first data (15, 59a) or to normalized first data (59, 76').

Description

[0001] introduction

[0002] Different embodiments and aspects of the invention will be described below. Further embodiments will be defined by the appended claims. It should be noted that any embodiment defined by the claims may be supplemented by any details (features and functions) described in this specification.

[0003] Furthermore, the embodiments described in this specification may be used alone or supplemented by any feature herein or any feature included in the claims.

[0004] Furthermore, it should be noted that the various aspects described herein can be used individually or in combination. Therefore, details can be added to each individual aspect without adding details to the other aspect.

[0005] It should also be noted that this disclosure expressly or implicitly describes features available in audio generators and / or methods and / or computer program products. Therefore, any feature described herein may be used in the context of a device, method, and / or computer program product.

[0006] Furthermore, the features and functions disclosed herein related to the methods can also be used with the device (configured to perform such functions). Additionally, any features and functions disclosed herein regarding the device can also be used in the corresponding methods. In other words, the methods disclosed herein can be supplemented by any features and functions described with respect to the device.

[0007] Furthermore, any features and functions described herein may be implemented in hardware or software, or using a combination of hardware and software, as described in the "Implementation Alternatives" section.

[0008] Implementing alternative solutions

[0009] Although certain aspects are described in the context of the device, it is clear that these aspects also represent a description of the corresponding method, where features correspond to method steps or features of method steps. Similarly, aspects described in the context of method steps also represent a description of corresponding features of the corresponding device. Some or all of the method steps may be performed by (or using) hardware means, such as a microprocessor, a programmable computer, or electronic circuitry. In some embodiments, one or more of the most important method steps may be performed by such means.

[0010] Depending on certain implementation requirements, embodiments of the present invention can be implemented in hardware or software. This implementation can be performed using digital storage media such as floppy disks, DVDs, Blu-ray discs, CDs, ROMs, PROMs, EPROMs, EEPROMs, or flash memory, which have electronically readable control signals stored thereon that cooperate (or are capable of cooperating with) a programmable computer system to perform the corresponding methods. Therefore, the digital storage media can be computer-readable.

[0011] Some embodiments of the invention include a data carrier having electronically readable control signals that are capable of cooperating with a programmable computer system to perform one of the methods described herein.

[0012] Typically, embodiments of the present invention can be implemented as a computer program product having program code that, when run on a computer, is operable to perform one of the methods. The program code may, for example, be stored on a machine-readable medium.

[0013] Other embodiments include a computer program stored on a machine-readable medium for performing one of the methods described herein.

[0014] In other words, therefore, an embodiment of the method of the present invention is a computer program having program code for executing one of the methods described herein when a computer program is run on a computer.

[0015] Therefore, a further embodiment of the method of the present invention is a data carrier (or digital storage medium, or computer-readable medium) including a computer program recorded thereon for performing one of the methods described herein. The data carrier, digital storage medium, or recording medium is generally tangible and / or non-transitory.

[0016] Therefore, a further embodiment of the method of the present invention represents a data stream or signal sequence for performing one of the methods described herein. The data stream or signal sequence may, for example, be configured to be transmitted via a data communication connection, such as via the Internet.

[0017] Further embodiments include processing means, such as a computer or programmable logic device, configured or adapted to perform one of the methods described herein.

[0018] A further embodiment includes a computer on which a computer program for performing one of the methods described herein is installed.

[0019] Further embodiments of the invention include an apparatus or system configured to transmit (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. For example, the receiver may be a computer, mobile device, storage device, etc. The apparatus or system may, for example, include a file server for transmitting the computer program to the receiver.

[0020] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, the field-programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware device.

[0021] The device described in this article can be implemented using hardware, a computer, or a combination of hardware and a computer.

[0022] The device described herein, or any component thereof, may be implemented, at least in part, in hardware and / or software.

[0023] The methods described herein can be performed using hardware devices, computers, or a combination of hardware devices and computers.

[0024] The methods described herein, or any part thereof, may be implemented, at least in part, by hardware and / or software.

[0025] The embodiments described above are merely illustrative of the principles of the invention. It should be understood that modifications and variations in the arrangements and details described herein will be apparent to those skilled in the art. Therefore, its intent is limited only by the scope of the forthcoming patent claims, and not by the specific details provided by the description and interpretation of the embodiments herein. Technical Field

[0026] This invention belongs to the technical field of audio generation.

[0027] Embodiments of the present invention relate to an audio generator configured to generate an audio signal from an input signal and target data, the target data representing the audio signal. Further embodiments relate to methods for generating audio signals and methods for training an audio generator. Further embodiments relate to computer program products. Background Technology

[0028] In recent years, neural vocoders have surpassed classical speech synthesis methods in terms of the naturalness and perceptual quality of synthesized speech signals. The best results can be achieved using computationally intensive neural vocoders such as WaveNet and WaveGlow, while lightweight architectures based on generative adversarial networks, such as MelGAN and Parallel WaveGAN, still lag behind in perceptual quality.

[0029] Generative models that use deep learning to generate audio waveforms, such as WaveNet, LPCNet, and WaveGlow, have made significant progress in natural speech synthesis. These generative models, known as neural vocoders in text-to-speech (TTS) applications, outperform parametric and concatenation synthesis methods. They can be tuned using a compressed representation of the target speech (e.g., a Mel spectrogram) to reproduce a given speaker and given utterance.

[0030] Previous research has shown that using this generative model on the decoder side can achieve clean speech coding at very low bit rates. This can be achieved by tuning the neural vocoder using parameters from a classic low-bit-rate speech encoder.

[0031] Neural vocoders are also used for speech enhancement tasks, such as speech denoising or noise reduction.

[0032] The main problems with these deep generative models are typically the large number of parameters required and the complexity that arises during training and synthesis (inference). For example, WaveNet, considered the state-of-the-art in synthesized speech quality, generates audio samples sequentially, one after another. This process is very slow, computationally demanding, and cannot be performed in real time.

[0033] In recent years, lightweight adversarial vocoders based on generative adversarial networks (GANs), such as MelGAN and ParallelWaveGAN, have been proposed for fast waveform generation. However, the perceptual quality of speech generated using these models is significantly lower than the baseline of neural vocoders such as WaveNet and WaveGlow. A text-to-speech GAN (GAN-TTS) has been proposed to bridge this quality gap, but it still requires high computational cost.

[0034] There are many types of neural vocoders, but each has its own drawbacks. Autoregressive vocoders, such as WaveNet and LPCNet, can have very high quality and are well-suited for optimization on CPUs for inference, but they are not suitable for use on GPUs because their processing cannot be easily parallelized, and they cannot provide non-real-time processing without compromising quality.

[0035] Normalized stream vocoders, such as WaveGlow, can also have very high quality and are suitable for inference on GPUs, but they involve very complex models that take a long time to train and optimize, and are not suitable for embedded devices.

[0036] GAN vocoders, such as MelGAN and ParallelWaveGAN, may be suitable for inference on GPUs and are lightweight, but their quality is lower than that of autoregressive models.

[0037] In summary, there is currently no low-complexity solution that provides high-fidelity speech. GANs are the most researched method for achieving this goal. This invention offers an effective solution to this problem.

[0038] The purpose of this invention is to provide a lightweight neural vocoder solution that generates speech with very high quality and can be trained with limited computing resources. Attached Figure Description

[0039] Embodiments of the present invention will then be described with reference to the accompanying drawings, wherein:

[0040] Figure 1 An audio generator architecture according to an embodiment of the present invention is shown.

[0041] Figure 2 A discriminator structure that can be used to train an audio generator according to the present invention is shown.

[0042] Figure 3 The structure of a portion of an audio generator according to an embodiment of the present invention is shown.

[0043] Figure 4 The structure of a portion of an audio generator according to an embodiment of the present invention is shown, and

[0044] Figure 5 The results of the MUSHRA expert hearing test for different models are shown.

[0045] Figure 6 The architecture of an audio generator according to an embodiment of the present invention is shown.

[0046] Figure 7 The operation performed on the signal according to the present invention is shown.

[0047] In the figures, similar reference numerals indicate similar elements and features. Summary of the Invention

[0048] Specifically, an audio generator (e.g., 10) is proposed, configured to generate an audio signal (e.g., 16) from an input signal (e.g., 14) and target data (e.g., 12), the target data (e.g., 12) representing the audio signal (e.g., 16), the audio generator comprising at least one of the following:

[0049] A first processing block (e.g., 40, 50, 50a-50h) is configured to receive first data (e.g., 15, 59a) derived from an input signal (e.g., 14) and output first output data (e.g., 69), wherein the first output data (e.g., 69) comprises multiple channels (e.g., 47), and

[0050] The second processing block (e.g., 45) is configured to receive the first output data (e.g., 69) or data obtained from the first output data (e.g., 69) as second data.

[0051] The first processing block (e.g., 50) may include for each channel of the first output data:

[0052] The conditional sets of learnable layers (e.g., 71, 72, 73) are configured to process target data (e.g., 12) to obtain conditional feature parameters (e.g., 74, 75); and

[0053] Style elements (e.g., 77) are configured to apply conditional feature parameters (e.g., 74, 75) to first data (e.g., 15, 59a) or normalized first data (e.g., 59, 76').

[0054] The second processing block (e.g., 45) may be configured to combine multiple channels (e.g., 47) of the second data (e.g., 69) to obtain an audio signal (e.g., 16).

[0055] A method is also proposed, for example, for generating an audio signal (e.g., 16) by an audio generator (e.g., 10) from an input signal (e.g., 14) and target data (e.g., 12), the target data (e.g., 12) representing the audio signal (e.g., 16), the method comprising:

[0056] The first data (e.g., 16559, 59a, 59b) obtained from the input signal (e.g., 14) is received through the first processing block (e.g., 50, 50a-50h);

[0057] For each channel of the first output data (e.g., 59b, 69):

[0058] The target data (e.g., 12) is processed through the conditional set of the learnable layers (e.g., 71, 72, 73) of the first processing block (e.g., 50) to obtain conditional feature parameters (e.g., 74, 75); and

[0059] Conditional feature parameters (e.g., 74, 75) are applied to first data (e.g., 15, 59) or normalized first data (e.g., 76') through style elements (e.g., 77) of the first processing block (e.g., 50).

[0060] The first processing block (e.g., 50) outputs first output data (e.g., 69) including multiple channels (e.g., 47);

[0061] The first output data (e.g., 69) or data derived from the first output data (e.g., 69) is received as second data through a second processing block (e.g., 45); and

[0062] The second processing block (e.g., 45) combines multiple channels of the second data (e.g., 47) to obtain an audio signal (e.g., 16).

[0063] A method for training a neural network for audio generation is also proposed, wherein the neural network:

[0064] From an input sequence (e.g., 14) representing the audio data to be generated (e.g., 16), output audio samples at a given time step;

[0065] Configured to shape the noise vector (e.g., 14) to create output audio samples (e.g., 16) using the input representative sequence (e.g., 12), and

[0066] The training was designed to optimize the loss function (e.g., 140).

[0067] A method for generating audio signals (e.g., 16) is also proposed, including a mathematical model configured to output audio samples at a given time step from an input sequence (e.g., 12) representing the audio data to be generated (e.g., 16). The mathematical model can shape a noise vector (e.g., 14) to create output audio samples using the input representative sequence (e.g., 12).

[0068] It is against this backdrop that we propose StyleMelGAN (e.g., Audio Generator 10), a lightweight neural vocoder that allows for the synthesis of high-fidelity speech with low computational complexity. StyleMelGAN is a fully convolutional feedforward model that uses Time Adaptive Denormalization (TADE) (e.g. Figure 4 60a and 60b in the middle, and Figure 3(60) The low-dimensional noise vector (e.g., a 128x1 vector) is styled (e.g., at 77) via acoustic features of the target speech waveform. This architecture allows for highly parallel generation, several times faster than real-time on both the control processing unit (CPU) and the graphics processing unit (GPU). For efficient and fast training, we can use a multi-scale spectral reconstruction loss and an adversarial loss computed by multiple discriminators (e.g., 132a-132d), and use randomized windowing (e.g., windows 105a, 105b, 105c, 105d) where multiple discriminators evaluate the speech signal in multiple frequency bands.16 MUSHRA and P.800 hearing tests show that StyleMelGAN (e.g., audio generator 10) outperforms known existing neural vocoders in both copy synthesis and TTS scenarios.

[0069] Specifically, this application proposes a neural vocoder for generating high-quality speech16, which can be based on a generative adversarial network (GAN). This solution, referred to herein as StyleMelGAN (and, for example, implemented in audio generator10), is a lightweight neural vocoder that allows the synthesis of high-quality speech16 with low computational complexity. StyleMelGAN is a feedforward fully convolutional model that uses temporally adaptive denormalization (TADE) to style (e.g., in block 77) the latent noise representation (e.g., 69) using, for example, the Mel spectrogram (12) of the target speech waveform. It allows for highly parallel generation, several times faster than real-time on both CPU and GPU. For training, a multi-scale spectral reconstruction loss can be used, followed by an adversarial loss. This enables a model capable of synthesizing high-quality outputs to be obtained after less than 2 days of training on a single GPU.

[0070] The potential applications and benefits of this invention are as follows:

[0071] This invention is applicable to text-to-speech and achieves quality, specifically the quality of generated speech for TTS and copy synthesis, approaching that of WaveNet and natural speech. It also offers fast training, making the model easy to retrain quickly and with personalization capabilities. It uses less memory because it is a relatively small neural network model. Finally, the proposed invention offers advantages in terms of complexity, achieving a very good quality / complexity tradeoff.

[0072] This invention can also be applied to speech enhancement, providing a low-complexity and robust solution for generating clean speech from noisy speech.

[0073] This invention can also be applied to speech coding, where it can significantly reduce the bit rate by transmitting only the parameters needed to adjust the neural vocoder. Furthermore, in this application, the lightweight neural vocoder-based solution is suitable for embedded systems, particularly for upcoming (end-user) devices (UEs) equipped with GPUs or neural processing units (NPUs).

[0074] Embodiments of this application relate to an audio generator configured to generate an audio signal from an input signal and target data, the target data representing the audio signal. The audio generator includes a first processing block configured to receive first data obtained from the input signal and output first output data, wherein the first output data includes multiple channels, and a second processing block configured to receive the first output data as second data or data obtained from the first output data. The first processing block includes: a set of conditions for a learnable layer for each channel of the first output data, the set of conditions for the learnable layer being configured to process the target data to obtain conditional feature parameters; and a style element configured to apply the conditional feature parameters to the first data or normalized first data; and wherein the second processing block is configured to combine multiple channels of the second data to obtain an audio signal.

[0075] According to one embodiment, the condition set of the learnable layer consists of one or two convolutional layers.

[0076] According to one embodiment, the first convolutional layer is configured to convolve the target data or upsampled target data using a first activation function to obtain first convolutional data.

[0077] According to one embodiment, the condition set and style element of the learnable layer are part of the weight layer in the residual block of a neural network that includes one or more residual blocks.

[0078] According to one embodiment, the audio generator further includes a normalization element configured to normalize the first data. For example, the normalization element can normalize the first data to a normal distribution with zero mean and unit variance.

[0079] According to one embodiment, the audio signal is a speech audio signal.

[0080] According to one embodiment, the target data is preferably upsampled by a factor of 2, a multiple of 2, or a power of 2 through nonlinear interpolation. In some examples, conversely, a factor greater than 2 may be used.

[0081] According to one embodiment, the first processing block further includes another set of learnable layers configured to process data derived from the first data using a second activation function, wherein the second activation function is a gated activation function.

[0082] According to one embodiment, another set of learnable layers consists of one or two convolutional layers.

[0083] According to one embodiment, the second activation function is the softmax-gated hyperbolic tangent TanH function.

[0084] According to one embodiment, the first activation function is a leakage rectified linear unit (leakage ReLU) function.

[0085] According to one embodiment, the convolution operation is run with a maximum dilation factor of 2.

[0086] According to one embodiment, the audio generator includes eight first processing blocks and one second processing block.

[0087] According to one embodiment, the first data has a lower dimension than the audio signal. The first data may have a first dimension or at least one dimension lower than the audio signal. The first data may have one dimension lower than the audio signal, but have more channels than the audio signal. The total number of samples of the first data across all dimensions may be lower than the audio signal.

[0088] According to one embodiment, the target data is a spectrum, preferably a Mel spectrum, or a bitstream.

[0089] According to one embodiment, the target data is obtained from text, the target data is a compressed representation of audio data, or the target data is a degraded audio signal.

[0090] Further embodiments relate to a method for generating an audio signal from an input signal and target data via an audio generator, the target data representing the audio signal. The method includes receiving first data obtained from the input signal via a first processing block; processing the target data for each channel of the first output data via a conditional set of a learnable layer of the first processing block to obtain conditional feature parameters; applying the conditional feature parameters to the first data or normalized first data via a style element of the first processing block; outputting first output data including multiple channels via the first processing block; receiving the first output data or data derived from the first output data as second data via a second processing block; and combining the multiple channels of the second data via the second processing block to obtain an audio signal.

[0091] Normalization can include, for example, normalizing the first data to a normal distribution with zero mean and unit variance.

[0092] The method can also be provided with any feature or combination of features from the audio generator.

[0093] Further embodiments relate to a method for training an audio generator as described above, wherein training includes repeating the steps of any of the methods described above once or multiple times.

[0094] According to one embodiment, the method for training further includes evaluating the generated audio signal by at least one evaluator and adjusting the weights of the audio generator based on the evaluation result, wherein the evaluator is preferably a neural network.

[0095] According to one embodiment, the method for training further includes adjusting the weights of the evaluator based on the evaluation results.

[0096] According to one embodiment, training includes optimizing the loss function.

[0097] According to one embodiment, optimizing the loss function includes calculating a fixed metric between the generated audio signal and a reference audio signal.

[0098] According to one embodiment, calculating a fixed metric includes calculating one or more spectral distortions between the generated audio signal and a reference signal.

[0099] According to one embodiment, one or more spectral distortions are calculated on the amplitude or logarithmic amplitude of the spectral representations of the generated audio signal and the reference signal, and / or at different time or frequency resolutions.

[0100] According to one embodiment, optimizing the loss function includes obtaining one or more adversarial metrics by randomly providing and evaluating a representation of the generated audio signal or a representation of a reference audio signal by one or more evaluators, wherein the evaluation includes classifying the provided audio signal into a predetermined number of categories that indicate the naturalness of the audio signal.

[0101] According to one embodiment, optimizing the loss function includes computing a fixed metric and deriving an adversarial metric through one or more evaluators.

[0102] According to one embodiment, the audio generator is first trained using a fixed metric.

[0103] According to one embodiment, four evaluators derive four adversarial metrics.

[0104] According to one embodiment, the evaluator operates after decomposing the representation of the generated audio signal or the representation of the reference audio signal through a filter bank.

[0105] According to one embodiment, each evaluator receives one or more portions of a representation of the generated audio signal or a representation of a reference audio signal as input.

[0106] According to one embodiment, a portion of the signal is generated by sampling a random window from the input signal using a random window function.

[0107] According to one embodiment, the sampling of the random window is repeated multiple times for each evaluator.

[0108] According to one embodiment, the number of times a random window is sampled for each evaluator is proportional to the length of the representation of the generated audio signal or the representation of the reference audio signal.

[0109] Further embodiments relate to a computer program product including a program for a processing device, the program including software code portions for performing steps of the methods described herein when the program is run on the processing device.

[0110] According to one embodiment, a computer program product includes a computer-readable medium on which software code is partially stored, wherein the program can be directly loaded into the internal memory of a processing device.

[0111] Further embodiments relate to a method for generating audio signals, including a mathematical model configured to output audio samples from an input sequence representing audio data to be generated at a given time step, wherein the mathematical model is configured to shape a noise vector to create output audio samples using the input representative sequence.

[0112] According to one embodiment, a mathematical model is trained using audio data. According to one embodiment, the mathematical model is a neural network. According to one embodiment, the network is a feedforward network. According to one embodiment, the network is a convolutional network.

[0113] According to one embodiment, the noise vector may have a lower dimension than the audio signal to be generated. The first data may have a first dimension or at least one dimension lower than the audio signal. The first data may have a lower total number of samples across all dimensions than the audio signal. The first data may have one dimension lower than the audio signal but have more channels than the audio signal.

[0114] According to one embodiment, the time-adaptive denormalization (TADE) technique is used to adjust the mathematical model using the input representative sequence, and thus to shape the noise vector.

[0115] According to one embodiment, a modified softmax-gated Tanh activates each layer of the neural network.

[0116] According to one embodiment, the convolution operation is run with a maximum dilation factor of 2.

[0117] According to one embodiment, the noise vector and the input representative sequence are upsampled to obtain the output audio at the target sampling rate.

[0118] According to one embodiment, upsampling is performed sequentially in different layers of the mathematical model.

[0119] In one embodiment, the upsampling factor for each layer is 2 or a multiple of 2, such as a power of 2. In some examples, the value of the upsampling factor can typically be greater than 2.

[0120] According to one embodiment, the generated audio signal is used for text-to-speech applications, where the input represents a sequence derived from text.

[0121] According to one embodiment, the generated audio signal is used for an audio decoder, wherein the input representation sequence is a compressed representation of the original audio to be transmitted or stored.

[0122] According to one embodiment, the generated audio signal is used to improve the audio quality of a degraded audio signal, wherein the input representative sequence is obtained from the degraded signal.

[0123] Further embodiments relate to training a neural network for audio generation, wherein the neural network outputs audio samples from an input sequence representing audio data to be generated at a given time step, wherein the neural network is configured to shape a noise vector to create output audio samples using the input representative sequence, wherein the neural network is designed as described above, and wherein training is designed to optimize a loss function.

[0124] According to one embodiment, the loss function includes a fixed metric calculated between the generated audio signal and the reference audio signal.

[0125] According to one embodiment, the fixed metric is one or more spectral distortions calculated between the generated audio signal and the reference signal.

[0126] According to one embodiment, one or more spectral distortions are calculated on the amplitude or logarithmic amplitude of the spectral representation of the generated audio signal and the reference signal.

[0127] According to one embodiment, one or more spectral distortions forming a fixed metric are calculated at different time or frequency resolutions.

[0128] According to one embodiment, the loss function includes an adversarial metric obtained by an additional discriminative neural network, wherein the discriminative neural network receives a representation of the generated audio signal or a representation of a reference audio signal as input, and wherein the discriminative neural network is configured to evaluate whether the generated audio signal is authentic.

[0129] According to one embodiment, the loss function includes a fixed metric and an adversarial metric derived from an additional discriminative neural network.

[0130] According to one embodiment, the neural network that generates audio samples is first trained using only a fixed metric.

[0131] According to one embodiment, the adversarial metric is obtained by four discriminative neural networks.

[0132] According to one embodiment, the discriminative neural network operates after the input audio signal is decomposed through a filter bank.

[0133] According to one embodiment, each discriminative neural network receives one or more random window versions of the input audio signal as input.

[0134] According to one embodiment, random window sampling is repeated multiple times for each discriminative neural network.

[0135] According to one embodiment, for each discriminative neural network, the number of sampling random windows is proportional to the length of the input audio samples. Detailed Implementation

[0136] Figure 6 An example of an audio generator 10 is shown, which can generate (e.g., synthesize) an audio signal (output signal) 16, for example, according to StyleMelGAN. The output audio signal 16 can be generated based on an input signal 14 (also called a latent signal, and can be noise, such as white noise) and target data 12 (also called an “input sequence”). For example, the target data 12 can include (e.g., is) a spectrogram (e.g., a Mel spectrogram) that provides, for example, mapping a time-sample sequence onto a Mel scale. Alternatively or concurrently, the target data 12 can include (e.g., is) a bitstream. For example, the target data can be or include text that will be reproduced in the audio (e.g., text-to-speech). The target data 12 is typically processed to obtain speech that can be recognized as natural by a human listener. The input signal 14 can be noise (which itself does not carry useful information), such as white noise; however, in generator 10, a noise vector extracted from the noise is styled into a noise vector with acoustic features modulated by the target data 12 (e.g., at 77). Finally, the output audio signal 16 will be understood as speech by a human listener. like Figure 1 As shown, noise vector 14 can be a 128x1 vector (a single sample, such as a time-domain sample or a frequency-domain sample, and 128 channels). Noise vector 14 of other lengths can be used in other examples.

[0137] First processing block 50 Figure 6 As shown. (For example, in...) Figure 1 In the middle), the first processing block 15 can consist of multiple blocks (in Figure 1In this context, each of blocks 50a, 50b, 50c, 50d, 50e, 50f, 50g, and 50h is instantiated. Blocks 50a-50h can be understood as forming a single block 40. It will be shown that in the first processing blocks 40 and 50, the conditional set of learnable layers (e.g., 71, 72, 73) can be used to process the target data 12 and / or the input signal 14. Therefore, conditional feature parameters 74 and 75 (in...) can be obtained during training, for example, through convolution. Figure 3 (also referred to as gammaγ and betaβ). Therefore, learnable layers 71-73 can be part of the weight layers of a learning network, or more generally, another learning structure. The first processing blocks 40, 50 may include at least one style element 77. At least one style element 77 can output first output data 69. At least one style element 77 can apply conditional feature parameters 74, 75 to the input signal 14 (potential) or the first data 15 obtained from the input signal 14.

[0138] The first output data 69 at each block 50 is located in multiple channels. The audio generator 10 may include a second processing block 45 (in... Figure 1 (Shown as blocks 42, 44, 46). The second processing block 45 can be configured to combine multiple channels 47 of the first output data 69 (which is input as second input data or second data) to obtain the output audio signal 16 in a single channel but in a sampling sequence.

[0139] The term "channel" is not understood in the context of stereo, but rather in the context of neural networks (e.g., convolutional neural networks). For example, an input signal (e.g., latent noise) 14 can be represented in 128 channels (in the time-domain representation) because a channel sequence is provided. For instance, when the signal has 176 samples and 64 channels, it can be understood as a matrix of 176 columns and 64 rows, while when the signal has 352 samples and 64 channels, it can be understood as a matrix of 352 columns and 64 rows (other schematics are also possible). Therefore, the generated audio signal 16 (in... Figure 1 The 1x22528 row matrix generated in the middle can be understood as a mono signal. If a stereo signal is to be generated, the disclosed technique is simply repeated for each stereo channel to obtain multiple audio signals that are subsequently mixed.

[0140] At least the original input signal 14 and / or the generated speech 16 can be vectors. Conversely, the outputs of each block 30 and 50a-50h, 42, 44 typically have different dimensions. The first data may have a first dimension or at least one dimension lower than the audio signal. The first data may have a total number of samples lower than the audio signal across all dimensions. The first data may have one dimension lower than the audio signal but have more channels than the audio signal. In each block 30 and 50a-50h, the signal evolving from noise 14 to speech 16 can be upsampled. For example, in the upsampling block 30 preceding the first block 50a in the 50a-50h blocks, 88 upsampling operations are performed. Examples of upsampling can include, for example, the following sequences: 1) repetition of the same value, 2) insertion of zeros, 3) another repetition or insertion of zeros + linear filtering, etc.

[0141] The generated audio signal 16 can typically be a single-channel signal (e.g., 1x22528). If multiple audio channels are required (e.g., for stereo playback), the required procedure should, in principle, be iterated multiple times.

[0142] Similarly, the target data 12 can, in principle, be in a single channel (e.g., if it is text) or in multiple channels (e.g., in a spectrum). In either case, it can be upsampled (e.g., by factors of 2, powers of 2, multiples of 2, or values ​​greater than 2) to adapt to the dimensions (59a, 15, 69) of the signal evolving along subsequent layers (50a-50h, 42), for example, to obtain conditional feature parameters 74, 75 in the dimensions adapted to the signal dimensions.

[0143] When the first processing block 50 is instantiated in multiple blocks 50a-50h, for example, the number of channels can remain the same for multiple blocks 50a-50h. The first data may have a first dimension or at least one dimension lower than the audio signal. The first data may have a total number of samples in all dimensions lower than the audio signal. The first data may have one dimension lower than the audio signal, but have more channels than the audio signal.

[0144] The signals in subsequent blocks can have different dimensions from each other. For example, samples can be sampled more and more times, for example, from 88 samples to 22,528 samples in the last block 50h. Similarly, the target data 12 is also upsampled at each processing block 50. Therefore, the conditional feature parameters 74 and 75 can be adapted to the number of samples of the signal to be processed. Thus, the semantic information provided by the target data 12 is not lost in the subsequent layers 50a-50h.

[0145] It is important to understand that examples can be executed based on the paradigm of Generative Adversarial Networks (GANs). GANs include GAN generators 11 ( Figure 1 ) and GAN discriminator 100 ( Figure 2 The GAN generator 11 attempts to generate an audio signal 16 that is as close as possible to a real signal. The GAN discriminator 100 should identify whether the generated audio signal is real (e.g., ...). Figure 2 The audio signal 104 generated by the GAN generator is either real or fake (e.g., generated audio signal 16). Both the GAN generator 11 and the GAN discriminator 100 can be obtained as neural networks. The GAN generator 11 should minimize its loss (e.g., through gradient methods or other methods) and update its conditional feature parameters 74, 75 by taking into account the results of the GAN discriminator 100. The GAN discriminator 100 should reduce its own discrimination loss (e.g., through gradient methods or other methods) and update its own internal parameters. Thus, the GAN generator 11 is trained to provide increasingly better audio signals 16, while the GAN discriminator 100 is trained to identify the real signal 16 from the fake audio signals generated by the GAN generator 11. In general, it can be understood that the GAN generator 11 may include the functionality of the generator 10, but at least not the functionality of the GAN discriminator 100. Therefore, in most of the above, it can be understood that the GAN generator 11 and the audio generator 10 may have more or less the same characteristics except for the characteristics of the discriminator 100. The audio generator 10 may include the discriminator 100 as an internal component. Therefore, the GAN generator 11 and the GAN discriminator 100 can together constitute the audio generator 10. In the example where the GAN discriminator 100 is absent, the audio generator 10 can be uniquely constituted by the GAN generator 11.

[0146] As explained by the “conditional set of learnable layers”, the audio generator 10 can be obtained according to the paradigm of conditional GANs, such as based on conditional information. For example, the conditional information can be composed of target data (or its upsampled version) 12, from which the conditional set of layers 71-73 (weight layers) is trained and conditional feature parameters 74, 75 are obtained. Therefore, the style element 77 is constrained by the learnable layers 71-73.

[0147] These examples can be based on convolutional neural networks. For example, it could be a small matrix (e.g., a filter or kernel) of 3x3 (or 4x4, etc.) convolved (convolved) along a larger matrix (e.g., channel x-sampled latent or input signal and / or spectrum and / or spectrum or upsampled spectrum or more general target data 12), such as implying a combination between the elements of the filter (kernel) and the elements of the larger matrix (activation map, or activation signal) (e.g., multiplication and sum of products; dot product, etc.). During training, the elements of the filter (kernel) are obtained (learned), which are the elements that minimize the loss. During inference, the elements of the filter (kernel) obtained during training are used. Examples of convolution are located in blocks 71-73, 61a, 61b, 62a, 62b (see below). In blocks with conditional (e.g.) Figure 3In the case of block 60), the convolution is not necessarily applied to the signal evolving from the input signal 14 through intermediate signals 59a(15), 69, etc., towards the audio signal 16, but can be applied to the target signal 14. In other cases (e.g., at blocks 61a, 61b, 62a, 62b), the convolution may not be conditional and can be applied, for example, directly to signals 59a(15), 69, etc., evolving from the input signal 14 towards the audio signal 16. Figure 3 and Figure 4 As can be seen, both conditional convolution and unconditional convolution can be performed.

[0148] In some examples, there can be activation functions downstream of the convolution (ReLU, TanH, softmax, etc.), which can vary depending on the desired effect. ReLU can map the maximum value between 0 and the value obtained during convolution (in practice, it keeps the same value if positive and outputs 0 if negative). Leaking ReLU can output x when x > 0 and 0.1*x when x ≤ 0, where x is the value obtained during convolution (in some examples, another value can be used instead of 0.1, such as a predetermined value in the range of 0.1 ± 0.05). TanH (which can be implemented, for example, at blocks 63a and / or 63b) can provide the hyperbolic tangent of the value obtained during convolution, for example:

[0149] TanH(x)=(e x -e -x ) / (e x +e -x )

[0150] Here, x is the value obtained during convolution (e.g., at blocks 61a and / or 61b). Softmax (e.g., applied at blocks 64a and / or 64b) applies an exponent to each element of the result of the convolution (e.g., obtained at blocks 62a and / or 62b) and normalizes it by dividing by the sum of the exponents. Softmax (e.g., at 64a and / or 64b) can provide a probability distribution for the entries in the matrix generated by the convolution (e.g., at 62a and / or 62b). After applying the activation function, a pooling step (not shown in the figure) may be performed in some examples, but this step can be avoided in other examples.

[0151] Figure 4 As shown, there can also be a softmax-gated TanH function, for example, multiplying the result of the TanH function (e.g., obtained at 63a and / or 63b) with the result of the softmax function (e.g., obtained at 64a and / or 64b) (e.g., obtained at 65a and / or 65b).

[0152] Multi-layer convolutions (e.g., conditional sets of learnable layers) can be one after another and / or in parallel to each other to improve efficiency. They can also be repeated in different layers if activation functions and / or pooling are provided (or, for example, different activation functions can be applied to different layers).

[0153] The input signal 14 (e.g., noise) is processed in different steps to become the generated audio signal 16 (e.g., under conditions set by the condition set of learnable layers 71-73, and on parameters 74, 75 learned by the condition set of learnable layers 71-73). Therefore, the input signal should be understood as being in the processing direction ( Figure 6 From 14 to 16, the process evolves towards becoming the generated audio signal 16 (e.g., speech). Conditions will be generated primarily based on the target signal 12 and training (to achieve the optimal set of parameters 74, 75).

[0154] It is also important to note that multiple channels of the input signal (or any evolution thereof) can be considered to have a set of learnable layers and associated style elements. For example, each row of matrices 74 and 75 is associated with a specific channel of the input signal (or one of its evolutions) and is therefore derived from a specific learnable layer associated with that specific channel. Similarly, style element 77 can be considered to consist of multiple style elements (each element for each row of the input signals x, c, 12, 76, 76′, 59, 59a, 59b, etc.).

[0155] Figure 1 An example of audio generator 10 is shown (which can illustrate) Figure 6 The audio generator 10 may also include (e.g., a GAN generator 11). The target data 12 is indicated as a Mel spectrogram, the input signal 14 may be latent noise, and the output of signal 16 may be speech (as described above, although other examples may exist). It can be seen that the input signal 14 has only one sample and 128 channels. The noise vector 14 can be obtained from a vector with 128 channels (but other numbers are also possible) and may have a zero-mean normal distribution. The noise vector can follow the formula z~ The noise vector can be generated as 128-dimensional random noise with a mean of 0, and its autocorrelation matrix (squared 128x128) is equal to unit I (different choices can be made). Therefore, in the example, the generated noise is perfectly decorrelated across channels and has a variance of 1 (energy). This can be achieved at every 22,528 generated samples (or other numbers can be chosen for different examples); therefore, the dimension can be 1 on the time axis and 128 on the channel axis.

[0156] As will be shown, noise vector 14 is processed stepwise (e.g., in blocks 50a-50h, 42, 44, 46, etc.) in order to evolve from, for example, noise 14 to, for example, speech 16 (e.g., the evolving signals will be indicated by different signals 15, 59a, x, c, 76′, 79, 79a, 59b, 79b, 69, etc.).

[0157] In block 30, the input signal (noise) 14 can be upsampled to have 88 samples (different numbers are possible) and 64 channels (different numbers are possible).

[0158] It can be seen that the eight processing blocks 50a, 50b, 50c, 50d, 50e, 50f, 50g, and 50h collectively reflect... Figure 6 The first processing block 50 can increase the number of samples by performing upsampling (e.g., maximum 2-upsampling). Along blocks 50a, 50b, 50c, 50d, 50e, 50f, 50g, and 50h, the number of channels can always remain the same (e.g., 64). For example, the number of samples can be the number of samples per second (or other time units): we can obtain sound exceeding 22kHz at the output of block 50h.

[0159] Each of blocks 50a-50h(50) can also be a TADEResBlock (a residual block in the TADE (time-adaptive denormalization) context). Notably, each block 50a-50h can be conditioned on target data 12 (e.g., a Mel spectrogram).

[0160] Second processing block 45 ( Figure 1 and Figure 6 This allows us to obtain a single channel and multiple samples in a single dimension. It can be seen that using another TADEResBlock42 (a further step of blocks 50a-50h) reduces this to a single channel. Then, convolutional layers 44 and activation functions (e.g., TanH46) can be performed. Afterward, speech 16 is obtained (and can be stored, rendered, encoded, etc.).

[0161] For example, at least one of blocks 50a-50h (or each of them in a specific example) can be a residual block. A residual block predicts only the residual components of the signal that evolves from the input signal 14 (e.g., noise) to the output audio signal 16. The residual signal is only a part (residual component) of the main signal. For example, multiple residual signals can be added together to obtain the final output audio signal 16.

[0162] Figure 4An example of one of blocks 50a-50h (50) is shown. It can be seen that each block 50 is input with first data 59a, which is either input signal 14 (or an upsampled version thereof, such as the upsampled version output by upsampled block 30) or the output from the previous block. For example, block 50b can be input with the output of block 50a; block 50c can be input with the output of block 50b, and so on.

[0163] Therefore, in Figure 4 As can be seen, the first data 59a provided to block 50 (50a-50h) is processed, and its output is output signal 69 (which will be provided as input to subsequent blocks). As indicated by line 59a', the main component of the first data 59a input to the first processing blocks 50a-50h (50) actually bypasses most of the processing of the first processing blocks 50a-50h (50). For example, blocks 60a, 61a, 62a, 63a, 65a, 60b, 61b, 62b, 63b, 64b, and 65b are bypassed by bypass line 59a'. Subsequently, the first data 59a will be processed in adder 65c (as shown in the image). Figure 4 As shown (but not shown), the addition is applied to the residual portion 64b' at the bypass line 59a' and adder 65c. This addition can be understood as instantiating the fact that each block 50 (50a-50h) processes the operation on the residual signal and then adds it to the main part of the signal. Therefore, each of the blocks 50a-50h can be considered a residual block.

[0164] It is worth noting that the addition at adder 65c does not necessarily need to be performed within residual blocks 50 (50a-50h). A single addition of multiple residual signals 65b' (each output by each residual block 50a-50h) can be performed (e.g., at the adder block in the second processing block 45). Therefore, different residual blocks 50a-50h can operate in parallel with each other.

[0165] exist Figure 4 In the example, each block 50 may repeat its convolutional layer twice (e.g., first at copy 600, including at least one of blocks 60a, 61a, 62a, 63a, 64a, 65a, and obtaining signal 59b; then at copy 601, including at least one of blocks 60b, 61b, 62b, 63b, 64b, 65b, and obtaining signal 65b', which may be added to the principal component 59a').

[0166] For each copy (600, 601), the condition set and style element 77 of learnable layers 71-73 are applied to the signal evolving from input signal 16 to audio output signal 16 (e.g., twice per block 50). First time-adaptive denormalization (TADE) is performed on the first data 59a at the first copy 600 at TADE block 60a. Under the conditions set by target data 12, TADE block 60a modulates the first data 59a (input signal or, for example, processed noise). In the first TADE block 60a, upsampling of target data 12 can be performed at upsampling block 70 to obtain an upsampled version 12' of target data 12. Upsampling can be obtained through nonlinear interpolation, for example, using a factor of 2, a power of 2, a multiple of 2, or other values ​​greater than 2. Therefore, in some examples, the spectrum 12' can be made to have the same dimensions (e.g., conformal) as the signal to be conditioned on (76, 76', x, c, 59, 59a, 59b, etc.). The application of style information to the processed noise (first data) (76, 76′, x, c, 59, 59a, 59b, etc.) can be performed at block 77 (style element). In a subsequent copy 601, another TADE block 60b can be applied to the output 59b of the first copy 600. Figure 3 An example of TADE module 60 (60a, 60b) is provided (see below). After modulating the first data 59a, convolutions 61a and 62a are performed. Subsequently, activation functions TanH and softmax (e.g., constituting a softmax-gated TanH function) are also executed (63a, 64a). The outputs of activation functions 63a and 64a are multiplied at multiplier block 65a (e.g., to instantiate the gating) to obtain result 59b. If two different copies 600 and 601 (or more than two copies) are used, the pathways 60a, 61a, 62a, 63a, 64a, and 65a are repeated.

[0167] In the example, the first and second convolutions located at 61b and 62b, downstream of TADE blocks 60a and 60b respectively, can be performed with the same number of elements in the kernel (e.g., 9, or 3x3). However, the second convolutions 61b and 62b can have an inflation factor of 2. In the example, the maximum inflation factor of the convolutions can be 2.

[0168] Figure 3An example of TADE block 60 (60a, 60b) is shown. It can be seen that the target data 12 can be upsampled, for example, to conform to the input signal (or signals derived therefrom, such as 59, 59a, 76', also called latent or activation signals). Here, convolutions 71, 72, 73 (intermediate values ​​of the target data 12 are indicated by 71') can be performed to obtain parameters γ (gamma, 74) and β (beta, 75). Convolutions at any point in 71, 72, 73 may also require rectified linear units (ReLU), or leaky rectified linear units (leaky ReLU). Parameters γ and β can have the same dimensions as the activation signal (the signal processed to evolve from the input signal 14 into the generated audio signal 16, represented here in normalized form as x, 59, or 76'). Therefore, when the activation signal (x, 59, 76′) has two dimensions, γ and β (74 and 75) also have two dimensions, and each of them can overlap with the activation signal (the length and width of γ and β can be the same as the length and width of the activation signal). At style element 77, conditional feature parameters 74 and 75 are applied to the activation signal (first data 59a or 59b output by multiplier 65a). However, it should be noted that the activation signal 76′ can be a normalized version of the first data 59, 59a, 59b (15) (10 at instance normalization block 76). It should also be noted that the formula (γx+β) shown in style element 77 can be an element-wise product, rather than a convolutional product or a dot product.

[0169] The output signal follows style element 77. There is not necessarily an activation function downstream of convolutions 72 and 73. It should also be noted that parameter γ(74) can be interpreted as variance, and β(75) as bias. Furthermore, Figure 1 Block 42 in the middle can be instantiated as Figure 3 Block 50 in the middle. Then, for example, convolutional layer 44 reduces the number of channels to 1, and then TanH 56 is performed to obtain speech 16.

[0170] Figure 7 An example of evolution in one of the copies 600 and 601 of one of blocks 50a-50h is shown.

[0171] Target data 14 (e.g., Mel spectrogram); and

[0172] The potential noise c(12), also indicated by 59a, or as the signal that evolves from the input signal 12 into the generated audio signal 16.

[0173] The following programs can be executed:

[0174] 1) Spectrum 12 undergoes at least one of the following steps:

[0175] a. Sampled at upsampling block 70 to obtain upsampled spectrum 12′;

[0176] b. Perform convolution at convolutional layers 71-73 (part of the weight layer) (e.g., along the upsampled spectrum 12' convolution kernel 12a);

[0177] C. Obtain (learn) γ(74) and β(75);

[0178] D. Apply γ(74) and β(75) (e.g., by convolution) to the latent signal 59a(15) derived from the input signal 14 and the generated audio signal 16.

[0179] GAN Discriminator

[0180] During training, it can be used Figure 2 The GAN discriminator 100 in the training process obtains parameters 74 and 75, for example, applied to the input signal 12 (or its processed and / or normalized version). Training can be performed before inference, and parameters 74 and 75 can, for example, be stored in non-transitory memory and used subsequently (however, in some examples, parameters 74 or 75 are also computed online).

[0181] The GAN discriminator 100 has the function of learning how to recognize generated audio signals (e.g., synthesized audio signals 16 as described above) from real input signals (e.g., real speech) 104. Therefore, the role of the GAN discriminator 100 is mainly played during training (e.g., for learning parameters 72 and 73), and is seen in the adversarial position of the role of the GAN generator 11 (which can be viewed as an audio generator 10 without the GAN discriminator 100).

[0182] Generally, the GAN discriminator 100 can be input with an audio signal 16 synthesized by the GAN generator 10 and a real audio signal (e.g., real speech) 104 obtained, for example, through a microphone, and these signals are processed to obtain a metric (e.g., loss) to be minimized. The real audio signal 104 can also be considered as a reference audio signal. During training, the operations for synthesizing speech 16 as described above can be repeated, for example, multiple times, to obtain, for example, parameters 74 and 75.

[0183] In this example, instead of analyzing the entire reference audio signal 104 and / or the entire generated audio signal 16, only a portion of them (e.g., a segment, slice, window, etc.) can be analyzed. The signal portions generated from the generated audio signal 16 and from random windows (105a-105d) sampled from the reference audio signal 104 are obtained. For example, a random window function can be used such that which windows 105a, 105b, 105c, and 105d will be used is not predefined a priori. Furthermore, the number of windows is not necessarily four and can vary.

[0184] Within the window (105a-105d), PQMF (orthogonal mirror filter bank (PQMF)) 110 can be applied. Thus, sub-band 120 is obtained. Thus, a decomposition (110) of the representation of the generated audio signal (16) or the representation of the reference audio signal (104) is obtained.

[0185] Evaluation can be performed using evaluation block 130. Multiple evaluators 132a, 132b, 132c, 132d (indicated by 132 composite) can be used (different numbers can be used). Typically, each window 105a, 105b, 105c, 105d can be input to the corresponding evaluator 132a, 132b, 132c, 132d. Sampling of random windows (105a-105d) can be repeated multiple times for each evaluator (132a-132d). In the example, the number of times random windows (105a-105d) are sampled for each evaluator (132a-132d) can be proportional to the length of the representation of the generated audio signal or the representation of the reference audio signal (104). Therefore, each evaluator (132a-132d) can receive as input one or more portions (105a-105d) of the representation of the generated audio signal (16) or the representation of the reference audio signal (104).

[0186] Each estimator 132a-132d can itself be a neural network. In particular, each estimator 132a-132d can follow the paradigm of a convolutional neural network. Each estimator 132a-132d can be a residual estimator. Each estimator 132a-132d can have parameters (e.g., weights) that are adjusted during training (e.g., in a manner similar to one of those explained above).

[0187] like Figure 2 As shown, each evaluator 132a-132d also performs downsampling (e.g., by 4 or by another downsampling ratio). For each evaluator 132a-132d, the number of channels is increased (e.g., by 4, or in some examples by the same number as the downsampling ratio).

[0188] Convolutional layers 131 and / or 134 can be provided upstream and / or downstream of the evaluator. The upstream convolutional layer 131 may have, for example, a kernel of dimension 15 (e.g., 5x3 or 3x5). The downstream convolutional layer 134 may have, for example, a kernel of dimension 3 (e.g., 3x3).

[0189] During training, the loss function (adversarial loss) 140 can be optimized. The loss function 140 may include a fixed metric between the generated audio signal (16) and the reference audio signal (104) (e.g., obtained during the pre-training step). The fixed metric can be obtained by calculating one or more spectral distortions between the generated audio signal (16) and the reference audio signal (104). This distortion can be measured by considering the following factors:

[0190] - The amplitude or logarithmic amplitude represented by the spectrum of the generated audio signal (16) and the reference audio signal (104), and / or

[0191] - Different time or frequency resolutions.

[0192] In the example, the adversarial loss can be obtained by randomly providing and evaluating a representation of the generated audio signal (16) or a representation of a reference audio signal (104) by one or more evaluators (132). The evaluation may include classifying the provided audio signal (16, 132) into a predetermined number of classes, which indicate a pre-trained classification level of the naturalness of the audio signal (14, 16). For example, the predetermined number of classes could be "real" vs. "fake".

[0193] Examples of losses can be obtained as follows:

[0194]

[0195] in:

[0196] x represents the actual voice 104.

[0197] z represents the potential noise 14 (or more generally, the input signal or the first data or the potential noise).

[0198] s is the Mel spectrum of x (or more generally, the target signal 12).

[0199] D(...) is the output of the evaluator in terms of probability distribution (D(...) = 0 means "definitely false", D(...) = 1 means "definitely true").

[0200] Spectral reconstruction loss It is still used for regularization to prevent adversarial artifacts. The final loss could be, for example:

[0201]

[0202] Where each i is the contribution of each evaluator at position 132a-132d (e.g., each evaluator (132a-132d) provides a different D). i )and It is the pre-trained (fixed) loss.

[0203] During training, The minimum value can be searched, for example, it can be expressed as follows:

[0204]

[0205] Other types of minimization can be performed.

[0206] Generally, the minimum adversarial loss 140 is associated with the optimal parameters (e.g., 74, 75) to be applied to the style element 77.

[0207] discuss

[0208] In the following description, examples of the present disclosure will be described in detail using the accompanying description. In order to provide a more thorough explanation of embodiments of the present disclosure, numerous details will be described in the description below. However, it will be apparent to those skilled in the art that other embodiments may be practiced without these specific details. Features of the different embodiments described may be combined with each other unless the features of the corresponding combinations are mutually exclusive or explicitly exclude such combinations.

[0209] It should be noted that identical or similar elements, or elements having the same function, may have identical or similar reference numerals, or be designated identically. Elements with identical or similar reference numerals may be described repeatedly, or elements compared in identical or similar figures may generally be omitted. Descriptions of elements with identical or similar reference numerals, or those labeled as identical, are interchangeable.

[0210] Neural vocoders have proven superior to classical methods in synthesizing natural, high-quality speech in many applications, such as text-to-speech, speech coding, and speech enhancement. The first pioneering generative neural network to synthesize high-quality speech was WaveNet, and many other methods were developed shortly afterward. These models offered state-of-the-art quality but typically came at very high computational cost and very slow synthesis. In recent years, a large number of computationally less expensive speech generation models have emerged. Some of these are optimized versions of existing models, while others utilize integrations with classical methods. Furthermore, many entirely new methods have been introduced, often relying on GANs. Most GAN vocoders offer very fast generation on GPUs, but at the cost of sacrificing the quality of the synthesized speech.

[0211] One of the main goals of this work is to propose a GAN architecture, which we call StyleMelGAN (implemented in the audio generator 10), that can synthesize very high-quality speech with low computational cost and fast training 16. The StyleMelGAN generator network may contain 3.86M trainable parameters and synthesizes speech at a frequency of 22.05kHz, which is 2.6 times faster than real-time on a CPU and more than 54 times faster than on a GPU. For example, the model can consist of 8 upsampling blocks that gradually reduce the low-dimensional noise vector (e.g., ...) Figure 1 30) is transformed into the original speech waveform (e.g., 16). Synthesis can be conditioned on the Mel spectrogram of the target speech (or more generally on the target data 12), which can be inserted into each generator block (50a-50h) via time-adaptive denormalization (TADE) layers (60, 60a, 60b). This method of inserting conditional features is very efficient and, to our knowledge, is a novel approach in the audio domain. The adversarial loss is computed through a set of four discriminators 132a-132d (e.g., by...). Figure 2 The structure is as follows (in GAN discriminator 100) (but in some examples there may be a different number of discriminators), each discriminator operating after a distinguishable pseudo-orthogonal mirror filter bank (PQMF) 110. This allows for analysis of different frequency bands of the speech signal (104 or 16) during training. To make training more robust and conducive to generalization, the discriminators (e.g., four discriminators 132a-132d) are not conditioned on the input acoustic features used by generator 10, and the speech signal (104 or 16) is sampled using random windows (e.g., 105a-105d).

[0212] In summary, StyleMelGAN is proposed, a low-complexity GAN for high-quality speech synthesis conditioned on Mel spectrograms (e.g., 12) via TADE layers (e.g., 60, 60a, 60b). Generator 10 can be highly parallelized. Generator 10 can be fully convolutional. The aforementioned generator 10 can be adversarially trained using an ensemble of PQMF multisampled random window discriminators (e.g., 132a-132d), which can be regularized using a multi-scale spectral reconstruction loss. The quality of the generated speech can be evaluated using objective (e.g., Fréchet scores) and / or subjective assessments. Two listening tests, the MUSHRA test in a replicated synthesis scenario and the P.800ACR test in a TTS scenario, both demonstrate that StyleMelGAN achieves state-of-the-art speech quality.

[0213] Existing neural vocoders typically synthesize speech signals directly in the time domain by modeling the amplitude of the final waveform. Most of these models are generative neural networks, meaning they model the probability distribution of observed speech samples in natural speech signals. They can be categorized into autoregressive and non-autoregressive or parallel models; the former explicitly decomposes the distribution into a product of conditional distributions, while the latter directly models the joint distribution. Autoregressive models such as WaveNet, SampleRNN, and WaveRNN have been reported for synthesizing high-perceptual-quality speech signals. A large class of non-autoregressive models is the normalization flow model, such as WaveGlow. Hybrid approaches use inverse autoregressive flow, which employs a factorial transformation between the noisy latent representation and the target speech distribution. The examples above primarily refer to autoregressive neural networks.

[0214] Early applications of GANs in audio include WaveGAN for unconditional speech generation and GAN-Synth for music generation. MelGAN learns the mapping between the Mel spectrogram of a speech segment and its corresponding time-domain waveform. It ensures faster performance than real-time generation and leverages adversarial training of a multi-scale discriminator regularized by a spectral reconstruction loss. GAN-TTS was the first GAN vocoder to use unique adversarial training for acoustic feature-based speech generation. Its adversarial loss is computed using an ensemble of conditional and unconditional random window discriminators. Parallel WaveGAN uses a generator, structurally similar to WaveNet, trained with an unconditional discriminator regularized by a multi-scale spectral reconstruction loss. A similar idea is used in Multiband-MelGAN, which generates each subband of the target speech separately, saving computational resources, and then uses synthetic PQMF to obtain the final waveform. Its multi-scale discriminator evaluates the full-band speech waveform and is regularized using a multi-band scale spectral reconstruction loss. Research in this area is very active, and we can cite recent GAN vocoders such as VocGAN and HooliGAN.

[0215] Figure 1 The generator architecture of StyleMelGAN implemented in audio generator 10 is shown. The generator model upsamples the noise vector by asymptotic upsampling (e.g., in blocks 50a-50h) with the Mel spectrogram (or more generally, the target data) as condition 12. (exist Figure 1 (denoted by 30) is mapped to speech waveform 16 (e.g., at 22050Hz). It uses time-adaptive denormalization, TADE (see blocks 60, 60a, 60b), which is likely based on feature modulation of linear modulation of the normalized activation map. Figure 3 (76' in the middle). Modulation parameter γ (gamma, Figure 3 74) and β(beta, Figure 3 75) is adaptively learned from conditional features and has the same dimension as the latent signal in one example. This provides conditional features for all layers of the generator model, thus preserving the signal structure at all upsampling stages. In formula z~ In this example, 128 represents the number of channels for the potential noise (different numbers can be chosen in different examples). Therefore, random noise of dimension 128 can be generated with a mean of 0 and an autocorrelation matrix (128 × 128 squared) equal to unit I. Thus, in this example, the generated noise can be considered perfectly decorrelated between channels, with a variance of 1 (energy). This can be achieved at every 22,528 generated samples (or other numbers can be chosen for different examples); therefore, the dimension can be 1 on the time axis and 128 on the channel axis.

[0216] Figure 3 The structure of a portion of the audio generator 10 is shown, along with the structure of TADE blocks 60 (60a, 60b). The input activation c(76') is adaptively modulated via c⊙γ+β, where ⊙ denotes element-wise multiplication (notably, γ and β have the same dimension of the activation map; also note that c is...). Figure 3 The normalized version of x is c⊙γ+β, therefore c⊙γ+β is the normalized version of xγ+β, and can also be represented as x⊙γ+β. Before modulation of block 77, an instance normalization layer 76 is used. Layer 76 (the normalization element) normalizes the first data to a normal distribution with zero mean and unit variance. A softmax-gated Tanh activation function (e.g., ...) can be used. Figure 4 The first one, instantiated from blocks 63a-64a-65a, and the second one, instantiated from blocks 63b-64b-65b, reportedly outperform the rectified linear unit ReLU function. Softmax gating (e.g., obtained via multiplication of 65a and 65b) allows for fewer artifacts in audio waveform generation.

[0217] Figure 4 The structure of a portion of the audio generator 10 is shown, along with TADEResBlock50 (which can be any of blocks 50a-50h), the basic building blocks of the generator model. The complete architecture is as follows. Figure 1 As shown. It comprises eight upsampling stages 50a-50h (other numbers are possible in other examples), including, for example, a TADEResBlock and a layer 601 that upsamples the signal 79b by a factor of 2, plus a final activation module 46 (as shown). Figure 1(As shown). The final activation consists of a TADEResBlock42, followed by a channel-variable convolutional layer44, for example, with a tanh nonlinearity46. For example, this design allows for convolution operations using a channel depth of 64, thus saving complexity. Furthermore, this upsampling process allows keeping the dilation factor below 2.

[0218] Figure 2 The structure of the Filter Bank Random Window Discriminator (FB-RWD) is shown. StyleMelGAN can be adversarially trained using multiple (e.g., four) discriminators 132a-132d, where, in the example, the architecture of discriminators 132a-132d does not employ average pooling downsampling. Furthermore, each discriminator (132a-132d) can operate on a random window (105a-105d) of slices from the input speech waveform (104 or 16). Finally, each discriminator (132a-132d) can analyze subbands 120 of the input speech signal (104 or 16) obtained by analyzing PQMF (e.g., 110). More precisely, in the example, we can use 1, 2, 4, and 8 subbands respectively calculated from selected random segments extracted from a 1-second waveform, consisting of 512, 1024, 2048, and 4096 samples. This enables multi-resolution adversarial evaluation of speech signals (104 or 16) in both the time and frequency domains.

[0219] Training GANs is known to be challenging. Random initializations of weights (e.g., 74 and 75) and adversarial losses (e.g., 140) can lead to severe audio artifacts and unstable training. To avoid this problem, the generator 10 can be pre-trained first using only a spectral reconstruction loss consisting of an error estimate from spectral convergence and logarithmic amplitudes calculated from different STFT analyses. The generator obtained in this way can produce very tonal signals, although with significant tailing in the high frequencies. Nevertheless, this is still a good starting point for adversarial training, which can benefit from a better harmonic structure compared to starting directly from a complete random noise signal. Adversarial training then drives the generation of naturalness by removing tonal effects and sharpening the tailed frequency bands. The hinge loss 140 is used to evaluate the adversarial metric, as shown in Equation 1 below.

[0220] (1)

[0221] Where x is the real speech 104, z is the latent noise 14 (or more generally, the input signal), and s is the Mel spectrum of x (or more generally, the target signal 12). It is worth noting that the spectral reconstruction loss... (140) is still used for regularization to prevent adversarial artifacts. The final loss (140) is calculated according to Equation 2, as shown below.

[0222] (2)

[0223] Weight normalization can be applied to all convolutional operations in G (or more precisely, the GAN generator 11) and D (or more precisely, the discriminator 100). In experiments, StyleMelGAN was trained on the LJSpeech corpus at a frequency of 22050 Hz using an NVIDIA Tesla V100 GPU. The log-amplitude Mel spectrogram was calculated for 80 Mel bands and normalized to have zero mean and unit variance. Of course, this is only one possibility; other values ​​are also possible. The generator was pre-trained for 100,000 steps using the Adam optimizer with a learning rate (lr) of [missing value]. g ) is 10 -4 β = {0.5, 0.9}. At the start of adversarial training, the learning rate of G (lr) is... g The value was set to 5*10. -5 And using FB-RWD with Adam optimizer, the discriminator learning rate (lr d ) is 2*10 -4 And β is the same. FB-RWD repeats random windowing 1s / window_length (i.e., each window is 1 second long) times in each training step to support a model with sufficient gradient updates. For each sample in a batch, a batch size of 32 and a segment of 1s (i.e., 1 second) are used. Training lasts for approximately 1.5 million steps, or 1,500,000 steps.

[0224] The model used in the experiment is as follows:

[0225] WaveNet was used for experiments involving replication synthesis and text-to-speech.

[0226] PWGAN was used for experiments involving replication synthesis and text-to-speech.

[0227] MelGAN is used for target experiments in replication synthesis and has objective evaluation capabilities.

[0228] WaveGlow is used for the target experiment of replication synthesis.

[0229] Transformer.v3, used for text-to-speech target experiments.

[0230] StyleMelGAN has been objectively and subjectively evaluated against the aforementioned pre-trained baseline vocoder model. The subjective quality of the audio TTS output was assessed via a P.800 listening test performed by listeners in a controlled environment. This test set contained unseen utterances recorded by the same speaker, randomly selected from the LibriVox online corpus. These utterances tested the model's generalization ability because they were recorded under slightly different conditions and exhibited different prosodices. The original utterances were resynthesized using the GriffinLim algorithm, and these utterances were used instead of the usual anchoring conditions. This facilitates the use of an overall rating scale.

[0231] Traditional objective measurement methods such as PESQ and POLQA are unreliable for evaluating speech waveforms generated by neural vocoders. Instead, conditional Fréchet deep speech distance (cFDSD) was used. The cFDSD scores of different neural vocoders shown below demonstrate that StyleMelGAN significantly outperforms the other models.

[0232] MelGAN training cFDSD 0.235, testing cFDSD 0.227

[0233] • PWGAN training cFDSD 0.122 Testing cFDSD 0.101

[0234] WaveGlow training cFDSD 0.099, testing cFDSD 0.078

[0235] WaveNet training cFDSD 0.176, testing cFDSD 0.140

[0236] StyleMelGAN training cFDSD 0.044, testing cFDSD 0.068

[0237] It can be seen that StyleMelGAN outperforms other adversarial and non-adversarial vocoders.

[0238] A panel of 15 expert listeners conducted the MUSHRA hearing test. This type of test was chosen because it can more accurately assess the quality of the generated speech. The anchors were generated using a Py-Torch implementation of the Griffin-Lim algorithm with 32 iterations. Figure 5 The results of the MUSHRA test are shown. It can be seen that StyleMelGAN significantly outperforms other vocoders by approximately 15 MUSHRA points. The results also show that the output quality generated by WaveGlow is comparable to WaveNet and on par with Parallel WaveGAN.

[0239] The subjective quality of the audio TTS output was assessed via a P.800 ACR hearing test performed by 31 listeners in a controlled environment. The ESPNET Transformer.v3 model was used to generate Mel spectrograms of the transcripts for the test set. The same Griffin-Lim anchor could also be added, as this facilitates the use of the overall rating scale.

[0240] The following P800 Mean Opinion Score (MOS) of different TTS systems show similar findings, with StyleMelGAN significantly outperforming other models:

[0241] ·GriffinLim P800 MOS:1.33+ / -0.04

[0242] • Converter + Parallel WaveGAN P800 MOS: 3.19 + / - 0.07

[0243] • Converter + WaveNet P800 MOS: 3.82 + / - 0.07

[0244] • Converter + StyleMelGAN P800 MOS: 4.00 + / - 0.07

[0245] • Recorded P800 MOS: 4.29 + / - 0.06

[0246] The following shows the generation speed and number of parameters of different parallel vocoder models in terms of real-time factor (RTF). StyleMelGAN offers a clear trade-off between generation quality and inference speed.

[0247] This section presents the number of parameters and real-time factors generated on CPUs (e.g., Intel Core i7-6700 3.40GHz) and GPUs (e.g., Nvidia GeForce GTX1060) for the various models studied.

[0248] Parallel WaveGAN parameters: 1.44M CPU: 0.8x GPU: 17x

[0249] MelGAN parameters: 4.26M CPU: 7x GPU: 110x

[0250] StyleMelGAN parameters: 3.86M CPU: 2.6x GPU: 54x

[0251] WaveGlow parameters: 80M - GPU: 5x

[0252] at last, Figure 5The results of the MUSHRA expert hearing test are shown. It can be seen that StyleMelGAN outperforms existing models.

[0253] in conclusion

[0254] This work proposes StyleMelGAN, a lightweight and efficient adversarial vocoder for high-fidelity speech synthesis. Instead of providing conditioning only to the first layer, the model uses Time Adaptive Normalization (TADE) to provide sufficient and accurate conditions to all generation layers. For adversarial training, the generator competes with a filter bank random window discriminator, which provides a multi-scale representation of the speech signal in both the time and frequency domains. StyleMelGAN operates an order of magnitude faster than real-time speeds on CPUs and GPUs. Objective and subjective experimental results demonstrate that StyleMelGAN significantly outperforms previous adversarial vocoders, as well as autoregressive, stream-based, and diffusion-based vocoders, providing a new state-of-the-art baseline for neural waveform generation.

[0255] In summary, the embodiments described herein may optionally be supplemented by any of the points or aspects described herein. However, it should be noted that the points and aspects described herein may be used alone or in combination, and may be incorporated individually and in combination into any of the embodiments described herein.

[0256] Although some aspects are described in the context of apparatus, it is clear that these aspects also represent a description of the corresponding method, wherein the apparatus or a portion thereof corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of method steps also represent a description of the corresponding apparatus or a portion thereof, or an item or feature of the corresponding apparatus. Some or all of the method steps may be performed by (or using) hardware means, such as a microprocessor, a programmable computer, or electronic circuitry. In some embodiments, one or more of the most important method steps may be performed by such means.

[0257] Depending on certain implementation requirements, embodiments of the present invention can be implemented in hardware or software. This implementation can be performed using digital storage media such as floppy disks, DVDs, Blu-ray discs, CDs, ROMs, PROMs, EPROMs, EEPROMs, or flash memory, which have electronically readable control signals stored thereon that cooperate (or are capable of cooperating with) a programmable computer system to perform the corresponding methods. Therefore, the digital storage media can be computer-readable.

[0258] Some embodiments of the invention include a data carrier having electronically readable control signals, which is capable of cooperating with a programmable computer system to perform one of the methods described herein.

[0259] Typically, embodiments of the present invention can be implemented as a computer program product having program code that, when run on a computer, can be used to perform one of the methods. The program code may, for example, be stored on a machine-readable medium.

[0260] Other embodiments include a computer program stored on a machine-readable medium for performing one of the methods described herein.

[0261] In other words, therefore, an embodiment of the method of the present invention is a computer program having program code for executing one of the methods described herein when the computer program is run on a computer.

[0262] Therefore, a further embodiment of the method of the present invention is a data carrier (or digital storage medium, or computer-readable medium) including a computer program recorded thereon for performing one of the methods described herein. The data carrier, digital storage medium, or recording medium is generally tangible and / or non-transitory.

[0263] Therefore, a further embodiment of the method of the present invention represents a data stream or signal sequence for performing one of the methods described herein. The data stream or signal sequence may, for example, be configured to be transmitted via a data communication connection, such as via the Internet.

[0264] Further embodiments include processing means, such as a computer or programmable logic device, configured or adapted to perform one of the methods described herein.

[0265] A further embodiment includes a computer on which a computer program for performing one of the methods described herein is installed.

[0266] Further embodiments of the invention include an apparatus or system configured to transmit (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. For example, the receiver may be a computer, mobile device, memory device, etc. The apparatus or system may, for example, include a file server for transmitting the computer program to the receiver.

[0267] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, the field-programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware device.

[0268] The apparatus described herein may be implemented using hardware devices, computers, or a combination of hardware devices and computers. The apparatus described herein, or any component thereof, may be implemented at least partially in hardware and / or software. The methods described herein may be performed using hardware devices, computers, or a combination of hardware devices and computers. The methods described herein, or any part thereof, may be performed at least partially in hardware and / or software.

[0269] The embodiments described above are merely illustrative of the principles of the invention. It should be understood that modifications and variations in the arrangements and details described herein will be apparent to those skilled in the art. Therefore, its intent is limited only by the scope of the forthcoming patent claims, and not by the specific details provided by the description and interpretation of the embodiments herein.

[0270] References

[0271] A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, et al., “WaveNet: AGenerative Model for Raw Audio,” arXiv:1609.03499, 2016.

[0272] R. Prenger, R. Valle, and B. Catanzaro, “Waveglow: A Flow-based Generative Network for Speech Synthesis,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019, pp. 3617–3621.

[0273] S. Mehri, K. Kumar, I. Gulrajani, R. Kumar, et al., “SampleRNN: An Unconditional End-to-End Neural Audio Generation Model,” arXiv:1612.07837, 2016.

[0274] N.Kalchbrenner, E.Elsen, K.Simonyan, S.Noury,

[0275] et al.,“Efficient neural audio synthesis,”

[0276] arXiv:1802.08435,2018.

[0277] A.van den Oord,Y.Li,I.Babuschkin,K.Simonyan,et al.,“Parallel WaveNet:Fast High-Fidelity Speech Synthesis,”in Proceedings of the 35th ICML,2018,pp.3918–3926.

[0278] J.Valin and J.Skoglund,“LPCNET:Improving Neural Speech Synthesisthrough Linear Prediction,”in IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),2019,pp.5891–5895.

[0279] K.Kumar,R.Kumar,de T.Boissiere,L.Gestin,et al.,“MelGAN:GenerativeAdversarial Networks for Con-ditional Waveform Synthesis,”in Advances inNeurIPS 32,pp.14910–14921.2019.

[0280] R.Yamamoto,E.Song,and J.Kim,“Parallel WaveGAN:A Fast WaveformGeneration Model Based on Genera-tive Adversarial Networks with Multi-Resolution Spec-trogram,”in IEEE International Conference on Acous-tics,Speech and Signal Processing(ICASSP),2020,pp.6199–6203.

[0281] M.Bin′kowski,J.Donahue,S.Dieleman,A.Clark,et al.,“High FidelitySpeech Synthesis with Adversarial Networks,”arXiv:1909.11646,2019.

[0282] T.Park,M.Y.Liu,T.C.Wang,and J.Y.Zhu,“Se-mantic Image Synthesis WithSpatially-Adaptive Nor-malization,”in Proc.of the IEEE Conference on ComputerVision and Pattern Recognition(CVPR),2019.

[0283] P.Govalkar,J.Fischer,F.Zalkow,and C.Dittmar,“A Comparison of RecentNeural Vocoders for Speech Signal Reconstruction,”in Proceedings of the ISCASpeech Synthesis Workshop,2019,pp.7–12.

[0284] I.Goodfellow,J.Pouget-Abadie,M.Mirza,B.Xu,et al.,“GenerativeAdversarial Nets,”in Advances in NeurIPS 27,pp.2672–2680.2014.C.Donahue,J.McAuley,and M.Puckette,“Adversarial Audio Synthesis,”arXiv:1802.04208,2018.

[0285] J.Engel,K.K.Agrawal,S.Chen,I.Gulrajani,et al.,“GANSynth:AdversarialNeural Audio Synthesis,”arXiv:1902.08710,2019.

[0286] G.Yang,S.Yang,K.Liu,P.Fang,et al.,“Multiband MelGAN:Faster WaveformGeneration for High-Quality Text-to-Speech,”arXiv:2005.05106,2020.

[0287] J.Yang,J.Lee,Y.Kim,H.Cho,and I.Kim,“VocGAN:A High-Fidelity Real-timeVocoder with a Hierarchically-nested Adversarial Network,”arXiv:2007.15256,2020.Jungil Kong,Jaehyeon Kim,and Jaekyoung Bae,“Hifi-GAN:Generativeadversarial networks for efficient and high fidelity speech synthesis,”arXivpreprint arXiv:2010.05646,2020.

[0288] D.Ulyanov,A.Vedaldi,and V.Lempitsky,“Instance normalization:Themissing ingredient for fast styliza-tion,”arXiv:1607.08022,2016.

[0289] A.Mustafa,A.Biswas,C.Bergler,J.Schottenhamml,and A.Maier,“Analysis byAdversarial Synthesis-A Novel Approach for Speech Vocoding,”in Proc.Inter-speech,2019,pp.191–195.

[0290] T.Q.Nguyen,“Near-perfect-reconstruction pseudo-QMF banks,”IEEETransactions on Signal Processing,vol.42,no.1,pp.65–76,1994.

[0291] T.Salimans and D.P.Kingma,“Weight normalization:A simplereparameterization to accelerate training of deep neural networks,”inAdvances in NeurIPS,2016,pp.901–909.

[0292] K.Ito and L.Johnson,“The LJ Speech Dataset,” https: / / keithito.com / LJ- Speech-Dataset / ,2017.

[0293] D.P.Kingma and J.Ba,“Adam:A method for stochas-tic optimization,”arXiv:1412.6980,2014.

[0294] T.Hayashi,R.Yamamoto,K.Inoue,T.Yoshimura,et al.,“Espnet-TTS:Unified,reproducible,and inte-gratable open source end-to-end text-to-speechtoolkit,”in IEEE International Conference on Acoustics,Speech and SignalProcessing(ICASSP).IEEE,2020,pp.7654–7658.

[0295] A.Gritsenko,T.Salimans,R.van den Berg,J.Snoek,and N.Kalchbrenner,“ASpectral Energy Distance for Parallel Speech Synthesis,”arXiv:2008.01160,2020.

[0296] “P.800:Methods for subjective determination of trans-missionquality,”Standard,International Telecommuni-cation Union,1996.

Claims

1. An audio generator (10) configured to generate an audio signal (16) from an input signal (14) and target data (12), the target data (12) representing the audio signal (16), the audio generator (10) comprising: The first processing block (40, 50, 50a-50h) is configured to receive first data (15, 59a) derived from the input signal (14) and output first output data (69), wherein the first output data (69) includes multiple channels (47), and The second processing block (45) is configured to receive the first output data (69) or data obtained from the first output data (69) as the second data; The first processing block (50) includes the following for each channel of the first output data: The conditional set of the learnable layers (71, 72, 73) is configured to process the target data (12) to obtain conditional feature parameters (74, 75); and Style element (77) is configured to apply conditional feature parameters (74, 75) to first data (15, 59a) or normalized first data (59, 76'); and The second processing block (45) is configured to combine multiple channels (47) of the second data (69) to obtain an audio signal (16). The first processing block (40, 50, 50a-50k) further includes: Another set of learnable layers (61a, 62a, 61b, 62b) is configured to process the data obtained from the first data (15, 59, 59a, 59b) using a second activation function (63a, 64a, 63b, 64b). The second activation function (63a, 64a, 63b, 64b) is a gated activation function.

2. The audio generator according to claim 1, wherein the condition set of the learnable layer consists of one or at least two convolutional layers (71-73).

3. The audio generator according to claim 2, wherein the first convolutional layer (71-73) is configured to convolve the target data (12) or upsampled target data using a first activation function to obtain the first convolutional data (71').

4. The audio generator of claim 1, wherein the condition set and style element (77) of the learnable layer (71-73) are part of the weight layer in the residual block (50, 50a-50h) of the neural network comprising one or more residual blocks (50, 50a-50h).

5. The audio generator according to claim 1, wherein the audio generator (10) further includes a normalization element (76) configured to normalize the first data (59a, 15).

6. The audio generator according to claim 1, wherein the audio signal (16) is a speech audio signal.

7. The audio generator of claim 1, wherein the target data (12) is upsampled by a factor of at least 2.

8. The audio generator according to claim 7, wherein the target data (12) is upsampled (70) by nonlinear interpolation.

9. The audio generator of claim 1, wherein another set of learnable layers (61a, 62a, 61b, 62b) consists of one, two or more convolutional layers.

10. The audio generator according to claim 1, wherein the second activation function (63a, 63b) is a softmax-gated hyperbolic tangent TanH function.

11. The audio generator according to claim 3, wherein the first activation function is a leakage rectified linear unit function.

12. The audio generator of claim 1, wherein the convolution operations (61a, 61b, 62a, 62b) are operated with a maximum dilation factor of 2.

13. The audio generator according to claim 1, comprising eight first processing blocks (50a-50h) and one second processing block (45).

14. The audio generator of claim 1, wherein the first data (15, 59, 59a, 59b) has a lower dimension than the audio signal.

15. The audio generator according to claim 1, wherein the target data (12) is a spectrogram.

16. The audio generator according to claim 1, wherein the target data (12) is a Mel spectrogram.

17. The audio generator according to claim 1, wherein the target data (12) is a bitstream.

18. The audio generator according to claim 1, wherein the target data (12) is a degraded audio signal.

19. The audio generator according to claim 1, wherein the target data (12) is a compressed representation of the audio data.

20. A method for generating an audio signal (16) from an input signal (14) and target data (12) via an audio generator (10), the target data (12) representing the audio signal (16), the method comprising: The first data (15, 59, 59a, 59b) obtained from the input signal (14) is received through the first processing block (50, 50a-50h). For each channel of the first output data (59b, 69): The target data (12) is processed through the conditional set of the learnable layers (71, 72, 73) of the first processing block (50) to obtain conditional feature parameters (74, 75); and Conditional feature parameters (74, 75) are applied to the first data (15, 59) or the normalized first data (76') through the style element (77) of the first processing block (50). The first processing block (50) outputs first output data (69) including multiple channels (47); The second processing block (45) receives the first output data (69) or the data derived from the first output data (69) as the second data; as well as The second processing block (45) combines multiple channels (47) of the second data to obtain an audio signal (16). The method further includes: The data obtained from the first data (15, 59a) is processed using a second activation function (63a, 64a, 63b, 64b) on another set of learnable layers (61a, 62a, 61b, 62b) through the first processing block (50). The second activation function (63a, 64a, 63b, 64b) is a gated activation function.

21. The method for generating an audio signal from an input signal and target data by means of an audio generator according to claim 20, wherein the condition set of the learnable layer (71-73) consists of one or two convolutional layers.

22. The method for generating an audio signal from an input signal and target data by means of an audio generator according to claim 21, wherein the processing of the condition set of the learnable layers (71-73) includes convolving the target data (12) or upsampled target data with a first activation function through a first convolutional layer (71) to obtain first convolutional data (71').

23. The method of claim 21 for generating an audio signal from an input signal and target data via an audio generator, wherein the condition set and style element (77) of the learnable layer (71-73) are part of a weight layer in the residual block (50, 50a-50h) of a neural network comprising one or more residual blocks (50, 50a-50h).

24. The method for generating an audio signal from an input signal and target data by means of an audio generator according to claim 21, wherein the method further comprises normalizing the first data (15, 59) by means of a normalization element (76).

25. The method for generating an audio signal from an input signal and target data by means of an audio generator according to claim 21, wherein the audio signal (16) is a speech audio signal.

26. The method for generating an audio signal from an input signal and target data by means of an audio generator according to claim 21, wherein the target data (12) is upsampled (70) by a factor of 2.

27. The method for generating an audio signal from an input signal and target data by means of an audio generator according to claim 21, wherein the target data (12) is upsampled (70) by nonlinear interpolation.

28. The method of claim 20 for generating an audio signal from an input signal and target data via an audio generator, wherein another set of learnable layers (61a, 62a, 61b, 62b) consists of one or two convolutional layers.

29. The method for generating an audio signal from an input signal and target data by means of an audio generator according to claim 20, wherein the second activation function (63a, 64a, 63b, 64b) is a softmx-gated hyperbolic tangent TanH function.

30. The method for generating an audio signal from an input signal and target data by means of an audio generator according to claim 20, wherein the first activation function is a leaky rectified linear unit function.

31. The method for generating an audio signal from an input signal and target data by means of an audio generator according to claim 20, wherein the convolution operations (61a, 62a, 61b, 62b) are operated with a maximum dilation factor of 2.

32. The method for generating an audio signal from an input signal and target data by means of an audio generator according to claim 20, comprising performing the first processing block (50, 50a-50h) eight times and performing the second processing block (45) once.

33. The method for generating an audio signal from an input signal and target data by means of an audio generator according to claim 20, wherein the first data (15, 59) has a lower dimension than the audio signal.

34. The method for generating an audio signal from an input signal and target data by means of an audio generator according to claim 20, wherein the target data (12) is a spectrogram or a bitstream.

35. The method of claim 34, wherein the spectrum is a Mel spectrum.

36. The method of claim 20 for generating an audio signal from an input signal and target data via an audio generator, wherein the target data is a compressed representation of audio data, or the target data is a degraded audio signal.

Images