Audio generator, audio signal generation method, and audio generator training method

Abstract

Techniques for generating an audio signal and training an audio generator are disclosed. The audio generator (10) can be generated from an audio signal (16), an input signal (14), and target data (12) representing the audio signal (16). The target data (12) is derived from text. The audio generator includes a first processing block (40, 50, 50a-50h) that receives first data (15, 59a) derived from the input signal (14) and outputs first output data, and a second processing block (45) that receives the first output data (69) or data derived from the first output data (69) as second data. The first processing block (50) comprises a conditioning set (71, 72, 73) of a learnable layer configured to process the target data (12) to obtain conditioning characteristic parameters (74, 75), and a styling element (77) configured to apply the conditioning characteristic parameters (74, 75) to the first data (15, 59a) or a normalized version (59, 76′) of the first data.

Description

[Technical field]

[0001] Introduction Different inventive embodiments and aspects are described below, and further embodiments are specified in the appended claims. All of the embodiments specified by the claims can be supplemented by any of the particulars (features and functions) described in this specification.

[0002] Likewise, all of the embodiments described in this specification can be used independently and can be supplemented by any of the features contained in this specification or in the claims.

[0003] It is also noted that the individual aspects described herein can be used independently or in combination, and thus details can be added to each individual aspect without adding details to another one of the aspects.

[0004] This disclosure also explicitly or implicitly describes features that can be used in an audio generator and / or an audio generation method and / or a computer program product, and thus any of the features described herein can be used in the context of an apparatus, method and / or computer program product.

[0005] Moreover, features and functionality related to the methods disclosed herein can also be used in an apparatus (configured to perform such functionality). Also, any features and functionality disclosed herein in relation to an apparatus can also be used in the corresponding method. In other words, the methods disclosed herein can be supplemented by any of the features and functionality described in relation to the apparatus.

[0006] Similarly, any of the features and functionality described herein can be implemented in hardware and software, or using a combination of hardware and software, as described in the "Implementation Options" section below.

[0007] Implementation Choices Although some aspects are described in the context of an apparatus, it will be apparent that these aspects also represent a description of the corresponding method, whose features correspond to method steps or features of the method steps. Similarly, aspects described in the context of method steps represent a description of the corresponding features of the corresponding apparatus. Some or all of the method steps can be performed by (or using) a hardware apparatus, such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.

[0008] Depending on certain implementation requirements, embodiments of the present invention can be implemented in hardware or software. Implementation can also be performed using a digital storage medium, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM or FLASH memory, that stores electronically readable control signals that cooperate (or can cooperate) with a programmable computer system so that the respective method is executed. Thus, the digital storage medium may be computer readable.

[0009] Some embodiments of the present invention include a data carrier having electronically readable control signals that can cooperate with a programmable computer system to cause one of the methods described herein to be performed.

[0010] Generally, embodiments of the present invention can be implemented as a computer program product comprising program code which is operative to perform one of the methods when the computer program is run on a computer. The program code can for example be stored on a machine readable carrier.

[0011] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0012] Thus, an embodiment of the inventive method is therefore a computer program having a program for performing one of the methods described herein, when the computer program is run on a computer.

[0013] A further embodiment of the inventive method is thus a data carrier (or digital storage medium or computer readable medium) storing and comprising a computer program for performing one of the methods described herein. The data carrier, digital storage medium or recorded medium is typically tangible and non-transitory.

[0014] A further embodiment of the inventive methods is thus a data stream or a sequence of signals representing the computer program for performing one of the methods described herein, the data stream or the sequence of signals being for example adapted to be transmitted over a data communication connection, for example via the Internet.

[0015] A further embodiment comprises a processing means, such as a computer, or a programmable logic device apparatus, configured to or adapted to perform one of the methods described herein.

[0016] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0017] Another embodiment of the present invention includes a device or system configured to transmit (e.g. electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The device or system may, for example, include a file server for transmitting the computer program to the receiver.

[0018] In some embodiments, a programmable logic device (such as a field programmable gate array) may be used to perform some or all of the functionality of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, it is intended that the methods are preferably performed by any hardware apparatus.

[0019] The apparatus described herein may be implemented using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0020] The apparatus described herein, or any components of the apparatus described herein, may be implemented at least in part in hardware and / or software.

[0021] The methods described herein may be performed using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0022] The methods described herein, or any part of the methods described herein, may be at least partially performed by hardware and / or software.

[0023] The above examples merely illustrate the principles of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be obvious to others skilled in the art. It is therefore intended to be limited only by the scope of the appended claims, and not by the specific details represented by the description and explanation of the embodiments set forth herein. [Background technology]

[0024] Technical Field The present invention resides in the technical field of audio generation.

[0025] An embodiment of the invention relates to an audio generator configured to generate an audio signal from an input signal and target data, the target data representing the audio signal. Further embodiments are methods for generating an audio signal and for training an audio generator. Another embodiment is a computer program product.

[0026] background In recent years, neural vocoders have outperformed traditional speech synthesis approaches in terms of the naturalness and perceptual quality of the synthesized speech signal. The best results have been achieved by computationally intensive neural vocoders such as WaveNet and WaveGlow, while lightweight architectures based on generative adversarial networks, such as MelGAN and parallel WaveGAN, still lag behind in terms of perceptual quality.

[0027] Generative models that use deep learning to generate audio waveforms, such as WaveNet, LPCNet, and WaveGlow, have made significant progress in natural-sounding speech synthesis. These generative models, called neural vocoders in text-to-speech (TTS) applications, outperform both parametric and concatenative synthesis methods. To reproduce a given speaker and a given utterance, they can be conditioned using a compressed representation of the target speech (such as a mel spectrogram).

[0028] Previous work has shown that using such generative models at the decoder side, it is possible to achieve very low bitrate speech coding with clean speech by conditioning a neural vocoder with parameters from a traditional low bitrate speech coder.

[0029] Neural vocoders have also been used for speech enhancement tasks, such as denoising or dereverberating speech.

[0030] The main problem with these deep generative models is usually the large number of parameters required and the resulting complexity (interference) in both training and synthesis. For example, WaveNet, which is considered state-of-the-art in terms of synthetic speech quality, generates audio samples one by one sequentially. This process is very slow and computationally demanding and cannot be done in real-time.

[0031] Recently, lightweight adversarial vocoders based on generative adversarial networks (GANs), such as MelGAN and Parallel WaveGAN, have been proposed for rapid waveform generation. However, the perceptual quality of speech generated using these models is reportedly significantly below baseline neural vocoders such as WaveNet and WaveGlow. Text-to-speech GANs (GAN-TTS) have been proposed to fill this quality gap, but the computational cost is still high.

[0032] There are many different neural vocoders available, but they all have their drawbacks: for example, autoregressive vocoders such as WaveNet and LPCNet can be very high quality and suitable for optimizing interference on the CPU, but are not suitable for use on GPUs, as the processing is not easily parallelizable and they cannot offer real-time processing without compromising on quality.

[0033] Normalized flow vocoders such as WaveGlow are also of very high quality and may be suitable for injecting into a GPU, but they contain extremely complex models that take a very long time to train and optimize, making them similarly unsuitable for embedded devices.

[0034] GAN vocoders such as MelGAN and parallel WaveGAN may be suitable for GPU-intensive and lightweight, but suffer from lower quality compared to autoregressive models. Summary of the Invention

[0035] In short, there is still no low-complexity solution that emits high-fidelity audio. The most researched approach to achieve such a goal is GAN. The present invention is an effective solution to this problem. [Problem to be solved by the invention]

[0036] The objective of the present invention is to provide a lightweight neural vocoder solution, e.g. for TTS (Text-to-Speech), that produces very high quality speech and can be trained with limited computational resources.

[0037] BRIEF DESCRIPTION OF THE DRAWINGS An embodiment of the present invention will now be described with reference to the enclosed drawings, in which: [Brief description of the drawings]

[0038] [Figure 1] FIG. 1 shows the architecture of an audio generator according to an embodiment of the present invention. [Diagram 2] FIG. 2 illustrates a classifier that can be used to train an audio generator according to the present invention. [Diagram 3] FIG. 3 shows the structure of a portion of an audio generator according to an embodiment of the present invention. [Figure 4] FIG. 4 shows the structure of a portion of an audio generator according to an embodiment of the present invention. [Diagram 5]Figure 5 shows the results of the MUSHRA expert listening test for the different models. [Figure 6] FIG. 6 illustrates an architecture of an audio generator according to an embodiment of the present invention. [Figure 7] FIG. 7 illustrates the operations performed on a signal according to the present invention. [Figure 8] FIG. 8 illustrates the operation of a text-to-speech application using an audio generator. [Figure 9a] FIG. 9a shows an embodiment of the generator. [Figure 9b] FIG. 9b shows an embodiment of the generator. [Figure 9c] FIG. 9c shows an embodiment of the generator. [Figure 10] FIG. 10 shows some possibilities for inputs and outputs of blocks that can be internal or external to the generator of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0039] In the drawings, like numbers refer to like elements and features.

[0040] According to an aspect, an audio generator is provided that is configured to generate an audio signal from input data and target data, the target data representing the audio signal, the audio generator comprising: a first processing block configured to receive first data derived from the input signal and output first output data, the first output data including a plurality of channels; and a second processing block configured to receive the first output data or data derived from the first output data as second data, the first processing block including, for each channel of the first output data, a conditioning set of learnable layers configured to process the target data derived from text to obtain conditioning feature parameters, and a stylization element configured to apply the conditioning feature parameters to the first data or a normalized version of the first data, and the second processing block is configured to combine the plurality of channels of the second data to obtain the audio signal.

[0041] The audio generator may be configured such that the target data is a spectrogram.The audio generator may be configured such that the target data is a mel spectrogram.

[0042] The audio generator may be configured such that the target data comprises at least one acoustic characteristic of a log spectrogram, MFCC, mel spectrogram or other type of spectrogram derived from text.

[0043] The audio generator may be configured to obtain the target data by converting an input in the form of text or text elements into at least one acoustic characteristic.

[0044] The audio generator may be configured to obtain the target data by transforming at least one linguistic feature into at least one acoustic feature.

[0045] The audio generator may include at least one of the following linguistic features derived from the text: phonemes, prosody, intonation, sentence breaks, and voiced pauses.

[0046] The audio generator may be configured to obtain the target data by converting an input in the form of text or text elements into at least one linguistic characteristic.

[0047] The audio generator may be configured such that the target data includes at least one of a character or a word taken from a text.

[0048] For the audio generator, the target data may be derived from the text using statistical models that perform text analysis and / or use acoustic models.

[0049] For audio generators, the target data may be derived from text using a trainable model that performs text analysis and / or uses acoustic models.

[0050] For audio generators, the target data may be derived from text using rule-based algorithms for text analysis and / or acoustic models.

[0051] The audio generator may be configured to obtain the target data by elaborating an input.

[0052] The audio generator may be configured to derive the target data through at least one deterministic layer.

[0053] The audio generator may be configured to derive the target data through at least one learnable layer.

[0054] The audio generator may have a conditioning set of learnable layers consisting of one or at least two convolutional layers.

[0055] The audio generator may be configured such that a first convolutional layer convolves the target data or an upsampled version of the target data with a first activation function to obtain first convolved data.

[0056] The audio generator may include a neural network in which the conditioning set and stylization elements of a learnable layer are part of a weighting layer within a residual block of the neural network that includes one or more residual blocks.

[0057] The method may comprise deriving at least one acoustic feature from a log spectrogram, a MFCC, a mel spectrogram or another type of spectrogram obtained from the text.

[0058] The method may obtain the target data by converting an input in the form of text or text elements into at least one acoustic feature.

[0059] The method may obtain the target data by transforming at least one linguistic feature into at least one acoustic feature.

[0060] The method may include linguistic features of at least one of phonemes, prosody, intonation, sentence breaks, and voiced pauses obtained from the text.

[0061] The method may obtain the target data by converting input in the form of text or text elements into at least one linguistic characteristic.

[0062] The method may include obtaining at least one of a character or a word from the text.

[0063] The method may derive the target model using a statistical model that performs text analysis and / or uses an acoustic model.

[0064] The method may derive the target data using a learnable model that performs text analysis and / or uses an acoustic model.

[0065] The method may derive the target data using rule-based algorithms for text analysis and / or acoustic models.

[0066] The method may derive the target data through at least one deterministic layer.

[0067] The method may derive the target data through at least one learnable layer.

[0068] The method for generating an audio signal may further comprise the step of deriving target data from the text.

[0069] The method includes the case where the input representative sequence is text.

[0070] The method includes where the input representative sequence is a spectrogram.The method includes where the spectrogram is a mel-spectrogram.

[0071] In particular, an audio generator (e.g. 12) is proposed that is configured to generate an audio signal (e.g. 16) from an input signal (e.g. 14) and target data (e.g. 12), the target data (e.g. 12) being representative of the audio signal (e.g. 16) and derivable from text, the audio generator comprising at least one of a first processing block (e.g. 40, 50, 50a-50h) configured to receive first data (e.g. 15, 59a) derived from the input signal (e.g. 14) and to output first output data (e.g. 69), where the first output data (e.g. 69) comprises a plurality of channels (e.g. 47), and a second processing block (e.g. 45) configured to receive the first output data (e.g. 69) or data derived from the first output data (e.g. 69) as second data.

[0072] The first processing block (e.g., 50) comprises, for each channel of the first output data, a conditioning set of learnable layers (e.g., 71, 72, 73) for processing the target data (e.g., 12) to obtain conditioning characteristic parameters (e.g., 74, 75), and a styling element (e.g., 77) configured to apply the conditioning characteristic parameters (e.g., 74, 75) to the first data (e.g., 15, 59a) or a normalized version of the first data (e.g., 59, 76').

[0073] A second processing block (eg 45) may be configured to combine multiple channels (eg 47) of the second data (eg 69) to obtain an audio signal (eg 16).

[0074] Also proposed is a method for generating an audio signal (e.g., 16) from an input signal (e.g., 14) and target data (e.g., 12), for example by an audio generator (e.g., 10). The target data (e.g., obtained from text) represents the audio signal (e.g., 16), and the method includes the steps of receiving, by a first processing block (e.g., 50, 50a-50h), first data (e.g., 16559, 59a, 59b) derived from the input signal (e.g., 14), processing the target data (e.g., 12), which may be derived from text, for each channel of the first output data (e.g., 59b, 69), by a conditioning set (e.g., 71, 72, 73) of a learnable layer of the first processing block (e.g., 50) to obtain conditioning feature parameters (e.g., 74, 75), and processing the style imparting elements (e.g., 50) of the first processing block (e.g., 50) into a training set (e.g., 50, 50a-50h) of the ... The method includes applying conditioning characteristic parameters (e.g., 74, 75) to the first data (e.g., 15, 59) or a normalized version of the first data (e.g., 76') by (77); outputting first output data (e.g., 69) including a plurality of channels (e.g., 47) by a first processing block (e.g., 50); receiving the first output data (e.g., 69) or data derived from the first output data (e.g., 69) as second data by a second processing block (e.g., 45); and combining the plurality of channels (e.g., 47) of the second data to obtain an audio signal (e.g., 16) by the second processing block (e.g., 45).

[0075] A method has also been proposed for training a neural network for audio generation, where the neural network is configured to output, at a given time step, audio samples from an input sequence (e.g., 12) representing an audio signal (e.g., 16) to be generated, and to shape a noise vector (e.g., 14) using the input representative sequence (e.g., 12) to produce the output audio sample (e.g., 16), where the training is designed to optimize a loss function (e.g., 140).

[0076] Also proposed is a method for generating an audio signal (e.g., 16) that includes a mathematical model, where the mathematical model is configured to output, at a given time step, audio samples from an input sequence (e.g., 12) that represents the audio data (e.g., 16) to be generated. The mathematical model may shape a noise vector (e.g., 14) using the input representative sequence (e.g., 12) to produce the output audio samples.

[0077] In this context, we propose StyleMelGAN (e.g., audio generator 10), a lightweight neural vocoder capable of synthesizing high-fidelity speech with low computational complexity. StyleMelGAN is a fully convolutional forward-propagation model that uses time-adaptive denormalization (TADE) (e.g., 60a and 60b in Fig. 4 and 60 in Fig. 3) to stylize (e.g., at 77) small-sized noise vectors (e.g., 128 × 1 vectors) via acoustic properties of the target speech waveform. This architecture allows highly parallelizable generation, sometimes faster than real-time on both central computing units (CPUs) and graphics processing units (GPUs). For efficient and rapid training, a multi-scale spectral reconstruction loss can be used in conjunction with an adversarial loss computed with multiple classifiers (e.g., 132a-132d) that evaluate the speech signal 16 by random windowing (e.g., windows 150a, 150b, 105c, 105d) in multiple frequency bands. According to the MUSHRA listening test and the P.800 listening test, StyleMelGAN (e.g., audio generator 10) outperforms known existing neural vocoders in both copy synthesis and TTS scenarios.

[0078] This application proposes, inter alia, a neural vocoder for generating high-quality speech 16, which may be based on a generative adversarial network (GAN). This solution, referred to here as StyleMelGAN (and implemented, for example, in the audio generator 10), is a lightweight vocoder that allows for the synthesis of high-quality speech 16 with low computational complexity. StyleMelGAN is a forward-propagated fully convolutional model that uses time-adaptive denormalization (TADE) to style (e.g., in block 77) a latent noise representation (e.g., 69) using, for example, a mel spectrogram (12) of the target speech waveform. This allows for highly parallelizable generation, sometimes faster than real-time on both CPU and GPU. For training, a multiscale spectral reconstruction loss can be used, followed by an adversarial loss. This results in a model that can synthesize high-quality output with less than two days of training on one GPU.

[0079] Possible applications and advantages of the present invention include:

[0080] The present invention can be applied to text-to-speech and the resulting quality, i.e., for TTS and copy synthesis of generated voice, is close to WaveNet and natural voice. In addition, it provides fast learning and the model can be easily retrained and personalized. It uses less memory due to the relatively small neural network model. Finally, the present invention has a complexity advantage, i.e., it has a good trade-off between quality and complexity.

[0081] The invention can also be used for speech enhancement, providing a low-complexity, robust solution for generating clean speech from noisy speech.

[0082] The invention can also be applied to speech coding, where the bitrate can be significantly reduced by transmitting only the parameters required to condition the neural vocoder, and in this application, lightweight neural vocoder-based solutions are well suited for embedded systems, especially future (end) user equipment (UE) equipped with GPUs or neural processing units (NPUs).

[0083] An embodiment of the present application relates to an audio generator configured to generate an audio signal from an input signal and target data, the target data representing an input signal (e.g. derived from text), the first processing block configured to receive first data derived from the input signal and output first output data, the first output data having a plurality of channels, and a second processing block configured to receive the first output data or data derived from the first output data as second data, the first processing block including, for each channel of the first output data, a conditioning set of learnable layers configured to process the target data to obtain conditioning feature parameters and a stylization element configured to apply the conditioning feature parameters to the first data or a normalized version of the first data, and the second processing block configured to combine the plurality of channels of the second data to obtain an audio signal.

[0084] According to one embodiment, the conditioning set of learnable layers consists of one or two convolutional layers.

[0085] According to one embodiment, the first convolutional layer convolves the target data or an upsampled version of the target data using a first activation function to obtain first convolved data.

[0086] According to one embodiment, the conditioning sets and styling elements of a learnable layer are part of a weighting layer within a residual block of a neural network that includes one or more residual blocks.

[0087] According to one embodiment, the audio generator further comprises a normalization element configured to normalize the first data, for example, the normalization element may normalize the first data to a standard normal distribution with zero mean and unit variance.

[0088] According to one embodiment, the audio signal is a voice audio signal.

[0089] According to one embodiment, the target data is upsampled, preferably by non-linear interpolation, by a factor of 2 or a multiple or power of 2. In some implementations, a factor greater than 2 may be used instead.

[0090] According to one embodiment, the first processing block further includes another set of learnable layers configured to process data derived from the first data using a second activation function, where the second activation function is a gated activation function.

[0091] According to one embodiment, another set of learnable layers consists of one or two (or more) convolutional layers.

[0092] According to one embodiment, the second activation function is a softmax-gated hyperbolic tangent (TanH) function.

[0093] According to one embodiment, the first activation function is a leaky rectified linear unit (leaky ReLu) function.

[0094] According to one embodiment, the convolution operation is performed with a maximum expansion factor of two.

[0095] According to one embodiment, the audio generator comprises eight first processing blocks and one second processing block.

[0096] According to one embodiment, the first data has a lower dimensionality than the audio signal. The first data may be one-dimensional or at least one dimension lower than the audio signal. The first data may have one dimension lower than the audio signal but a greater number of channels than the audio signal. The first data may have a smaller total number of samples across all dimensions than the audio signal.

[0097] According to one embodiment, the target data may be a spectrogram, preferably a mel-spectrogram, or a bitstream.

[0098] The target data may be derived from text. The audio generator may be configured to derive the target data from text. The target data may include, for example, at least one of text data (characters, words, etc.), linguistic features, acoustic features, etc.

[0099] In another embodiment, the target data may be a compressed representation of the audio data, or the target data is a degraded audio signal.

[0100] Another embodiment is a method of generating an audio signal by an audio generator from an input signal and target data, the target data representing an audio signal (e.g., from derived text), comprising: receiving, by a first processing block, first data derived from the input signal; for each channel of the first output data, processing the target data by a conditioning set of a learnable layer of the first processing block to obtain conditioning feature parameters; applying, by a style imparting element of the first processing block, the conditioning feature parameters to the first data or a normalized version of the first data; outputting, by the first processing block, the first output data including a plurality of channels; receiving, by a second processing block, the first output data or data derived from the first output data as second data; and combining, by the second processing block, the plurality of channels of the second data to obtain the audio signal. In some implementations, the method may derive the target data from text.

[0101] Normalization may include, for example, first normalizing the data to a standard normal distribution with zero mean and unit variance.

[0102] The method may be supplemented with any characteristic or combination of characteristics from an audio generator.

[0103] Another embodiment relates to a method for training an audio generator as described above, the training step comprising repeating any one of the steps of the above method one or more times.

[0104] According to one embodiment, the method for training comprises the steps of evaluating the generated audio signal by at least one evaluator, preferably a neural network, and adapting the weights of the evaluator according to the evaluation results.

[0105] According to one embodiment, the method for training comprises a step of adapting the weights of the evaluator according to the evaluation results.

[0106] According to one embodiment, the training step includes optimizing a loss function.

[0107] According to one embodiment, optimizing the loss function comprises calculating a fixed metric between the generated audio signal and a reference audio signal.

[0108] According to one embodiment, calculating the fixed metric comprises calculating one or more spectral distortions of the generated audio signal and the reference audio signal.

[0109] According to one embodiment, the step of calculating one or more spectral distortions is applied for magnitude or log-magnitude and / or for different time or frequency resolutions of the spectral representations of the generated audio signal and the reference signal.

[0110] According to one embodiment, optimizing the loss function includes deriving one or more adversarial metrics by randomly feeding and evaluating representations of the generated audio signals or representations of reference audio signals by one or more evaluators, where the evaluation includes classifying the fed audio signals into a pre-defined number of classes indicating a classification level of naturalness of pre-training of the audio signals.

[0111] According to one embodiment, optimizing the loss function includes computing a fixed metric and deriving an adversarial metric by one or more evaluators.

[0112] According to one embodiment, the audio generator is first trained using a fixed metric.

[0113] According to one embodiment, four adversarial metrics are derived by four evaluators.

[0114] According to one embodiment, the estimator operates after decomposing the representation of the generated audio signal and the representation of the reference audio signal with a filter bank.

[0115] According to one embodiment, each evaluator receives as input one or more parts of a representation of the generated audio signal or a representation of the reference audio signal.

[0116] According to one embodiment, the signal portions are generated by sampling random windows from the input signal using a random window function.

[0117] According to one embodiment, the random window sampling is repeated multiple times for each estimator.

[0118] According to one embodiment, the number of times the random window is sampled for each estimator is proportional to the length of the generated audio signal representation or the reference audio signal representation.

[0119] Another embodiment relates to a computer program product comprising a program for a processing device, the program comprising software code portions for performing the steps of the methods described herein when the program is executed on a processing device.

[0120] According to one embodiment, the computer program product comprises a computer readable medium having stored thereon the software code portions, the program being capable of being directly loaded into the internal memory of a processing device.

[0121] Another embodiment relates to a method for generating an audio signal comprising a mathematical model configured to output audio samples at a given time step from an input sequence (e.g. derived from text) representative of audio data to be generated, the mathematical model configured to shape a noise vector to produce the output audio samples using the input representative sequence.

[0122] According to one embodiment, the mathematical model is trained using audio data. According to one embodiment, the mathematical model is a neural network. According to one embodiment, the network is a feedforward network. According to one embodiment, the network is a convolutional network.

[0123] According to one embodiment, the noise vector may have a lower dimensionality than the audio signal to be generated. The first data may be one dimensional or at least one dimensional lower than the audio signal. The first data may have a lower number of samples across all dimensions than the audio signal. The first data may be one dimensional lower than the audio signal but have a higher number of channels than the audio signal.

[0124] According to one embodiment, a Time-Adaptive Denormalization (TADE) method is used to condition the mathematical model with the input representative sequence and thus shape the noise vector.

[0125] According to one embodiment, a modified softmax-gated Tanh activates each layer of the neural network.

[0126] According to one embodiment, the convolution operation is performed with a maximum dilation factor of 2.

[0127] According to one embodiment, the noise vector as well as the input representative sequence are upsampled to obtain the output audio at the target sampling rate.

[0128] According to one embodiment, the upsampling is performed sequentially at different layers of the mathematical model.

[0129] According to one embodiment, the upsampling factor for each layer is a multiple of 2, such as 2 or a power of 2. In some implementations, the value of the upsampling factor may generally be greater than 2.

[0130] The generated audio signal can generally be used in text-to-speech applications, where the input representative sequence is derived from the text.

[0131] According to one embodiment, the generated audio signal is used in an audio decoder, where the input representative sequence is a compressed representation of the original audio for transmission or storage.

[0132] According to one embodiment, the generated audio signal is used to enhance the audio quality of a degraded audio signal, and the input representative sequence is derived from the degraded signal.

[0133] Another embodiment relates to a method of training a neural network for audio generation, where the neural network outputs audio samples at given time steps from an input sequence representative of audio data to be generated, and the neural network is configured to shape a noise vector to produce the output audio samples using the input representative sequence, the neural network being designed as above, and the training being designed to optimize a loss function.

[0134] According to one embodiment, the loss function comprises a fixed metric calculated between the generated audio signal and a reference audio signal.

[0135] According to one embodiment, the fixed metric is one or more spectral distortions calculated between the generated audio signal and a reference signal.

[0136] According to one embodiment, the one or more spectral distortions are calculated relative to the magnitude or log magnitude of the spectral representation of the generated audio signal and the reference signal.

[0137] According to one embodiment, one or more of the spectral distortions forming the fixed metric are calculated for different time or frequency resolutions.

[0138] According to one embodiment, the loss function comprises an adversarial metric derived from an additional discriminative neural network, the discriminative neural network receiving as input a representation of the generated audio signal or a reference audio signal, the discriminative neural network being configured to evaluate how realistic the generated audio samples are.

[0139] According to one embodiment, the loss function includes both a fixed metric and an adversarial metric derived from an additional discriminative neural network.

[0140] According to one embodiment, a neural network that generates audio samples is first trained using only a fixed metric.

[0141] According to one embodiment, the adversarial metric is derived by four discriminative neural networks.

[0142] According to one embodiment, the discriminative neural network operates after decomposition of the input audio signal by a filter bank.

[0143] According to one embodiment, each discriminative neural network receives as input one or more randomly windowed versions of an input signal.

[0144] According to one embodiment, the random window sampling is repeated multiple times for each discriminative neural network.

[0145] According to one embodiment, the number of times the random window is sampled for each discriminative neural network is proportional to the length of the input audio sample.

[0146] Detailed Description of the Embodiments Fig. 8 shows an embodiment of an audio generator 10. The audio generator 10 can convert text 112 into an output audio signal 16. The text 112 can be converted into target data 12, which in some embodiments can also be understood as an audio representation (e.g. a spectrogram, or more generally MFCCs such as spectrograms, log spectrograms, or spectrograms, MFCCs or mel spectrograms, or other acoustic features) (see below). The target data 12 can be used to condition an input signal 14 (e.g. noise) and process the input signal 14 to make it audible. The audio synthesis block (text analysis block) 1110 can convert the text 112 into an audio representation (e.g. a spectrogram, or more generally MFCCs such as spectrograms, log spectrograms, or spectrograms, MFCCs or mel spectrograms, or other acoustic features) for example under conditions set by the target data 12. The audio synthesis block 1110 may be responsible for at least one of the following processing, for example: speech voicing, phase matching, intonation, duration, etc. The audio synthesis block 1110 (text analysis block) may perform at least one task, for example: text normalization, word segmentation, prosody prediction, and grapheme string to phoneme string conversion, etc. The generated target data 12 is then input to a waveform synthesis block 1120 (such as a vocoder) that may generate, for example, a waveform 16 (output audio signal) from an input signal 14 based on conditions obtained from the target data 12 obtained from the text 112.

[0147] However, it is noted that block 1110 may in some embodiments not be part of generator 10, and block 1110 may be external to generator 10. In some embodiments, block 1110 may be sub-divided into multiple sub-blocks (and in some particular cases, at least one of the sub-blocks may be part of generator 10 and at least one of the sub-blocks may be external to generator 10).

[0148] Generally, the input, which is text (or other input derived from text) input to block 1110 (or to generator 10 in some embodiments), is Text 112 (for example, ASCII code) At least one linguistic feature (e.g., obtained from the text, e.g., at least one of phonemes, prosody, intonation, sentence breaks, and voiced pauses); At least one acoustic feature (e.g. obtained from the text, e.g. at least one of log-spectrogram, MFCC, mel-spectrogram) may be in at least one of the following forms:

[0149] The input may be processed (for example by block 1110) to obtain target data 12. According to a different embodiment, block 1110 may: At least one of the letters or words in the text At least one linguistic feature (e.g., obtained from the text, e.g., at least one of phonemes, prosody, intonation, sentence breaks, and voiced pauses); At least one acoustic feature (e.g. obtained from the text, e.g. at least one of log-spectrogram, MFCC, mel-spectrogram) Processing may be performed to obtain a (text-derived) target 12 in at least one of the following forms:

[0150] The target data 12 (whether in the form of characters, linguistic characteristics or acoustic characteristics) is to be used by a generator 10 (e.g. a waveform synthesis block, vocoder, 1120) to condition the processing of an input signal 14 and thus generate an output audio signal (acoustic waves).

[0151] FIG. 10 shows a summary table of some possibilities for instantiating block 1110. A) In case A, the input entered into block 1110 is plain text 112 and the output (target data 12) from block 1110 is at least one of the characters of the text or words (which are also text). In case A, block 1110 performs a selection of the text 112 into the form of elements of the text 112. The target data 12 (in the form of elements of the text 112) then conditions processing on the input signal 14 to obtain an output signal 16 (acoustic waves).

[0152] B) In case B, the input entered into block 1110 is plain text 112, and the output from block 1110 (target data 12) includes at least one linguistic feature, such as linguistic features of phonemes, prosody, intonation, sentence breaks, voiced pauses, etc., obtained from text 112. In case B, block 1110 applies linguistic analysis to the elements of text 112, thus obtaining at least one linguistic feature from at least one of phonemes, prosody, intonation, sentence breaks, voiced pauses, etc. Then, the target data 12 (in the form of at least one of phonemes, prosody, intonation, sentence breaks, voiced pauses, etc.) is conditioned for processing on the input signal 14 to obtain an output signal 16 (acoustic waves).

[0153] C) In case C, the input to block 1110 is plain text 112 and the output from block 1110 (target data 12) includes at least one acoustic feature, e.g., one of log spectrogram, MFCC, mel spectrogram obtained from the text. In case C, block 1110 performs acoustic analysis on the elements of text 112, thus obtaining at least one acoustic feature, e.g., one of log spectrogram, MFCC, or mel spectrogram obtained from the text 112. The target data 12 (e.g., in the form of at least one of phonemes, prosody, intonation, sentence breaks, and voiced pauses) is then conditioned to processing the input signal 14 to obtain an output signal 16 (acoustic waves).

[0154] D) In ​​case D, the inputs entered into block 1110 are linguistic features (e.g., at least one of phonemes, prosody, intonation, sentence breaks, and voiced pauses) and the output is also processed linguistic features (e.g., at least one of phonemes, prosody, intonation, sentence breaks, and voiced pauses). Target data 12 (in the form of at least one of phonemes, prosody, intonation, sentence breaks, and voiced pauses) then conditions processing on the input signal 14 to obtain an output signal 16 (acoustic wave).

[0155] E) In case E, the input to block 1110 is linguistic features (e.g., at least one of phonemes, prosody, intonation, sentence breaks, and voiced pauses) and the output from block 1110 (target data 12) includes at least one acoustic feature, e.g., one of log spectrograms or MFCCs, and mel spectrograms obtained from the text. In case E, block 1110 applies acoustic analysis to elements of text 112 to obtain at least one acoustic feature, e.g., log spectrograms or MFCCs, and mel spectrograms. The target data 12 (e.g., in the form of at least one acoustic feature, e.g., log spectrograms, MFCCs, mel spectrograms obtained from the text, etc.) then conditions processing on the input signal 14 to obtain an output signal 16 (acoustic wave).

[0156] F) In case F, the input to block 1110 is in the form of acoustic features (e.g. at least one of log spectrograms, MFCCs, mel spectrograms derived from text, etc.) and the output (target data 12) is in the form of processed acoustic features (e.g. at least one of log spectrograms, MFCCs, mel spectrograms derived from text, etc.). The target data 12 (in the form of processed acoustic features, e.g. at least one of log spectrograms, MFCCs, mel spectrograms derived from text, etc.) then conditions the processing of the input signal 14 to obtain an output signal 16 (acoustic wave).

[0157] Fig. 9a shows an embodiment in which block 1110 includes sub-block 1112 (text analysis block) and downstream sub-block 1114 (e.g. audio synthesis using an acoustic model). Sub-block 1112 provides intermediate target data 212 and sub-block 1114 generates target data 12 in the form of acoustic features. Thus, in Fig. 9a, if sub-block 1112 and sub-block 1114 are both part of the generator 10, this is case C. If sub-block 1112 is not part of the generator 10 but sub-block 1114 is part of the generator 10, this is case E.

[0158] Figure 9b shows an embodiment where block 1110 only performs text analysis to yield target data 12 in the form of linguistic features. Thus, in Figure 9b, if block 1110 is part of the generator 10, then it is case B.

[0159] FIG. 9c illustrates an embodiment in which block 1110 is absent and the target data 12 is in the form of a language property.

[0160] Generally, block 1110 operates to refine the text (or other input derived from the text), if present, in the process toward target data that is more refined than the input entered into block 1110. Block 1110 may also use constraints not present in the source text (attention features, male / female voice, accent, emotional trait assessment, etc.) These constraints are generally supplied by the user.

[0161] It is also noted that in the above and below cases, block 1110 (or any sub-blocks, if present, such as either blocks 1112 or 1114) may use a statistical model, for example performing text analysis and / or using an acoustic model. Additionally or alternatively, block 1110 (or any sub-blocks, if present, such as either blocks 1112 or 1114) may use a trainable model, for example performing text analysis and / or using an acoustic model. The trainable model may be based on, for example, a neural network, a Markov chain, etc. Additionally or alternatively, block 1110 (or any sub-blocks, if present, such as either blocks 1112 or 1114) may use a rule-based algorithm, for example performing text analysis and / or based on an acoustic model.

[0162] Block 1110 (or any sub-blocks, if present, such as either block 1112 or block 1114) may, in some implementations, deterministically derive the target data. Thus, some sub-blocks may be trainable and others may be deterministic.

[0163] Block 1110 may also be called the "text analysis block" (e.g., when converting text into at least one linguistic feature) or the "audio synthesis block" (e.g., when converting text or at least one linguistic feature into at least one acoustic feature, e.g., a spectrogram). In any case, it remains that the target data is in the form of text, linguistic feature or acoustic feature, depending on the embodiment.

[0164] In particular, Figure 10 shows that some combinations of transformations are not common because transformations from sophisticated features to simple features (e.g., linguistic features to text, or acoustic features to text or linguistic features) are not anticipated.

[0165] Fig. 6 shows an example of an audio generator 10 that can generate (e.g., synthesize) an audio signal (output signal) 16, for example by StyleMelGAN. In Fig. 6, text 112 can be processed, for example in a text analysis block 1110, to obtain target data 12. Then, in a waveform synthesis block 1120, an input signal 14 (e.g., noise) can be processed using the target data 12 to obtain an audible audio signal 16 (acoustic waveform). The obtained target data 12 can be derived from the text.

[0166] In particular, the output audio signal 16 is generated based on an input signal 14 (also called a latent signal, which may be noise such as white noise) and target data 12 (also called an "input sequence", which in some embodiments is derived from text) and may be obtained, for example, from text 112 in block 1110. The target data 12 may, for example, comprise (e.g. be) a spectrogram (e.g. a mel spectrogram), which provides, for example, a mapping of a sequence of time samples to a mel scale. Additionally or alternatively, the target data 12 may comprise (e.g. be) a bitstream. For example, the target data may be text to be reproduced in audio (e.g. text-to-speech) or may include (or, more generally, be derived from) text. The target data 12 is generally processed to obtain speech sounds that a human listener may perceive as natural. The input signal 14 may be noisy (and thus does not carry any useful information), such as white noise, but in the generator 10, a noise vector obtained from the noise is styled (e.g., at 77) to obtain a noise vector with acoustic characteristics conditioned by the target data 12. Ultimately, a human listener will understand the output audio signal 16 as speech. As in FIG. 1, the noise vector 14 may be a 128×1 vector (a single sample, e.g., a time-domain sample or a frequency-domain sample, and 128 channels). In other embodiments, a different length of the noise vector 14 may be used.

[0167] A first processing block 50 is shown in FIG. 6. As illustrated (e.g. in FIG. 1), the first processing block 50 may be instantiated by each of a number of blocks (in FIG. 1, blocks 50a, 50b, 50c, 50d, 50e, 50f, 50g, 50h). The blocks 50a-50h may be understood to form one single block 40. In the first processing block 40, 50, it is shown that the target data 12 and / or the input signal 14 may be processed using a conditioning set of learnable layers (e.g. 71, 72, 73). Thus, conditioning characteristic parameters 74, 75 (also called gamma (γ) and beta (β) in FIG. 3) are obtained, for example, by convolution during training. The learnable layers 71-73 may thus be part of a training network, or more generally, weight layers of another training structure. The first training block 40, 50 may include at least one stylizing element 77. The at least one stylizing element 77 may output first output data 69. The at least one stylizing element 77 may apply conditioning characteristic parameters 74, 75 to the input signal 14 (latent) or to first data 15 obtained from the input signal 14.

[0168] The first output data of each block 50 is in multiple channels. The audio generator 10 may include a second processing block 45 (shown in FIG. 1 as including blocks 42, 44, 46). The second processing block 45 may be configured to combine multiple channels 47 of the first output data 69 (input as second input data or second data) to obtain an output audio signal 16 in a single channel, but in a sequence of samples.

[0169] The term "channel" should be understood in the context of neural networks (e.g., convolutional neural networks) and not in the context of stereo sound. For example, the input signal (e.g., latent noise) 14 is provided with a sequence of channels (in a time-domain representation), so that there may be 128 channels. For example, if the signal has 176 samples and 64 channels, it can be understood as a matrix with 176 columns and 64 rows, and if the signal has 352 samples and 64 channels, it can be understood as a matrix with 352 columns and 64 rows (other diagrams are possible). Thus, the generated audio signal 16 (which in FIG. 1 is a 1×22528 matrix, where 22528 can be replaced by another number) can be understood as a mono signal. To generate a stereo signal, one simply repeats the disclosed technique for each stereo channel to obtain multiple audio signals 16, which are then mixed together.

[0170] At least the original input signal 14 and / or the generated speech 16 may be vectors. Conversely, the output of each of the blocks 30 and 50a-50h, 42, 44 generally differs in dimension. The first data is one-dimensional or at least one dimension lower than the dimension of the audio signal. The total number of samples across all dimensions of the first data may be less than the audio signal. The first data may have one dimension lower than the audio signal, but the number of channels may be greater than the audio channels. In each of the blocks 30 and 50a-50h, the signal evolving from noise 14 to speech 16 may be upsampled. For example, in the upsampling block 30 before the first block 50a of the blocks 50a-50h, 88 upsamplings are performed. Examples of upsampling may include procedures such as 1) repeating the same value, 2) inserting zeros, and 3) repeating again or inserting zeros and linear filtering.

[0171] The generated audio signal 16 may generally be a single channel signal (e.g., 1×22528). If multiple audio channels are required (e.g., for stereo sound reproduction), the procedures described herein should in principle be repeated multiple times.

[0172] Likewise, the target data 12 may in principle be single channel (if it is text or, more generally, derived from text, such as in case A or in FIG. 9c) or multi-channel (e.g. derived from text, e.g. a spectrogram, e.g. a mel-spectrogram, such as in cases C, E, F). In any case, it may be upsampled (e.g. by 2, a power of 2, a multiple of 2 or a value greater than 2) to match the dimension of the signal (59a, 15, 69) progressing along the subsequent layers (50a-50h, 42) to obtain, for example, conditioning characteristic parameters 74, 75 with dimensions matched to the dimension of the signal.

[0173] When the first processing block is instantiated in, for example, at least a plurality of blocks 50a-50h, the number of channels may remain the same for, for example, a plurality of blocks 50a-50h. The first data may be one-dimensional or one dimension lower than the dimension of the audio signal. The total number of samples across all dimensions of the first data may be lower than the audio signal. The dimension of the first data may be one dimension lower than the audio signal, but the number of channels may be higher than the audio signal.

[0174] The signals in the following blocks may have different dimensions. For example, the number of samples may be upsampled more times, for example from 88 samples to 22528 samples in the last block 50h. Similarly, the target data 12 is also upsampled in each processing block 50. The conditioning characteristic parameters 74, 75 can thus be adapted to the number of samples of the signal to be processed. Thus, the semantic information provided by the target data 12 is not lost in the following layers 50a-50h.

[0175] It is understood that the embodiment may also be implemented in the paradigm of Generative Adversarial Networks (GAN). GAN comprises a GAN generator 11 (FIG. 1) and a GAN classifier 100 (FIG. 2), which may also be understood as a part of the waveform synthesis block 1120. The GAN generator 11 tries to generate an audio signal 16 that is as close as possible to the real signal. The GAN classifier 100 is to distinguish whether the generated audio signal is real (like the real audio signal 104 in FIG. 2) or fake (like the generated audio signal 16). Both the GAN generator 11 and the GAN classifier 100 may be obtained as neural networks. The GAN generator 11 minimizes the loss (e.g., by gradient method or other methods) and updates the conditioning characteristic parameters 74, 75 by considering the result in the GAN classifier 100. The GAN classifier 100 reduces its own classification loss (e.g., by gradient method or other methods) and updates its internal parameters. Thus, the GAN generator 11 is trained to provide a better audio signal 16, while the GAN classifier 100 is trained to distinguish the real signal 16 from the fake audio signal generated by the GAN generator 11. In general, it can be understood that the GAN generator 11 may include the functions of the generator 10, at least excluding the functions of the GAN classifier 100. Thus, in most of the above, the GAN generator 11 and the audio generator 10 may have more or less the same characteristics, except for the characteristics of the GAN classifier 100. The audio generator 10 may include the classifier 100 as a built-in component. Thus, the GAN generator 11 and the GAN classifier 100 may coincide in constructing the audio generator 10. In embodiments without the GAN classifier 100, the audio generator 10 may be composed of only the GAN generator 11.

[0176] The audio generator 10 may be obtained according to the conditional GAN ​​paradigm, for example based on conditional information, as also explained by the term "conditioning set of trainable layers". For example, the conditional information may consist of target data (or an upsampled version of the target data) 12 on which a conditioning set of layers 71-73 (weighting layers) are trained and from which conditioning feature parameters 74, 75 are obtained. Thus, the stylized elements 77 are conditioned by the trainable layers 71-73.

[0177] An embodiment may be based on a convolutional neural network. A small matrix (such as a filter or kernel), for example a 3x3 matrix (or a 4x4 matrix, etc.), is convolved (convolved) along with a larger matrix (channels x sample latents or input signals and / or spectrograms and / or spectrograms or upsampled spectrograms or more generally target data 12) with a combination (multiplication and sum of products, dot product, etc.) of the filter (kernel) elements with the elements of the larger matrix (activation map or activation signal). During training, the filter (kernel) elements that minimize the loss are obtained (trained). During interference, the filter (kernel) elements obtained during training are used. Examples of convolutions are given in blocks 71-73, 61a, 61b, 62a, 62b (see below). If a block is conditional (e.g. block 60 in FIG. 3), the convolution is not necessarily applied to signals evolving from input signal 14 via intermediate signals 59a(15), 69 towards audio signal 16, etc., but may be applied to target signal 14. In other cases (e.g. blocks 61a, 61b, 62a, 62b), the convolution may be unconditional, e.g. applied directly to signals 59a(15), 69 etc. evolving from input signal 14 towards audio signal 16. As can be seen from FIGS. 3 and 4, both conditional and unconditional convolutions may be performed.

[0178] In some embodiments, downstream of the convolution there may be activation functions (ReLu, TanH, softmax, etc.) that differ in their intended effect. ReLu may map a maximum between 0 and the value obtained in the convolution (in effect, keeping the same value for positive values ​​and outputting 0 for negative values). Leaky ReLu outputs x if x>0 and 0.1×x if x≦0, where x is the value obtained by the convolution (in some embodiments, another value may be used instead of 0.1, such as a default value in the range 0.1±0.05). TanH (which may be implemented, for example, in blocks 63a and / or 63b) may be implemented, for example, in blocks 63b and / or 63c. TanH(x) = (e x -e -x ) / (e x +e -x ) The hyperbolic tangent of the convolution value may be given by x = 1 , where x is the convolution value (e.g. in blocks 61a and / or 61b). The softmax (e.g. applied in blocks 64a and / or 64b) normalizes by applying an exponent to each element of the convolution result (e.g. provided in blocks 62a and / or 62b) and dividing by the sum of the exponents. The softmax (e.g. in 64a and / or 64b) may provide a probability distribution of the entries in the matrix resulting from the convolution (e.g. provided in 62a and / or 62b). After application of the activation function, a pooling step may be performed in some embodiments (not shown in the figures), but may be avoided in other embodiments.

[0179] FIG. 4 shows that it is also possible to obtain a softmax-gated TanH function, for example by multiplying (e.g. in 65a and / or 65b) the result of the TanH function (e.g. obtained in 63a and / or 63b) with the result of a softmax function (e.g. obtained in 64a and / or 64b).

[0180] Multiple layers of convolutions (e.g., the conditioning set of learnable layers) may be sequential and / or parallel to each other for efficiency. Application of activation functions and / or pooling may be repeated in different layers (or, for example, different activation functions may be applied to different layers).

[0181] An input signal 14 (e.g., noise) is processed in different steps to become a generated audio signal 16 (e.g., under conditions set by the conditioning sets 71-73 of the trainable layer and based on the parameters 74, 75 learned by the conditioning sets 71-73 of the trainable layer). It is therefore understood that the input signal progresses in a processing direction (from 14 to 16 in FIG. 6) to become a generated audio signal 16 (e.g., speech). The conditions are substantially generated based on the target signal 12 and the training (to become the most preferred set of parameters 74, 75).

[0182] It is also noted that the multiple channels of the input signal (or any of its evolved forms) can be considered to have associated sets of learnable layers and stylization elements. For example, each row of matrices 74 and 75 is associated with a particular channel of the input signal (or one of its evolved forms) and thus is obtained from a particular learnable layer associated with that particular channel. Similarly, stylization element 77 can be considered to be formed by multiple stylization elements (one for each row of input signal x, c, 12, 76, 76', 59, 59a, 59b, etc.).

[0183] TIFF0007617261000001.tif63170

[0184] TIFF0007617261000002.tif33170

[0185] The noise vector 14 is processed step by step (e.g. in blocks 50a to 50h, 42, 44, 46, etc.) to evolve from e.g. noise 14 to e.g. speech 16 (the evolved signal is represented by different signals e.g. 15, 59a, x, c, 76', 79, 79a, 59b, 79b, 69, etc.).

[0186] In block 30, the input signal (noise) 14 may be upsampled to have 88 (or other number possible) samples and 64 (or other number possible) channels.

[0187] As can be seen, the eight processing blocks 50a, 50b, 50c, 50d, 50e, 50f, 50g, 50h (which together embody the first processing block 50 of FIG. 6) can increase the number of samples by performing upsampling (for example upsampling up to two times). Along the blocks 50a, 50b, 50c, 50d, 50e, 50f, 50g, 50h the number of channels may always remain the same (for example 64). The samples may for example be the number of samples per second (or other time unit). At the output of the block 50h it is possible to obtain a sound at more than 22 kHz.

[0188] Each of the blocks 50a-50h (50) may also be a TADEResBlock (residual block in the context of time-adaptive denormalization (TADE)). In particular, each of the blocks 50a-50h may be conditioned by the target data 12 (e.g., text features, linguistic features, or acoustic features such as mel spectrograms).

[0189] In the second processing block 45 (FIGS. 1 and 6), only one channel can be acquired, and multiple samples are acquired in a single magnitude. As can be seen, another TADE-ResBlock 42 (reducing to a single channel) is used (in addition to blocks 50a-50h). Then, a convolution layer and an activation function (which can be TanH 46 for example) can be performed. After that, the audio 16 is acquired (and possibly stored, rendered, encoded, etc.).

[0190] At least one (or in a particular embodiment each) of the blocks 50a-50h may for example be a residual block. A residual block performs a prediction only on the residual component of the signal evolving from the input signal 14 (e.g. noise) to the output audio signal 16. The residual signal is only a part (residual component) of the main signal. For example, multiple residual signals can be added together to obtain the final output audio signal 16.

[0191] 4 shows one embodiment of blocks 50a-50h (50). As can be seen, each block 50 receives a first data 59a as input, which can be either the input signal 14 (or an upsampled version output by upsampling block 30) or the output from a preceding block. For example, block 50b receives the output of block 50a, block 50c receives the output of block 50b, and so on.

[0192] Thus, in Fig. 4, it can be seen that the first data 59a provided to the block 50 (50a-50h) is processed and its output is the output signal 69 (provided as an input to the subsequent block). As shown by the line 59a', the main components of the first data 59a input to the first processing block 50a-50h essentially bypass most of the processing of the first processing block 50a-50h (50). For example, blocks 60a, 61a, 62a, 63a, 65a, 60b, 61b, 62b, 63b, 64b, 65b are bypassed by the bypass line 59a'. The first data 59a is then added to the residual portion 64b' in adder 65c (described in Fig. 4 but not shown). The bypass line 59a' and the summation at summer 65c can be understood to instantiate that each block 50 (50a-50h) performs an operation on the residual signal that is then added to the main portion of the signal. Thus, each of blocks 50a-50h can be considered a residual block.

[0193] In particular, the addition in the adder 65c does not necessarily have to be performed within the residual block 50 (50a-50h). The addition of the multiple residual signals 65b' (each output from each of the residual blocks 50a-50h) can be performed only once (e.g., in an addition block of the second processing block 45). Thus, the different residual blocks 50a-50h can operate in parallel with each other.

[0194] In the embodiment of FIG. 4, each block 50 may repeat the convolutional layer twice (e.g., once with replica 600 including at least one of blocks 60a, 61a, 62a, 63a, 64a, 65a to obtain signal 59b, and a second time with replica 601 including at least one of blocks 60b, 61b, 62b, 63b, 64b, 65b to obtain signal 65b' that may be added to main component 59a').

[0195] For each replica (600, 601), the conditioning set and stylization elements of the learnable layers 71-73 are applied (e.g. twice for each block 50) to the signal evolving from the input signal 16 to the audio output signal 16. A first time-adaptive denormalization (TADE) is performed in the TADE block 60a on the first data 59a of the first replica 600. The TADE block 60a performs a modulation of the first data 59a (the input signal or, for example, a processed noise) under conditions set by the target data 12. In the first TADE block 60a, an upsampling of the target data 12 is performed in the upsampling block 70 to obtain an upsampled version 12' of the target data 12. The upsampling can be obtained through a nonlinear interpolation, for example using a factor of 2, a power of 2, a multiple of 2 or any other value greater than 2. Thus, in some embodiments, the spectrogram 12' may be of the same magnitude (e.g., coincident) with the signal (76, 76', x, c, 59, 59a, 59b, etc.) conditioned by the spectrogram. In block 77 (the stylization element), application of stylization information may be performed to the processed noise (first data) (76, 76', x, c, 59, 59a, 59b, etc.). In the subsequent replica 601, another TADE block 60b may be applied to the output 59b of the first replica 600. An embodiment of the TADE block 60 (60a, 60b) is shown in FIG. 3 (see also below). After modulating the first data 59a, convolutions 61a and 62a are performed. Then, the activation function TanH and a softmax function (e.g., constituting a softmax-gated TanH function) are performed as well (63a, 64a). The outputs of activation functions 63a and 64a are multiplied (e.g., instantiating a gate) in multiplication block 65a to obtain result 59b. If two different replicas 600 and 601 are used (or more than two replicas are used), the passes through blocks 60a, 61a, 62a, 63a, 64a, 65a are repeated.

[0196] In an embodiment, the first and second convolutions in 61b and 62b downstream of TADE blocks 60a and 60b, respectively, may be performed with the same number of elements in the kernel (e.g., 9, e.g., 3×3), except that the second convolutions 61b and 62b may have a dilation factor of 2. In an embodiment, the maximum dilation factor of the convolutions may be 2.

[0197] TIFF0007617261000003.tif109170

[0198] After the styler element 77, the signal is output. The convolutions 72 and 73 do not necessarily have downstream activation functions. It is further noted that the parameter γ (74) can be understood as the variance and β (75) as the bias. Also, block 42 in FIG. 1 can be instantiated as block 50 in FIG. 3. Then, for example, the convolution layer 44 reduces the number of channels to 1, and then TanH 56 is performed to obtain the speech 16.

[0199] FIG. 7 shows an example of the evolution of the latent noise c (14) in one of the replicas 600 and 601 of one of the blocks 50a-50h as the target data 12 (e.g., a mel spectrogram) and the signal evolving into an audio signal 16, also shown at 59a or generated from the input signal 12.

[0200] In particular, 61a, 61b, 62a, 62b may be (or be part of) a set of learnable layers configured to process data derived from the first data (e.g., alternately from the input signal 14) with an activation function (63a, 64a, 63b, 64b, etc.) that is a gated activation function (second activation function). This set of learnable layers may consist of one or two or more convolutional layers. The second activation function may be a gated activation function (such as TanH and softmax). This property may be combined with the first activation function (for obtaining the first convolved data 71′) being ReLu or leaky ReLU.

[0201] The following steps (or at least one of the steps) may be performed: From an input such as text 112 (such as American Standard Code for Information Interchange (ASCII) or other type of code), target data 12 (textual, linguistic or acoustic features such as mel spectrograms) are generated (different types of target data may be used). The target data (eg mel spectrogram) 12 is subjected to at least one of the following steps: Upsampled in an upsampling block 70 to obtain an upsampled spectrogram 12'. Convolution is performed in the convolution layers 71-73 (part of the weighting layer) (e.g. the kernel 12a is convolved with the upsampled spectrogram 12'). · γ(74) and β(75) are obtained (learned). · γ (74) and β (75) are applied (eg by convolution) to the latent signal 59a (15) evolved from the input signal 14 and to the generated audio signal 16.

[0202] TTS Text-to-speech (TTS) (e.g., performed using block 1110) seeks to synthesize text 112 accompanied by an understandable, natural-sounding voice 16. The industrial applications are widespread, especially in machine-to-human communication.

[0203] The audio generator 10 of the present invention includes various components at the final stage, among them a vocoder 1120, and mainly comprises blocks for converting textual, linguistic or acoustic characteristics into an audio waveform 16.

[0204] In particular, in block 1110, text 112 (input) is analyzed and linguistic features are extracted from the text 112, for example by a text analysis module (sub-block) 1112 shown in FIG. 9a. Text analysis may include a number of tasks, such as text normalization, word segmentation, phonological prediction and grapheme-to-phoneme conversion (see also FIG. 8). These linguistic features (which may serve as intermediate target data 212) are then converted, for example through an acoustic model (e.g. by sub-block 1114), into acoustic features such as MFCCs, fundamental frequencies, mel spectrograms, or combinations thereof, which may constitute the target data 12 of FIGS. 1 and 3-8.

[0205] It should be noted that this pipeline can be replaced by end-to-end processing, for example by introducing DNN. For example, it would be possible to condition the neural vocoder 1120 directly from the linguistic features (e.g. in cases B and D of Fig. 10), or the acoustic model could process the characters directly, bypassing the text analysis stage (sub-block 1114 of Fig. 9a is not used). It is also possible to simplify the text analysis module by using end-to-end models such as Tacotron 1 and 2 in block 1110, for example, and take the character / phoneme sequence directly as the input sequence, outputting e.g. acoustic features (target data 12) in the form of e.g. mel spectrograms.

[0206] The solution can be used as a TTS system (i.e., includes both blocks 1110 and 1120), where the target data 12 may include, in some embodiments, an information stream or speech representation derived from the text 112. The representation may be, for example, characters or phonemes derived from the text 112, i.e., the usual input to the text analysis block. In this case, a pre-conditioned learning layer may be used in the block 1110 to extract acoustic or conditioning features (target data 12) suitable for, for example, a neural vocoder (such as block 1120). This pre-conditioning layer 1110 may use a deep neural network (DNN) such as an encoder-attention-decoder to directly map characters or phonemes to acoustic features. Alternatively, the representation (target data) 12 may be or include linguistic features, i.e., phonemes associated with information such as prosody, intonation, pauses, etc. Alternatively, the representation (target data) 12 may be or include linguistic features, i.e., phonemes associated with information such as prosody, intonation, pauses, etc. In this case, the pre-conditioned learnable layer 1110 can be an acoustic model that maps linguistic features to acoustic features based on a statistical model such as a hidden Markov model (HMM), a deep neural network (DNN) or a recurrent neural network (RNN). Finally, the target data 12 may directly contain acoustic features that are derived from the text 112 and can be used as conditioning features, for example after the learnable layer or the deterministic pre-conditioning layer 1110. In an extreme case (e.g., case F in Figure 10), acoustic features in the target data 12 can be used directly as conditioning features and any pre-conditioning layers bypassed. In accordance with the above, the audio synthesis block 1110 (text analysis block) may be deterministic in some embodiments, but in other cases may be obtained via at least one learnable layer.

[0207] In an embodiment, the target data 12 may include acoustic features such as log spectrograms or spectrograms, MFCCs or mel spectrograms obtained from the text 112 .

[0208] Alternatively, the target data 12 may include linguistic features such as phonemes, prosody, intonation, sentence breaks, or voiced pauses obtained from text.

[0209] The target data may be derived from the text using at least one of a statistical model, a learnable model, or a rule-based algorithm, which may include text analysis and / or an acoustic model.

[0210] Thus, in general, an audio synthesis block 1110 that outputs target data 12 from an input (e.g., text) such as text 112 (such that the target data 12 is derived from the text 112) can be either a deterministic block or a trainable block.

[0211] Generally, the target data 12 comprises multiple channels, whereas the text 112 (from which the target data 12 is derived) may only have one channel.

[0212] 9a shows an embodiment of a generator 10a (which may be an embodiment of the generator 10), where the target data 12 includes at least one acoustic feature such as a log spectrogram, or a spectrogram, MFCC or mel spectrogram obtained from the text 112. Here, the block 1110 includes a text analysis block 1112 providing intermediate target data 212, which may include at least one linguistic feature such as phonemes, prosody, intonation, sentence breaks or voiced pauses obtained from the text 112. Then, an audio synthesis block 1114 (e.g. using an acoustic model) generates the target data 12 as at least one acoustic feature such as a log spectrogram, or a spectrogram, MFCC or mel spectrogram obtained from the text 112.

[0213] Thereafter, a waveform synthesis block 1120 (which may be any of the waveform synthesis blocks described above) may be used to generate the output audio signal 16 .

[0214] FIG. 9b shows an embodiment of generator 10b (which may be an embodiment of generator 10), where target data includes at least one of linguistic features such as phonemes, prosody, intonation, sentence breaks, or voiced pauses obtained from text 112. Waveform synthesis (such as vocoder 1120) may be used to output audio signal 16. Waveform synthesis block 1120 may be any of those described in FIGS. 1-8 discussed above. In this case, for example, target data may be fed directly into the conditioning set of trainable layers 71-73 to obtain γ and β (74 and 75).

[0215] Figure 9c shows an embodiment of generator 10c (which may be an embodiment of any of generators 10 of Figures 1-8), where text 112 is used directly as target data. Since target data 12 includes characters and / or words taken from text 112, waveform synthesis block 1120 may be any of the above embodiments.

[0216] In general, any of the audio generators described above (and in particular any of the text analysis blocks 1110 (e.g., any of Figures 8 or 9a-9c)) may derive target data from text using at least one of statistical models, learnable models, or rule-based algorithms consisting of text analysis and / or acoustic models.

[0217] In some embodiments, the target data 12 may be obtained deterministically by block 1120. In other embodiments, the target data 12 may be obtained non-deterministically and block 1110 may be a learnable layer or multiple learnable layers.

[0218] GAN discriminator During training, for example, the GAN classifier of Figure 2 may be used to obtain parameters 74 and 75 to be applied to the input signal 12 (or a processed and / or normalized version of the input signal). Training may be performed prior to interference, and parameters 74 and 75 may be stored, for example in non-transitory memory, for subsequent use (although in some implementations parameters 74 and 75 may be calculated on the fly).

[0219] The GAN classifier 100 has the role of learning how to distinguish a generated audio signal (e.g., audio signal 16 synthesized as described above) from an actual input signal (e.g., real speech) 104. Thus, the role of the GAN classifier 100 is mainly exercised during training (e.g., training parameters 72 and 73) and is considered to be the opposite of the role of the GAN generator 11 (which can be considered as an audio generator 10 without the GAN classifier 100).

[0220] In general, the GAN classifier 100 is input with both the audio signal 16 synthesized and generated by the GAN generator 10 and an actual audio signal (e.g., real speech) 104, e.g., acquired via a microphone, and processes the signals to obtain a metric (e.g., loss) that is minimized. The actual audio signal 104 can also be considered as a reference audio signal. During training, the operations as described above for synthesizing the speech 16 may be repeated, e.g., multiple times, to obtain, e.g., parameters 74 and 75.

[0221] In an embodiment, instead of analysing the entire reference audio signal 104 and / or the entire generated audio signal 16, it is also possible to analyse only parts of it (e.g. a portion, a slice, a window, etc.). Signal parts generated with random windows (105a-105d) sampled from the generated audio signal 16 and the reference audio signal 104 are obtained. For example, a random window function can be used such that it is not pre-set a priori which of the windows 105a, 105b, 105c, 105d is used. Also the number of windows is not necessarily four but may vary.

[0222] In the windows (105a-105d) a PQMF (Quadrature Mirror Filter Bank) 110 may be applied, thus obtaining subbands 120. Thus, a decomposition (110) of a representation of the generated audio signal (16) or of the reference audio signal (104) is obtained.

[0223] An evaluation block 130 may be used to perform the evaluation. A number of evaluators 132a, 132b, 132c, 132d (compositely denoted 132) may be used (although a different number may be used). In general, a window 105a, 105b, 105c, 105d may be input to each evaluator 132a, 132b, 132c, 132d, respectively. The sampling of the random window (105a-105d) may be repeated multiple times for each evaluator (132a-132d). In an embodiment, the number of times the random window (105a-105d) is sampled for each evaluator (132a-132d) may be proportional to the length of the representation of the generated audio signal or the representation of the reference audio signal. Thus, each evaluator (132a-132d) may receive as input a representation of the generated audio signal (16) or a representation of the reference audio signal (104).

[0224] Each of the evaluators 132a-132d may be a neural network itself. Each of the evaluators 132a-132d may, among other things, follow the paradigm of convolutional neural networks. Each of the evaluators 132a-132d may be a residual evaluator. Each of the evaluators 132a-132d may have parameters (weights, etc.) that are adapted during training (e.g., similar to one of those described above).

[0225] 2, each estimator 132a-132d performs downsampling (e.g., by a downsampling factor of 4 or other). For each estimator 132a-132d, the number of channels is increased (e.g., by a factor of 4, or in some embodiments, by a multiple of the same number as the downsampling factor).

[0226] Upstream and downstream of the estimator may be provided convolutional layers 131 and / or 134. The upstream convolutional layer 131 may for example have a kernel of size 15 (e.g. 5×3 or 3×5). The downstream convolutional layer 134 may for example have a kernel of size 3 (e.g. 3×3).

[0227] During training, a loss function (adversarial loss) 140 may be optimized. The loss function 140 may include a fixed metric (obtained, for example, in a pre-training step) between the generated audio signal (16) and the reference audio signal (104). The fixed metric may be obtained by calculating one or more spectral distortions between the generated audio signal (16) and the reference audio signal (104). The distortions may be - the magnitude or log magnitude of the spectral representation of the generated audio signal (16) and the reference audio signal (104), and / or - Different time or frequency resolutions The measurement shall be made taking into account the above.

[0228] In an embodiment, the adversarial loss can be obtained by randomly feeding and evaluating representations of the generated audio signal (16) or representations of the reference audio signal (104) to one or more evaluators (132). The evaluation includes classifying the fed audio signals (16, 132) into a predefined number of classifications indicating classification levels of naturalness of the pre-trained audio signals (14, 16). The predefined number of classifications can be, for example, "real" vs. "fake".

[0229] TIFF0007617261000004.tif74170

[0230] TIFF0007617261000005.tif43170

[0231] TIFF0007617261000006.tif35170

[0232] Other types of minimization may also be performed.

[0233] In general, the 2-subadversarial loss 140 is associated with the best parameters (74, 75, etc.) applied to the stylization element 77.

[0234] Consider The embodiments of the present disclosure are described in detail with the accompanying drawings. Particularly, in the following description, many details are described in order to more comprehensively describe the embodiments of the disclosure. However, it is obvious to those skilled in the art that other embodiments can be implemented without these specific details. The features of different embodiments described in this specification can be combined with each other unless the features of the corresponding combinations are mutually exclusive or such combinations are expressly excluded.

[0235] It is noted that the same or similar elements having the same function are indicated with the same reference number, or are designated as identical by the description of the elements given the same or similar reference number, or are typically omitted. Descriptions of elements with the same or similar reference numbers or elements labeled with the same label are interchangeable with each other.

[0236] In many applications, such as text-to-speech, speech coding, and speech enhancement, neural vocoders have proven to outperform traditional approaches in synthesizing natural, high-quality speech. The first pioneering generative neural network for high-quality speech synthesis was WaveNet, and many other approaches were developed shortly thereafter. These models offer state-of-the-art quality, but are often computationally expensive and synthesis is very slow. In recent years, a large number of models have been introduced that generate speech at low computational cost. Some of these are optimized versions of existing models, others leverage integration with traditional methods. However, a number of completely new approaches have also been introduced, often relying on GANs. Most GAN vocoders offer fast generation on GPUs, but at the expense of a loss in the quality of the synthesized speech.

[0237] One of the main objectives of this paper is to propose a GAN architecture, which we call StyleMelGAN (and which can be implemented, for example, in an audio generator 10), capable of synthesizing extremely high quality speech 16 with low computational cost and rapid training. The generator network of StyleMelGAN has 3.86 million trainable parameters and is capable of synthesizing speech at 22.05 kHz, 2.6 times faster than real time on a CPU and over 54 times faster on a GPU. The model consists, for example, of eight upsampling blocks that progressively convert a low-dimensional noise vector (e.g., 30 in Fig. 1) into a raw speech waveform (e.g., 16). The synthesis can be conditioned against the mel spectrogram of the target speech (or more generally, by the target data 12), which can be inserted into all generator blocks (50a-50h) via time-adaptive denormalization (TADE) layers (60, 60a, 60b). This approach of inserting conditioning properties is extremely efficient and, to the best of our knowledge, is new in the audio domain. The adversarial loss is computed (e.g., in the GAN classifier 100, via the structure of FIG. 2) by an ensemble of four classifiers 132a-132d (although in some embodiments the number of classifiers may be different), each operating after a differentiable pseudo-quadrature mirror filter bank (PQMF) 110. This allows different frequency bands of the audio signal (104 or 16) to be analyzed during training. To make the training more robust and to stimulate the generation, the classifiers (e.g., the four classifiers 132a-132d) are not conditioned by the input acoustic features used by the generator 10, and the audio signal (104 or 16) is sampled using a random window (e.g., 105a-105d).

[0238] In summary, StyleMelGAN is proposed, a low-complexity GAN for high-quality speech synthesis conditioned by mel-spectrograms (e.g., 12) through TADE layers (e.g., 60, 60a, 60b). The generator 10 is highly parallelizable. The generator 10 can be fully convolutional. The generator 10 can be adversarially trained by an ensemble of PQMF multi-sampling random window classifiers (e.g., 132a-132d) that can be regularized by a multi-scale spectral reconstruction loss. The quality of the generated speech 16 can be evaluated using both objective evaluations (e.g., Fréchet scores) and / or subjective evaluations. Two listening tests have been conducted, the MUSHRA test for copy-synthesis scenarios and the P.800 ACR test for TTS, both of which confirmed that StyleMelGAN achieves state-of-the-art speech quality.

[0239] Existing neural vocoders usually synthesize speech signals directly in the time domain by modeling the amplitude of the final waveform. Most of these models are generative neural networks, i.e. they model the probability distribution of speech samples observed in natural speech signals. They can be decomposed autoregressively, which factorizes the distribution into a product of a conditional distribution and a non-autoregressive or parallel distribution that instead models the joint distribution directly. Autoregressive models such as WaveNet, SampleRNN and WaveRnn have been reported to synthesize speech signals with high perceptual quality. One major branch of non-autoregressive models is the regularization stream, such as WaveGlow. A hybrid approach is the use of inverse autoregressive streams that use a factorization transformation between the noise latent representation and the target speech distribution. The above examples are mainly about autoregressive neural networks.

[0240] Early applications of GANs for audio include WaveGAN for unconditioned speech generation and Gan-Synth for music generation. MelGAN learns a mapping between the mel spectrogram of speech segments and the corresponding time-domain waveform. It guarantees faster-than-real-time generation and leverages adversarial learning of a multi-scale classifier regularized by a spectral reconstruction loss. GAN-TTS is the first GAN vocoder to exclusively use adversarial learning for speech generation with acoustic conditioning. The adversarial loss is computed by an ensemble of conditioned and unconditioned random window classifiers. Parallel WaveGAN uses a generator similar in structure to WaveNet, trained with an unconditioned classifier regularized by a multi-scale spectral reconstruction loss. A similar idea is used in multiband MelGAN, which generates each subband of the target speech separately, saving computational power, and then uses a synthetic PQMF to obtain the final waveform. The multi-scale classifier evaluates the full-band speech waveform and is regularized with a multiband-scale spectral reconstruction loss. Research in this area is very active, with recent GAN vocoders such as VacGan and HooliGAN.

[0241] TIFF0007617261000007.tif98170

[0242] TIFF0007617261000008.tif75170

[0243] FIG. 4 shows the structure of a part of the audio generator 10, illustrating the basic building block of the generator model, the TADE-ResBlock 50 (which may be any of the blocks 50a to 50h). The complete architecture is shown in FIG. 1. It contains eight upsampling stages 50a to 50h (in other embodiments it may be a number other than eight), for example consisting of a TADEResBlock and a layer 601 for upsampling the signal 79b by a factor of 2, as well as a final activation module 46 (in FIG. 1). The final activation includes one TADEResBlock 42 followed by a channel modification convolution layer 44 with, for example, Tanh nonlinearity 46. This design allows for example to use a channel depth of 64 for the convolution operation, thus saving complexity. This upsampling procedure allows the expansion factor to be kept below 2.

[0244] FIG. 2 shows the architecture of a filter bank random window classifier (FB-RWD). StyleMelGAN can use multiple (e.g., four) classifiers 132a-132d for adversarial learning, and in an embodiment, the architecture of the classifiers 132-132d is free of average pooling downsampling. Furthermore, each classifier (132a-132d) may operate on a random window (105a-105d) sliced ​​from the input speech waveform (104 or 16). Finally, each classifier (132a-132d) may analyze a subband 120 of the input speech signal (104 or 16) obtained from the analysis PQMF (e.g., 110). More precisely, in an embodiment, 1, 2, 4, and 8 subbands may be used, respectively, calculated from a selected random segment of 512, 1024, 2048, and 4096 samples, respectively, extracted from a 1-second waveform. This allows multi-resolution adversarial evaluation of audio signals (104 or 16) in both the time and frequency domains.

[0245] TIFF0007617261000009.tif202170

[0246] The models used in the experiments are listed below. WaveNet for targeted experiments in copy synthesis and text-to-speech PWGAN for targeted experiments in copy synthesis and text-to-speech MelGAN for targeted experiments in copy synthesis for objective evaluation WaveGlow for target experiments in copy synthesis Transformer.v3 for targeted experiments in text-to-speech

[0247] Objective and subjective evaluations of StyleMelGAN against the pre-trained baseline vocoder models listed above were performed. The subjective quality of the audio TTS output was evaluated in a controlled environment via the P.800 listening test performed by listeners. The test set contains unseen utterances recorded by the same speaker and randomly selected from the LibriVox online corpus. Thus, the model is robust and does not rely primarily on the training data. These utterances were recorded under slightly different conditions and exhibit a variety of prosody, thus testing the generalization ability of the model. The original utterances were resynthesized using the GriffinLim algorithm and used instead of the usual anchor conditions. This facilitates the use of a full evaluation scale.

[0248] Classical objective measures such as PESQ and POLQA are unreliable for evaluating the speech waveforms generated by neural vocoders. Instead, the conditional Fréchet deep phonetic distance (cFDSD) is used. Below are the cFDSD scores for different neural vocoders, showing that StyleMelGAN significantly outperforms other models. MelGAN Training cFDSD 0.235 Experimental cFDSD 0.227 PWGAN Training cFDSD 0.122 Experimental cFDSD 0.101 WaveGlow Training cFDSD 0.099 Experimental cFDSD 0.078 WaveNet Training cFDSD 0.176 Experimental cFDSD 0.140 StyleMelGAN Training cFDSD 0.044 Experimental cFDSD 0.068

[0249] We can see that StyleMelGAN outperforms other adversarial and non-adversarial vocoders.

[0250] A MUSHRA listening test was conducted with a group of 15 experienced listeners. This type of test was chosen because it allows a more accurate assessment of the generated speech quality. The anchors were generated using a Py-Torch implementation of 32 iterations of the Griffin-Lim algorithm. Figure 5 shows the results of the MUSHRA test. It can be seen that StyleMelGAN significantly outperforms other vocoders by approximately 15 MUSHRA points. The results show that WaveGlow produces output of similar quality to WaveNet, but is comparable to Parallel WaveGAN.

[0251] The subjective quality of the audio TTS can be assessed through a P.800 ACR listening test administered by 31 listeners in a controlled environment. The Transformer.v3 model from ESPNET can be used to generate mel spectrograms of the transcriptions of the test set. The same Griffin-Lim anchors can also be added, as this makes the overall evaluation scale more accessible.

[0252] Below are the P800 Mean Opinion Score (MOS) for different TTS systems, which show the same finding that StyleMelGAN clearly outperforms other models. · GriffinLim P800MOS:1.33+ / -0.04 Transformer + Parallel WaveGAN P800MOS: 3.19+ / -0.07 · Transformer+WaveNet P800MOS:3.82+ / -0.07 · Transformer+StyleMelGAN P800MOS:4.00+ / -0.07 Recording P800MOS:4.29+ / -0.06

[0253] Below we show the generation speed and number of parameters in real-time factor (RTF) for different parallel vocoder models. StyleMelGAN offers a clear compromise between generation quality and inference speed.

[0254] Below we show the number of parameters and real-time coefficients for generation on a CPU (e.g., Intel Core i7-6700 3.40 GHz) and a GPU (e.g., Nvidia GeForce GTX1060) for the various models under consideration. Parallel WaveGAN parameters: 1.44M, CPU: 0.8x, GPU: 17x MelGAN parameters: 4.26M, CPU: 7x, GPU: 110x StyleMelGAN parameters: 3.86M, CPU: 2.6x, GPU: 54x WaveGlow parameters: 80M, -, GPU: 5x

[0255] Finally, Figure 5 shows the results of the MUSHRA expert listening test, where we can see that StyleMelGAN outperforms state-of-the-art models.

[0256] conclusion This paper presents StyleMelGAN, a lightweight and efficient adversarial vocoder for high-fidelity speech synthesis. Instead of simply feeding conditioning to the first layer, the model uses temporal adaptive normalization (TADE) to distribute sufficient and accurate conditioning to all generative layers. For adversarial learning, the generator competes with a filterbank random window classifier that provides a multi-scale representation of the speech signal in both the time and frequency domains. StyleMelGAN runs on both CPU and GPU at a magnitude scale faster than real time. Objective and subjective experimental results show that StyleMelGAN significantly outperforms previous adversarial vocoders as well as autoregressive, flow-based and diffusion-based vocoders, setting a new state-of-the-art baseline for neural waveform generation.

[0257] Finally, the embodiments described herein may be optionally supplemented by any of the key points or aspects described herein, although it is noted that the key points and aspects described herein may be used individually or in combination and may be incorporated individually or in combination into any of the embodiments described herein.

[0258] Although some aspects are described in the context of an apparatus, it will be apparent that these aspects also refer to a description of a corresponding method, where the apparatus or parts thereof correspond to method steps or features of method steps. Similarly, aspects described in the context of a method step also refer to a corresponding apparatus or part of an apparatus or a corresponding item or feature of an apparatus. Some or all of the method steps may be performed by (or using) a hardware apparatus, e.g. a microprocessor, a programmable computer or an electronic circuit. In some embodiments, one or more of the most significant method steps may be performed by such an apparatus.

[0259] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. Implementation may be performed using a digital storage medium, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM or flash memory, on which electronically readable control signals are stored and which cooperate (or can cooperate) with a programmable computer system to execute the respective method. The digital storage medium may thus be computer readable.

[0260] Some embodiments according to the present invention include a data carrier comprising electronically readable control signals capable of cooperating with a programmable computer system to cause one of the methods described herein to be performed.

[0261] Generally, embodiments of the present invention may be implemented as a computer program product having program code which operates to perform one of the methods when the computer program product is run on a computer, the program code may for example be stored on a machine readable carrier.

[0262] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0263] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0264] Another embodiment of the present invention is therefore a data carrier (or digital storage medium, or computer readable medium) having recorded thereon a computer program for performing one of the methods described herein. The data carrier, digital storage medium, or recording medium is typically tangible and non-transitory.

[0265] Another embodiment of the present invention is therefore a data stream or a sequence of signals representing the computer program for performing one of the methods described herein, the data stream or the sequence of signals being for example adapted to be transmitted via a data communication connection, for example the Internet.

[0266] Another embodiment comprises a processing means, such as a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0267] Another embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0268] Another embodiment according to the invention comprises an apparatus or system configured to transmit (e.g. electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be e.g. a computer, a mobile device, a memory device, etc. The apparatus or system may comprise e.g. a file server for transmitting the computer program to the receiver.

[0269] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by any hardware apparatus.

[0270] The apparatus described herein may be implemented using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer. The apparatus described herein or any components of the apparatus described herein may be implemented at least in part in hardware and / or software. The methods described herein may be performed using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer. The methods described herein, or any part of the methods described herein, may be performed at least in part in hardware and / or software.

[0271] The above-described embodiments are merely illustrative of the principles of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be obvious to those skilled in the art. It is therefore intended to be limited by the scope of the appended claims and not by the specific details shown by the description and illustration of the examples described herein.

[0272] Prior art documents A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, et al., "WaveNet: A Generative Model for Raw Audio," arXiv:1609.03499, 2016. R. Prenger, R. Valle, and B. Catanzaro, "Waveglow: A Flow-based Generative Network for Speech Synthesis," in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019, pp. 3617?3621. S. Mehri, K. Kumar, I. Gulrajani, R. Kumar, et al., "SampleRNN: An Unconditional End-to-End Neural Audio Generation Model," arXiv:1612.07837, 2016. N. Kalchbrenner, E. Elsen, K. Simonyan, S. Noury, et al., "Efficient neural audio synthesis," arXiv:1802.08435, 2018. A. van den Oord, Y. Li, I. Babuschkin, K. Simonyan, et al., "Parallel WaveNet: Fast High-Fidelity Speech Synthesis," in Proceedings of the 35th ICML, 2018, pp. 3918?3926. J. Valin and J. Skoglund, "LPCNET: Improving Neural Speech Synthesis through Linear Prediction," in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019, pp. 5891?5895. K. Kumar, R. Kumar, de T. Boissiere, L. Gestin, et al., "MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis," in Advances in NeurIPS 32, pp. 14910?14921. 2019. R. Yamamoto, E. Song, and J. Kim, "Parallel Wavegan: A Fast Waveform Generation Model Based on Generative Adversarial Networks with Multi-Resolution Spectrogram," in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 6199?6203. M. Bin´kowski, J. Donahue, S. Dieleman, A. Clark, et al., "High Fidelity Speech Synthesis with Adversarial Networks," arXiv:1909.11646, 2019. T. Park, M. Y. Liu, T. C. Wang, and J. Y. Zhu, "Se-mantic Image Synthesis With Spatially-Adaptive Normalization," in Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. P. Govalkar, J. Fischer, F. Zalkow, and C. Dittmar, "A Comparison of Recent Neural Vo-coders for Speech Signal Reconstruction," in Proceedings of the ISCA Speech Synthesis Workshop, 2019, pp. 7?12. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, et al., "Generative Adversarial Nets," in Advances in NeurIPS 27, pp. 2672?2680. 2014. C. Donahue, J. McAuley, and M. Puckette, "Adversarial Audio Synthesis," arXiv:1802.04208, 2018. J. Engel, K. K. Agrawal, S. Chen, I. Gulrajani, et al., "GANSynth: Adversarial Neural Au-dio Synthesis," arXiv:1902.08710, 2019. G. Yang, S. Yang, K. Liu, P. Fang, et al., "Multiband MelGAN: Faster Waveform Generation for High-Quality Text-to-Speech," arXiv:2005.05106, 2020. J. Yang, J. Lee, Y. Kim, H. Cho, and I. Kim, "VocGAN: A High-Fidelity Real-time Vocoder with a Hierarchically-nested Adversarial Network," arXiv:2007.15256, 2020. Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae, "Hifi-gan: Generative adversarial net-works for efficient and high fidelity speech synthesis," arXiv preprint arXiv:2010.05646, 2020. D. Ulyanov, A. Vedaldi, and V. Lempitsky, "Instance normalization: The missing ingredient for fast stylization," arXiv:1607.08022, 2016. A. Mustafa, A. Biswas, C. Bergler, J. Schottenhamml, and A. Maier, "Analysis by Adversarial Synthesis - A Novel Approach for Speech Vocoding," in Proc. Inter-speech, 2019, pp. 191?195. T. Q. Nguyen, "Near-perfect-reconstruction pseudo-QMF banks," IEEE Transactions on Signal Processing, vol. 42, no. 1, pp. 65?76, 1994. T. Salimans and D. P. Kingma, "Weight normalization: A simple reparameterization to accelerate training of deep neural networks," in Advances in NeurIPS, 2016, pp. 901?909. K. Ito and L. Johnson, "The LJ Speech Dataset," https: / / keithito.com / LJ-Speech-Dataset / , 2017. D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," arXiv:1412.6980, 2014. T. Hayashi, R. Yamamoto, K. Inoue, T. Yoshimura, et al., "Espnet-tts: Unified, reproducible, and integratable open source end-to-end text-to-speech toolkit," in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 7654?7658. A. Gritsenko, T. Salimans, R. van den Berg, J. Snoek, and N. Kalchbrenner, "A Spectral Energy Distance for Parallel Speech Synthesis," arXiv:2008.01160, 2020. “P.800: Methods for subjective determination of transmission quality," Standard, International Telecommunication Union, 1996.

Claims

1. An audio generator (10) configured to generate an audio signal (16) from an input signal (14) and target data (12), said target data (12) representing said audio signal (16); a first processing block (40, 50, 50a-50h) configured to receive first data (15, 59a) derived from said input signal (14) and to output first output data (69), said first output data (69) comprising a plurality of channels (47); a second processing block (45) configured to receive as second data the first output data (69) or data derived from the first output data (69); An audio generator (10) comprising: The first processing block (50) performs, for each channel of the first output data: a conditioning set (71, 72, 73) of a learnable layer configured to process the target data (12) to obtain conditioning characteristic parameters (74, 75), the target data (12) being derived from text; a styling element (77) configured to apply said conditioning characteristic parameters (74, 75) to said first data (15, 59a) or to a normalized version (59, 76') of said first data; Equipped with the second processing block (45) is configured to combine the multiple channels (47) of the second data (69) to obtain the audio signal (16); The target data (12) comprises at least one acoustic feature of a log spectrogram, a MFCC, a mel spectrogram or any other type of spectrogram obtained from text; The audio generator is configured to obtain the target data (12) by converting (1110) an input in a text form or in a text element form into the at least one acoustic characteristic.

2. 2. The audio generator of claim 1, wherein the target data (12) is a spectrogram.

3. 3. The audio generator of claim 2, wherein the target data (12) is a mel spectrogram.

4. An audio generator (10) configured to generate an audio signal (16) from an input signal (14) and target data (12), said target data (12) representing said audio signal (16); a first processing block (40, 50, 50a-50h) configured to receive first data (15, 59a) derived from said input signal (14) and to output first output data (69), said first output data (69) comprising a plurality of channels (47); a second processing block (45) configured to receive as second data the first output data (69) or data derived from the first output data (69); An audio generator (10) comprising: The first processing block (50) performs, for each channel of the first output data: a conditioning set (71, 72, 73) of a learnable layer configured to process the target data (12) to obtain conditioning characteristic parameters (74, 75), the target data (12) being derived from text; a styling element (77) configured to apply said conditioning characteristic parameters (74, 75) to said first data (15, 59a) or to a normalized version (59, 76') of said first data; Equipped with the second processing block (45) is configured to combine the multiple channels (47) of the second data (69) to obtain the audio signal (16); The target data (12) comprises at least one acoustic feature of a log spectrogram, a MFCC, a mel spectrogram or any other type of spectrogram obtained from text; The audio generator is configured to obtain the target data (12) by converting (1114) at least one linguistic feature into at least one acoustic feature.

5. An audio generator (10) configured to generate an audio signal (16) from an input signal (14) and target data (12), said target data (12) representing said audio signal (16); a first processing block (40, 50, 50a-50h) configured to receive first data (15, 59a) derived from said input signal (14) and to output first output data (69), said first output data (69) comprising a plurality of channels (47); a second processing block (45) configured to receive as second data the first output data (69) or data derived from the first output data (69); An audio generator (10) comprising: The first processing block (50) performs, for each channel of the first output data: a conditioning set (71, 72, 73) of a learnable layer configured to process the target data (12) to obtain conditioning characteristic parameters (74, 75), the target data (12) being derived from text; a styling element (77) configured to apply said conditioning characteristic parameters (74, 75) to said first data (15, 59a) or to a normalized version (59, 76') of said first data; Equipped with the second processing block (45) is configured to combine the multiple channels (47) of the second data (69) to obtain the audio signal (16); the target data (12) comprises at least one linguistic feature of phonemes, prosody, intonation, sentence breaks, and voiced pauses obtained from a text; The audio generator is configured to obtain the target data (12) by converting (1110) an input in a text form or in a text element form into the at least one linguistic characteristic.

6. An audio generator (10) configured to generate an audio signal (16) from an input signal (14) and target data (12), said target data (12) representing said audio signal (16); a first processing block (40, 50, 50a-50h) configured to receive first data (15, 59a) derived from said input signal (14) and to output first output data (69), said first output data (69) comprising a plurality of channels (47); a second processing block (45) configured to receive as second data the first output data (69) or data derived from the first output data (69); An audio generator (10) comprising: The first processing block (50) performs, for each channel of the first output data: a conditioning set (71, 72, 73) of a learnable layer configured to process the target data (12) to obtain conditioning characteristic parameters (74, 75), the target data (12) being derived from text; a styling element (77) configured to apply said conditioning characteristic parameters (74, 75) to said first data (15, 59a) or to a normalized version (59, 76') of said first data; Equipped with the second processing block (45) is configured to combine the multiple channels (47) of the second data (69) to obtain the audio signal (16); The target data (12) includes at least one of characters and words taken from a text.

7. An audio generator (10) configured to generate an audio signal (16) from an input signal (14) and target data (12), said target data (12) representing said audio signal (16); a first processing block (40, 50, 50a-50h) configured to receive first data (15, 59a) derived from said input signal (14) and to output first output data (69), said first output data (69) comprising a plurality of channels (47); a second processing block (45) configured to receive as second data the first output data (69) or data derived from the first output data (69); An audio generator (10) comprising: The first processing block (50) performs, for each channel of the first output data: a conditioning set (71, 72, 73) of a learnable layer configured to process the target data (12) to obtain conditioning characteristic parameters (74, 75), the target data (12) being derived from text; a styling element (77) configured to apply said conditioning characteristic parameters (74, 75) to said first data (15, 59a) or to a normalized version (59, 76') of said first data; Equipped with the second processing block (45) is configured to combine the multiple channels (47) of the second data (69) to obtain the audio signal (16); The audio generator, wherein the target data (12) is derived from text (112) using a statistical model that performs text analysis and / or uses an acoustic model.

8. An audio generator (10) configured to generate an audio signal (16) from an input signal (14) and target data (12), said target data (12) representing said audio signal (16); a first processing block (40, 50, 50a-50h) configured to receive first data (15, 59a) derived from said input signal (14) and to output first output data (69), said first output data (69) comprising a plurality of channels (47); a second processing block (45) configured to receive as second data the first output data (69) or data derived from the first output data (69); An audio generator (10) comprising: The first processing block (50) performs, for each channel of the first output data: a conditioning set (71, 72, 73) of a learnable layer configured to process the target data (12) to obtain conditioning characteristic parameters (74, 75), the target data (12) being derived from text; a styling element (77) configured to apply said conditioning characteristic parameters (74, 75) to said first data (15, 59a) or to a normalized version (59, 76') of said first data; Equipped with the second processing block (45) is configured to combine the multiple channels (47) of the second data (69) to obtain the audio signal (16); The audio generator, wherein the target data (12) is derived from text (112) using a trainable model that performs text analysis and / or uses an acoustic model.

9. An audio generator (10) configured to generate an audio signal (16) from an input signal (14) and target data (12), said target data (12) representing said audio signal (16); a first processing block (40, 50, 50a-50h) configured to receive first data (15, 59a) derived from said input signal (14) and to output first output data (69), said first output data (69) comprising a plurality of channels (47); a second processing block (45) configured to receive as second data the first output data (69) or data derived from the first output data (69); An audio generator (10) comprising: The first processing block (50) performs, for each channel of the first output data: a conditioning set (71, 72, 73) of a learnable layer configured to process the target data (12) to obtain conditioning characteristic parameters (74, 75), the target data (12) being derived from text; a styling element (77) configured to apply said conditioning characteristic parameters (74, 75) to said first data (15, 59a) or to a normalized version (59, 76') of said first data; Equipped with the second processing block (45) is configured to combine the multiple channels (47) of the second data (69) to obtain the audio signal (16); The target data (12) is derived from text (112) using rule-based algorithms for text analysis and / or acoustic models.

10. An audio generator (10) configured to generate an audio signal (16) from an input signal (14) and target data (12), said target data (12) representing said audio signal (16); a first processing block (40, 50, 50a-50h) configured to receive first data (15, 59a) derived from said input signal (14) and to output first output data (69), said first output data (69) comprising a plurality of channels (47); a second processing block (45) configured to receive as second data the first output data (69) or data derived from the first output data (69); An audio generator (10) comprising: The first processing block (50) performs, for each channel of the first output data: a conditioning set (71, 72, 73) of a learnable layer configured to process the target data (12) to obtain conditioning characteristic parameters (74, 75), the target data (12) being derived from text; a styling element (77) configured to apply said conditioning characteristic parameters (74, 75) to said first data (15, 59a) or to a normalized version (59, 76') of said first data; Equipped with the second processing block (45) is configured to combine the multiple channels (47) of the second data (69) to obtain the audio signal (16); The audio generator is configured to derive the target data through at least one deterministic layer.

11. An audio generator (10) configured to generate an audio signal (16) from an input signal (14) and target data (12), said target data (12) representing said audio signal (16); a first processing block (40, 50, 50a-50h) configured to receive first data (15, 59a) derived from said input signal (14) and to output first output data (69), said first output data (69) comprising a plurality of channels (47); a second processing block (45) configured to receive as second data the first output data (69) or data derived from the first output data (69); An audio generator (10) comprising: The first processing block (50) performs, for each channel of the first output data: a conditioning set (71, 72, 73) of a learnable layer configured to process the target data (12) to obtain conditioning characteristic parameters (74, 75), the target data (12) being derived from text; a styling element (77) configured to apply said conditioning characteristic parameters (74, 75) to said first data (15, 59a) or to a normalized version (59, 76') of said first data; Equipped with the second processing block (45) is configured to combine the multiple channels (47) of the second data (69) to obtain the audio signal (16); The audio generator is configured to derive the target data through at least one learnable layer.

12. An audio generator (10) configured to generate an audio signal (16) from an input signal (14) and target data (12), said target data (12) representing said audio signal (16); a first processing block (40, 50, 50a-50h) configured to receive first data (15, 59a) derived from said input signal (14) and to output first output data (69), said first output data (69) comprising a plurality of channels (47); a second processing block (45) configured to receive as second data the first output data (69) or data derived from the first output data (69); An audio generator (10) comprising: The first processing block (50) performs, for each channel of the first output data: a conditioning set (71, 72, 73) of a learnable layer configured to process the target data (12) to obtain conditioning characteristic parameters (74, 75), the target data (12) being derived from text; a styling element (77) configured to apply said conditioning characteristic parameters (74, 75) to said first data (15, 59a) or to a normalized version (59, 76') of said first data; Equipped with the second processing block (45) is configured to combine the multiple channels (47) of the second data (69) to obtain the audio signal (16); 1. An audio generator, wherein the conditioning set of learnable layers consists of one or at least two convolutional layers (71-73).

13. An audio generator (10) configured to generate an audio signal (16) from an input signal (14) and target data (12), said target data (12) representing said audio signal (16); a first processing block (40, 50, 50a-50h) configured to receive first data (15, 59a) derived from said input signal (14) and to output first output data (69), said first output data (69) comprising a plurality of channels (47); a second processing block (45) configured to receive as second data the first output data (69) or data derived from the first output data (69); An audio generator (10) comprising: The first processing block (50) performs, for each channel of the first output data: a conditioning set (71, 72, 73) of a learnable layer configured to process the target data (12) to obtain conditioning characteristic parameters (74, 75), the target data (12) being derived from text; a styling element (77) configured to apply said conditioning characteristic parameters (74, 75) to said first data (15, 59a) or to a normalized version (59, 76') of said first data; Equipped with the second processing block (45) is configured to combine the multiple channels (47) of the second data (69) to obtain the audio signal (16); an audio generator, wherein a first convolutional layer (71-73) is configured to convolve the target data (12) or an upsampled version of the target data to obtain first convolved data (71') using a first activation function.

14. An audio generator (10) configured to generate an audio signal (16) from an input signal (14) and target data (12), said target data (12) representing said audio signal (16); a first processing block (40, 50, 50a-50h) configured to receive first data (15, 59a) derived from said input signal (14) and to output first output data (69), said first output data (69) comprising a plurality of channels (47); a second processing block (45) configured to receive as second data the first output data (69) or data derived from the first output data (69); An audio generator (10) comprising: The first processing block (50) performs, for each channel of the first output data: a conditioning set (71, 72, 73) of a learnable layer configured to process the target data (12) to obtain conditioning characteristic parameters (74, 75), the target data (12) being derived from text; a styling element (77) configured to apply said conditioning characteristic parameters (74, 75) to said first data (15, 59a) or to a normalized version (59, 76') of said first data; Equipped with the second processing block (45) is configured to combine the multiple channels (47) of the second data (69) to obtain the audio signal (16); The conditioning set (71-73) of the learnable layer and the style imparting element (77) are part of a weighting layer within a residual block (50, 50a-50h) of a neural network that includes one or more residual blocks (50, 50a-50h).

15. An audio generator (10) configured to generate an audio signal (16) from an input signal (14) and target data (12), said target data (12) representing said audio signal (16); a first processing block (40, 50, 50a-50h) configured to receive first data (15, 59a) derived from said input signal (14) and to output first output data (69), said first output data (69) comprising a plurality of channels (47); a second processing block (45) configured to receive as second data the first output data (69) or data derived from the first output data (69); An audio generator (10) comprising: The first processing block (50) performs, for each channel of the first output data: a conditioning set (71, 72, 73) of a learnable layer configured to process the target data (12) to obtain conditioning characteristic parameters (74, 75), the target data (12) being derived from text; a styling element (77) configured to apply said conditioning characteristic parameters (74, 75) to said first data (15, 59a) or to a normalized version (59, 76') of said first data; Equipped with the second processing block (45) is configured to combine the multiple channels (47) of the second data (69) to obtain the audio signal (16); The audio generator (10) further comprises a normalization element (76) configured to normalize the first data (59a, 15).

16. 2. The audio generator of claim 1, wherein the audio signal is a voice audio signal.

17. An audio generator (10) configured to generate an audio signal (16) from an input signal (14) and target data (12), said target data (12) representing said audio signal (16); a first processing block (40, 50, 50a-50h) configured to receive first data (15, 59a) derived from said input signal (14) and to output first output data (69), said first output data (69) comprising a plurality of channels (47); a second processing block (45) configured to receive as second data the first output data (69) or data derived from the first output data (69); An audio generator (10) comprising: The first processing block (50) performs, for each channel of the first output data: a conditioning set (71, 72, 73) of a learnable layer configured to process the target data (12) to obtain conditioning characteristic parameters (74, 75), the target data (12) being derived from text; a styling element (77) configured to apply said conditioning characteristic parameters (74, 75) to said first data (15, 59a) or to a normalized version (59, 76') of said first data; Equipped with the second processing block (45) is configured to combine the multiple channels (47) of the second data (69) to obtain the audio signal (16); An audio generator, wherein the target data (12) is upsampled by at least a factor of two.

18. 18. The audio generator of claim 17, wherein the target data (12) is upsampled (70) with non-linear interpolation.

19. 14. The audio generator of claim 13, wherein the first activation function is a leaky rectified linear unit (leaky ReLu) function.

20. An audio generator (10) configured to generate an audio signal (16) from an input signal (14) and target data (12), said target data (12) representing said audio signal (16); a first processing block (40, 50, 50a-50h) configured to receive first data (15, 59a) derived from said input signal (14) and to output first output data (69), said first output data (69) comprising a plurality of channels (47); a second processing block (45) configured to receive as second data the first output data (69) or data derived from the first output data (69); An audio generator (10) comprising: The first processing block (50) performs, for each channel of the first output data: a conditioning set (71, 72, 73) of a learnable layer configured to process the target data (12) to obtain conditioning characteristic parameters (74, 75), the target data (12) being derived from text; a styling element (77) configured to apply said conditioning characteristic parameters (74, 75) to said first data (15, 59a) or to a normalized version (59, 76') of said first data; Equipped with the second processing block (45) is configured to combine the multiple channels (47) of the second data (69) to obtain the audio signal (16); The convolution operations (61a, 61b, 62a, 62b) are performed with a maximum dilation factor of 2. Audio generator.

21. An audio generator (10) configured to generate an audio signal (16) from an input signal (14) and target data (12), said target data (12) representing said audio signal (16); a first processing block (40, 50, 50a-50h) configured to receive first data (15, 59a) derived from said input signal (14) and to output first output data (69), said first output data (69) comprising a plurality of channels (47); a second processing block (45) configured to receive as second data the first output data (69) or data derived from the first output data (69); An audio generator (10) comprising: The first processing block (50) performs, for each channel of the first output data: a conditioning set (71, 72, 73) of a learnable layer configured to process the target data (12) to obtain conditioning characteristic parameters (74, 75), the target data (12) being derived from text; a styling element (77) configured to apply said conditioning characteristic parameters (74, 75) to said first data (15, 59a) or to a normalized version (59, 76') of said first data; Equipped with the second processing block (45) is configured to combine the multiple channels (47) of the second data (69) to obtain the audio signal (16); The audio generator comprises eight first processing blocks (50a to 50h) and one second processing block (45).

22. An audio generator (10) configured to generate an audio signal (16) from an input signal (14) and target data (12), said target data (12) representing said audio signal (16); a first processing block (40, 50, 50a-50h) configured to receive first data (15, 59a) derived from said input signal (14) and to output first output data (69), said first output data (69) comprising a plurality of channels (47); a second processing block (45) configured to receive as second data the first output data (69) or data derived from the first output data (69); An audio generator (10) comprising: The first processing block (50) performs, for each channel of the first output data: a conditioning set (71, 72, 73) of a learnable layer configured to process the target data (12) to obtain conditioning characteristic parameters (74, 75), the target data (12) being derived from text; a styling element (77) configured to apply said conditioning characteristic parameters (74, 75) to said first data (15, 59a) or to a normalized version (59, 76') of said first data; Equipped with the second processing block (45) is configured to combine the multiple channels (47) of the second data (69) to obtain the audio signal (16); An audio generator, wherein the first data (15, 59, 59a, 59b) has a dimension lower than the audio signal.

23. 1. A method for generating an audio signal (16) by an audio generator (10) from an input signal (14) and target data (12), said target data (12) being representative of said audio signal (16) and derived from text, comprising: receiving, by a first processing block (50, 50a-50h), first data (15, 59, 59a, 59b) derived from said input signal (14); For each channel of the first output data (59b, 69), processing the target data (12) by a conditioning set (71, 72, 73) of a learnable layer of the first processing block (50) to obtain conditioning characteristic parameters (74, 75); applying said conditioning characteristic parameters (74, 75) to said first data (15, 59a) or to a normalized version (76') of said first data by a styling element (77) of said first processing block (50); outputting, by said first processing block (50), first output data (69) comprising a plurality of channels (47); receiving as second data the first output data (69) or data derived from the first output data (69) by a second processing block (45); combining, by the second processing block (45), the multiple channels (47) of the second data to obtain the audio signal (16); Including, The target data (12) comprises at least one acoustic feature of a log spectrogram, a MFCC, a mel spectrogram or any other type of spectrogram obtained from text; The method for generating an audio signal includes obtaining the target data (12) by converting (1110) an input in the form of text or text elements into the at least one acoustic characteristic.

24. 1. A method for generating an audio signal (16) by an audio generator (10) from an input signal (14) and target data (12), said target data (12) being representative of said audio signal (16) and derived from text, comprising: receiving, by a first processing block (50, 50a-50h), first data (15, 59, 59a, 59b) derived from said input signal (14); For each channel of the first output data (59b, 69), processing the target data (12) by a conditioning set (71, 72, 73) of a learnable layer of the first processing block (50) to obtain conditioning characteristic parameters (74, 75); applying said conditioning characteristic parameters (74, 75) to said first data (15, 59a) or to a normalized version (76') of said first data by a styling element (77) of said first processing block (50); outputting, by said first processing block (50), first output data (69) comprising a plurality of channels (47); receiving as second data the first output data (69) or data derived from the first output data (69) by a second processing block (45); combining, by the second processing block (45), the multiple channels (47) of the second data to obtain the audio signal (16); Including, The target data (12) comprises at least one acoustic feature of a log spectrogram, a MFCC, a mel spectrogram or any other type of spectrogram obtained from text; The method for generating an audio signal includes obtaining the target data (12) by transforming (1114) at least one linguistic feature into the at least one acoustic feature.

25. 1. A method for generating an audio signal (16) by an audio generator (10) from an input signal (14) and target data (12), said target data (12) being representative of said audio signal (16) and derived from text, comprising: receiving, by a first processing block (50, 50a-50h), first data (15, 59, 59a, 59b) derived from said input signal (14); For each channel of the first output data (59b, 69), processing the target data (12) by a conditioning set (71, 72, 73) of a learnable layer of the first processing block (50) to obtain conditioning characteristic parameters (74, 75); applying said conditioning characteristic parameters (74, 75) to said first data (15, 59a) or to a normalized version (76') of said first data by a styling element (77) of said first processing block (50); outputting, by said first processing block (50), first output data (69) comprising a plurality of channels (47); receiving as second data the first output data (69) or data derived from the first output data (69) by a second processing block (45); combining, by the second processing block (45), the multiple channels (47) of the second data to obtain the audio signal (16); Including, the target data (12) comprises at least one linguistic feature of phonemes, prosody, intonation, sentence breaks, and voiced pauses obtained from a text; 1. A method for generating an audio signal, comprising the step of obtaining said target data (12) by converting (1110) an input in the form of text or text elements into said at least one linguistic characteristic.

26. 1. A method for generating an audio signal (16) by an audio generator (10) from an input signal (14) and target data (12), said target data (12) being representative of said audio signal (16) and derived from text, comprising: receiving, by a first processing block (50, 50a-50h), first data (15, 59, 59a, 59b) derived from said input signal (14); For each channel of the first output data (59b, 69), processing the target data (12) by a conditioning set (71, 72, 73) of a learnable layer of the first processing block (50) to obtain conditioning characteristic parameters (74, 75); applying said conditioning characteristic parameters (74, 75) to said first data (15, 59a) or to a normalized version (76') of said first data by a styling element (77) of said first processing block (50); outputting, by said first processing block (50), first output data (69) comprising a plurality of channels (47); receiving as second data the first output data (69) or data derived from the first output data (69) by a second processing block (45); combining, by the second processing block (45), the multiple channels (47) of the second data to obtain the audio signal (16); Including, 1. A method for generating an audio signal, wherein the target data (12) includes at least one of characters and words taken from a text.

27. 1. A method for generating an audio signal (16) by an audio generator (10) from an input signal (14) and target data (12), said target data (12) being representative of said audio signal (16) and derived from text, comprising: receiving, by a first processing block (50, 50a-50h), first data (15, 59, 59a, 59b) derived from said input signal (14); For each channel of the first output data (59b, 69), processing the target data (12) by a conditioning set (71, 72, 73) of a learnable layer of the first processing block (50) to obtain conditioning characteristic parameters (74, 75); applying said conditioning characteristic parameters (74, 75) to said first data (15, 59a) or to a normalized version (76') of said first data by a styling element (77) of said first processing block (50); outputting, by said first processing block (50), first output data (69) comprising a plurality of channels (47); receiving as second data the first output data (69) or data derived from the first output data (69) by a second processing block (45); combining, by the second processing block (45), the multiple channels (47) of the second data to obtain the audio signal (16); Including, The method for generating an audio signal further comprises the step of performing text analysis and / or using a statistical model that uses an acoustic model to derive target data (12).

28. 1. A method for generating an audio signal (16) by an audio generator (10) from an input signal (14) and target data (12), said target data (12) being representative of said audio signal (16) and derived from text, comprising: receiving, by a first processing block (50, 50a-50h), first data (15, 59, 59a, 59b) derived from said input signal (14); For each channel of the first output data (59b, 69), processing the target data (12) by a conditioning set (71, 72, 73) of a learnable layer of the first processing block (50) to obtain conditioning characteristic parameters (74, 75); applying said conditioning characteristic parameters (74, 75) to said first data (15, 59a) or to a normalized version (76') of said first data by a styling element (77) of said first processing block (50); outputting, by said first processing block (50), first output data (69) comprising a plurality of channels (47); receiving as second data the first output data (69) or data derived from the first output data (69) by a second processing block (45); combining, by the second processing block (45), the multiple channels (47) of the second data to obtain the audio signal (16); Including, The method for generating an audio signal further comprises the step of deriving target data (12) using a trainable model that performs text analysis and / or uses an acoustic model.

29. 1. A method for generating an audio signal (16) by an audio generator (10) from an input signal (14) and target data (12), said target data (12) being representative of said audio signal (16) and derived from text, comprising: receiving, by a first processing block (50, 50a-50h), first data (15, 59, 59a, 59b) derived from said input signal (14); For each channel of the first output data (59b, 69), processing the target data (12) by a conditioning set (71, 72, 73) of a learnable layer of the first processing block (50) to obtain conditioning characteristic parameters (74, 75); applying said conditioning characteristic parameters (74, 75) to said first data (15, 59a) or to a normalized version (76') of said first data by a styling element (77) of said first processing block (50); outputting, by said first processing block (50), first output data (69) comprising a plurality of channels (47); receiving as second data the first output data (69) or data derived from the first output data (69) by a second processing block (45); combining, by the second processing block (45), the multiple channels (47) of the second data to obtain the audio signal (16); Including, The method for generating an audio signal further comprises deriving target data (12) using a rule-based algorithm for text analysis and / or an acoustic model.

30. 1. A method for generating an audio signal (16) by an audio generator (10) from an input signal (14) and target data (12), said target data (12) being representative of said audio signal (16) and derived from text, comprising: receiving, by a first processing block (50, 50a-50h), first data (15, 59, 59a, 59b) derived from said input signal (14); For each channel of the first output data (59b, 69), processing the target data (12) by a conditioning set (71, 72, 73) of a learnable layer of the first processing block (50) to obtain conditioning characteristic parameters (74, 75); applying said conditioning characteristic parameters (74, 75) to said first data (15, 59a) or to a normalized version (76') of said first data by a styling element (77) of said first processing block (50); outputting, by said first processing block (50), first output data (69) comprising a plurality of channels (47); receiving as second data the first output data (69) or data derived from the first output data (69) by a second processing block (45); combining, by the second processing block (45), the multiple channels (47) of the second data to obtain the audio signal (16); Including, The method for generating an audio signal, the method further comprising deriving the target data (12) through at least one deterministic layer.

31. 1. A method for generating an audio signal (16) by an audio generator (10) from an input signal (14) and target data (12), said target data (12) being representative of said audio signal (16) and derived from text, comprising: receiving, by a first processing block (50, 50a-50h), first data (15, 59, 59a, 59b) derived from said input signal (14); For each channel of the first output data (59b, 69), processing the target data (12) by a conditioning set (71, 72, 73) of a learnable layer of the first processing block (50) to obtain conditioning characteristic parameters (74, 75); applying said conditioning characteristic parameters (74, 75) to said first data (15, 59a) or to a normalized version (76') of said first data by a styling element (77) of said first processing block (50); outputting, by said first processing block (50), first output data (69) comprising a plurality of channels (47); receiving as second data the first output data (69) or data derived from the first output data (69) by a second processing block (45); combining, by the second processing block (45), the multiple channels (47) of the second data to obtain the audio signal (16); Including, The method for generating an audio signal, the method further comprising deriving the target data (12) through at least one learnable layer.

32. 1. A method for generating an audio signal (16) by an audio generator (10) from an input signal (14) and target data (12), said target data (12) being representative of said audio signal (16) and derived from text, comprising: receiving, by a first processing block (50, 50a-50h), first data (15, 59, 59a, 59b) derived from said input signal (14); For each channel of the first output data (59b, 69), processing the target data (12) by a conditioning set (71, 72, 73) of a learnable layer of the first processing block (50) to obtain conditioning characteristic parameters (74, 75); applying said conditioning characteristic parameters (74, 75) to said first data (15, 59a) or to a normalized version (76') of said first data by a styling element (77) of said first processing block (50); outputting, by said first processing block (50), first output data (69) comprising a plurality of channels (47); receiving as second data the first output data (69) or data derived from the first output data (69) by a second processing block (45); combining, by the second processing block (45), the multiple channels (47) of the second data to obtain the audio signal (16); Including, 1. A method for generating an audio signal, wherein the conditioning set of learnable layers (71-73) consists of one or two convolutional layers.

33. 1. A method for generating an audio signal (16) by an audio generator (10) from an input signal (14) and target data (12), said target data (12) being representative of said audio signal (16) and derived from text, comprising: receiving, by a first processing block (50, 50a-50h), first data (15, 59, 59a, 59b) derived from said input signal (14); For each channel of the first output data (59b, 69), processing the target data (12) by a conditioning set (71, 72, 73) of a learnable layer of the first processing block (50) to obtain conditioning characteristic parameters (74, 75); applying said conditioning characteristic parameters (74, 75) to said first data (15, 59a) or to a normalized version (76') of said first data by a styling element (77) of said first processing block (50); outputting, by said first processing block (50), first output data (69) comprising a plurality of channels (47); receiving as second data the first output data (69) or data derived from the first output data (69) by a second processing block (45); combining, by the second processing block (45), the multiple channels (47) of the second data to obtain the audio signal (16); Including, The method for generating an audio signal, wherein the step of processing by a conditioning set (71-73) of the learnable layer includes a step of convolving the target data (12) or an upsampled version of the target data by a first convolutional layer (71) using a first activation function to obtain first convolved data (71').

34. 1. A method for generating an audio signal (16) by an audio generator (10) from an input signal (14) and target data (12), said target data (12) being representative of said audio signal (16) and derived from text, comprising: receiving, by a first processing block (50, 50a-50h), first data (15, 59, 59a, 59b) derived from said input signal (14); For each channel of the first output data (59b, 69), processing the target data (12) by a conditioning set (71, 72, 73) of a learnable layer of the first processing block (50) to obtain conditioning characteristic parameters (74, 75); applying said conditioning characteristic parameters (74, 75) to said first data (15, 59a) or to a normalized version (76') of said first data by a styling element (77) of said first processing block (50); outputting, by said first processing block (50), first output data (69) comprising a plurality of channels (47); receiving as second data the first output data (69) or data derived from the first output data (69) by a second processing block (45); combining, by the second processing block (45), the multiple channels (47) of the second data to obtain the audio signal (16); Including, 1. A method for generating an audio signal, wherein the conditioning sets (71-73) of the learnable layer and the stylization elements (77) are part of a weighting layer within a residual block (50, 50a-50h) of a neural network comprising one or more residual blocks (50, 50a-50h).

35. 1. A method for generating an audio signal (16) by an audio generator (10) from an input signal (14) and target data (12), said target data (12) being representative of said audio signal (16) and derived from text, comprising: receiving, by a first processing block (50, 50a-50h), first data (15, 59, 59a, 59b) derived from said input signal (14); For each channel of the first output data (59b, 69), processing the target data (12) by a conditioning set (71, 72, 73) of a learnable layer of the first processing block (50) to obtain conditioning characteristic parameters (74, 75); applying said conditioning characteristic parameters (74, 75) to said first data (15, 59a) or to a normalized version (76') of said first data by a styling element (77) of said first processing block (50); outputting, by said first processing block (50), first output data (69) comprising a plurality of channels (47); receiving as second data the first output data (69) or data derived from the first output data (69) by a second processing block (45); combining, by the second processing block (45), the multiple channels (47) of the second data to obtain the audio signal (16); Including, The method for generating an audio signal, the method further comprising the step of normalizing the first data (15, 59) by a normalization element (76).

36. 24. The method for generating an audio signal according to claim 23, wherein the audio signal (16) is a voice audio signal.

37. 1. A method for generating an audio signal (16) by an audio generator (10) from an input signal (14) and target data (12), said target data (12) being representative of said audio signal (16) and derived from text, comprising: receiving, by a first processing block (50, 50a-50h), first data (15, 59, 59a, 59b) derived from said input signal (14); For each channel of the first output data (59b, 69), processing the target data (12) by a conditioning set (71, 72, 73) of a learnable layer of the first processing block (50) to obtain conditioning characteristic parameters (74, 75); applying said conditioning characteristic parameters (74, 75) to said first data (15, 59a) or to a normalized version (76') of said first data by a styling element (77) of said first processing block (50); outputting, by said first processing block (50), first output data (69) comprising a plurality of channels (47); receiving as second data the first output data (69) or data derived from the first output data (69) by a second processing block (45); combining, by the second processing block (45), the multiple channels (47) of the second data to obtain the audio signal (16); Including, A method for generating an audio signal, wherein the target data (12) is upsampled (70) by a factor of two.

38. 1. A method for generating an audio signal (16) by an audio generator (10) from an input signal (14) and target data (12), said target data (12) being representative of said audio signal (16) and derived from text, comprising: receiving, by a first processing block (50, 50a-50h), first data (15, 59, 59a, 59b) derived from said input signal (14); For each channel of the first output data (59b, 69), processing the target data (12) by a conditioning set (71, 72, 73) of a learnable layer of the first processing block (50) to obtain conditioning characteristic parameters (74, 75); applying said conditioning characteristic parameters (74, 75) to said first data (15, 59a) or to a normalized version (76') of said first data by a styling element (77) of said first processing block (50); outputting, by said first processing block (50), first output data (69) comprising a plurality of channels (47); receiving as second data the first output data (69) or data derived from the first output data (69) by a second processing block (45); combining, by the second processing block (45), the multiple channels (47) of the second data to obtain the audio signal (16); Including, A method for generating an audio signal, wherein the target data (12) is upsampled (70) with non-linear interpolation.

39. 34. The method for generating an audio signal according to claim 33, wherein the first activation function is a leaky rectified linear unit (leaky ReLu) function.

40. 1. A method for generating an audio signal (16) by an audio generator (10) from an input signal (14) and target data (12), said target data (12) being representative of said audio signal (16) and derived from text, comprising: receiving, by a first processing block (50, 50a-50h), first data (15, 59, 59a, 59b) derived from said input signal (14); For each channel of the first output data (59b, 69), processing the target data (12) by a conditioning set (71, 72, 73) of a learnable layer of the first processing block (50) to obtain conditioning characteristic parameters (74, 75); applying said conditioning characteristic parameters (74, 75) to said first data (15, 59a) or to a normalized version (76') of said first data by a styling element (77) of said first processing block (50); outputting, by said first processing block (50), first output data (69) comprising a plurality of channels (47); receiving as second data the first output data (69) or data derived from the first output data (69) by a second processing block (45); combining, by the second processing block (45), the multiple channels (47) of the second data to obtain the audio signal (16); Including, A method for generating an audio signal, wherein the convolution operations (61a, 61b, 62a, 62b) are performed with a maximum expansion factor of 2.

41. 1. A method for generating an audio signal (16) by an audio generator (10) from an input signal (14) and target data (12), said target data (12) being representative of said audio signal (16) and derived from text, comprising: receiving, by a first processing block (50, 50a-50h), first data (15, 59, 59a, 59b) derived from said input signal (14); For each channel of the first output data (59b, 69), processing the target data (12) by a conditioning set (71, 72, 73) of a learnable layer of the first processing block (50) to obtain conditioning characteristic parameters (74, 75); applying said conditioning characteristic parameters (74, 75) to said first data (15, 59a) or to a normalized version (76') of said first data by a styling element (77) of said first processing block (50); outputting, by said first processing block (50), first output data (69) comprising a plurality of channels (47); receiving as second data the first output data (69) or data derived from the first output data (69) by a second processing block (45); combining, by the second processing block (45), the multiple channels (47) of the second data to obtain the audio signal (16); Including, A method for generating an audio signal, comprising the steps of repeating the steps of the first processing block (50, 50a to 50h) eight times and performing the steps of the second processing block (45) only once.

42. 24. A method for generating an audio signal as claimed in claim 23, wherein the first data (15, 59) has a lower dimension than the audio signal.

43. 1. A method for generating an audio signal (16) by an audio generator (10) from an input signal (14) and target data (12), said target data (12) being representative of said audio signal (16) and derived from text, comprising: receiving, by a first processing block (50, 50a-50h), first data (15, 59, 59a, 59b) derived from said input signal (14); For each channel of the first output data (59b, 69), processing the target data (12) by a conditioning set (71, 72, 73) of a learnable layer of the first processing block (50) to obtain conditioning characteristic parameters (74, 75); applying said conditioning characteristic parameters (74, 75) to said first data (15, 59a) or to a normalized version (76') of said first data by a styling element (77) of said first processing block (50); outputting, by said first processing block (50), first output data (69) comprising a plurality of channels (47); receiving as second data the first output data (69) or data derived from the first output data (69) by a second processing block (45); combining, by the second processing block (45), the multiple channels (47) of the second data to obtain the audio signal (16); Including, The method for generating an audio signal, further comprising the step of deriving the target data from text.

44. 24. The method for generating an audio signal according to claim 23, wherein the target data (12) is a spectrogram.

45. 45. The method of claim 44, wherein the spectrogram is a mel-spectrogram.

Images