Method, device and system for hybrid speech synthesis

Abstract

The invention relates to a method, device and system for mixed speech synthesis. The method comprises: (a) receiving quantized original linear predictive coding parameters estimated by applying linear predictive coding analysis filtering to an original speech signal and a quantized compressed representation of residuals of the original speech signal; (b) de-quantizing the raw linear predictive coding parameter and the compressed representation of the residual; (c) inputting the dequantized compressed representation of the residual to a decoder portion of a generator to apply an adversarial mapping from a compressed residual domain to a pseudo (first) signal domain; (d) outputting a pseudo-speech signal by the decoder portion of the generator; (e) applying linear predictive coding analysis filtering to the pseudo voice signal to obtain a corresponding pseudo residual error; (f) reconstructing the original speech signal by applying linear predictive coding cross-synthesis filtering to the pseudo residual and the de-quantized original linear predictive coding analysis parameters.

Description

[0001] Information about divisional applications

[0002] This case is a divisional application. The parent case of the divisional application is an invention patent application with application date of December 20, 2019, application number 201980087069.3, and invention name “Method, device and system for hybrid speech synthesis”.

[0003] CROSS-REFERENCE TO RELATED APPLICATIONS

[0004] This application claims priority to the following priority applications: U.S. Provisional Application No. 62 / 787,831 filed on January 3, 2019 (reference: D18127USP1) and EP Application No. 19150154.3 filed on January 3, 2019 (reference: D18127EP), which are hereby incorporated by reference. Technical Field

[0005] The present disclosure generally relates to methods, apparatus, and systems for encoding and decoding raw speech signals for hybrid adversarial parameter speech synthesis, and more specifically, to improving the synthesis of raw speech signals using compact learned parameter representations by implementing a generator trained in a generative adversarial network setting in conjunction with linear predictive coding.

[0006] Although some embodiments will be described herein with particular reference to said disclosure, it will be appreciated that the disclosure is not limited to such fields of use and is applicable in a much broader context. Background Art

[0007] Any discussion of the background art throughout the present disclosure should in no way be considered as an admission that such art is widely known or forms part of the common general knowledge in the field.

[0008] Speech is an acoustic signal generated by the human vocal system that conveys the speaker's intention in the form of language and emotional information. In digital systems, speech signals are represented as waveforms that describe the time-dependent progression of the amplitude of the corresponding speech signal.

[0009] Challenges arise when speech signals are to be transmitted using digital communication systems. Especially for communication channels with limited bandwidth (e.g., mobile phone networks), efficient representation of speech signal waveforms is important. Reliable signal representation requires high sampling rates. However, transmitting the original speech signal waveform at a high sampling rate results in high bit rates and power consumption, thereby violating channel bandwidth conservation. In order to save channel bandwidth, speech signal compression can transmit compact representations of the corresponding speech signal waveforms. These compact representations are usually sufficient for reliable speech signal waveform reconstruction.

[0010] In this context, a successful approach used so far is the model-based representation of speech signals, which is able to describe the speech signal waveform in terms of model parameters. The source-filter model is a well-known approach in speech modeling, which utilizes the creation of a glottal excitation signal (source component) and the spectral shaping of the glottal excitation signal (filter component).

[0011] A speech codec can consist of two parts: an encoder that decomposes the speech signal into its glottal excitation plus its spectral envelope, and a decoder that reconstructs the speech signal back again. In this context, the encoder can perform a linear predictive coding analysis task to create the corresponding components of the source filter model, and the decoder can perform a corresponding linear predictive coding synthesis task by reconstructing the speech signal.

[0012] However, the overall goal of signal compression is to find a compact representation that can encode the speech signal with less data usage and allow reliable and fast reconstruction.

[0013] Recently, Generative Adversarial Networks (GANs) have gained increasing attention due to their improved reliability when applied to tasks such as inclusion-conditional image synthesis, image-to-image translation, image style transfer, image super-resolution, image rendering, text-to-image synthesis, video generation, etc. After being applied to speech enhancement, GANs have also gained increasing attention in the field of speech and audio signal processing.

[0014] For example, L. Juvela, B. Bollepalli, X. Wang, H. Kameoka, M. Airaksinen, J. Yamagishi, and P. Alku proposed a method for speech reconstruction from filter bank mel-frequency cepstral coefficients (MFCC) in their publication "Speech waveform synthesis from MFCC sequences with Generative Adversarial Networks" (IEEE ICASSP, Calgary, AB, 2018, pp. 5679-5683).

[0015] S. Kankanahalli proposed a proof-of-concept for applying deep neural networks (DNNs) to speech coding in the publication “End-to-End optimized speech coding with deep neural networks” (IEEE ICASSP, Calgary, AB, 2018, pp. 2521-2525). The wideband speech encoder in this publication is learned end-to-end from an initial signal with little audio-specific processing other than a relatively simple perceptual loss.

[0016] Furthermore, L. Juvela, V. Tsiaras, B. Bollepalli, M. Airaksinen, J. Yamagishi and P. Alku in their publication “Speaker-independent raw waveform model for glottal excitation” (Proceedings of Interspeech 2018, pp. 2012–2016) proposed a speaker-independent neural waveform generator that combines a linear autoregressive (vocal tract filter) process with a nonlinear (glottal source) excitation process, parameterized by a WaveNet.

[0017] Despite ongoing research, a challenging limitation of deep generative models for parametric speech synthesis is that the generation process is very slow. Current deep generative models for parametric speech synthesis typically work in an autoregressive sequential manner, where the signal is generated sequentially in a sample-by-sample manner.

[0018] Therefore, there remains a need for efficient signal compression of speech signals while allowing reliable and fast reconstruction (especially at lower bit rates). Summary of the invention

[0019] According to a first aspect of the present disclosure, a method for encoding an original speech signal for use in hybrid adversarial parameter speech synthesis is provided. The method may include (a) the step of receiving the original speech signal. The method may further include (b) the step of applying a linear prediction coding analysis filter to the original speech signal to obtain a corresponding residual. The method may further include (c) the step of inputting the obtained residual into the encoder part of the generator to encode the residual. The method may further include (d) the step of outputting a compressed representation of the residual by the encoder part of the generator. The method may further include (e) the step of applying a linear prediction coding analysis filter to the original speech signal to estimate the original linear prediction coding parameters. And the method may further include (f) the step of quantizing and transmitting the original linear prediction coding parameters and the compressed representation of the residual.

[0020] In one embodiment, the order of the linear prediction coding analysis filter in step (e) may be equal to or higher than the order in step (b).

[0021] In one embodiment, the order of the linear prediction coding analysis filter in step (b) may be 16.

[0022] In one embodiment, the order of the linear prediction coding analysis filter used in step (e) may be from 16 to 50.

[0023] In one embodiment, the generator may be a generator trained in a generative adversarial network setting.

[0024] In one embodiment, the generative adversarial network setting may include one or more of a geometric setting, a Wasserstein setting, and an energy-based setting.

[0025] In one embodiment, the encoder portion of the generator may include L layers with N filters in each layer, where L is a natural number ≥1, and where N is a natural number ≥1.

[0026] In one embodiment, in at least one layer of the L layers, a 1D convolution operation may be performed, followed by a nonlinear operation, wherein the nonlinear operation includes a parametric rectified linear unit (PReLU), a rectified linear unit (ReLU), a leaky rectified linear unit (LReLU), an exponential linear unit (eLU), and a scaled exponential linear unit (SeLU).

[0027] In one embodiment, the N filters in each of the L layers may be the same size.

[0028] In one embodiment, the N filters in each of the L layers may operate with a stride of 2.

[0029] In one embodiment, an output layer may subsequently follow the last of the L layers of the encoder portion of the generator.

[0030] In one embodiment, the output layer may include N filters operating with a stride of 1.

[0031] In one embodiment, a 1D convolution operation may be performed in the output layer, followed by a nonlinear operation including a parametric rectified linear unit (PReLU), a rectified linear unit (ReLU), a leaky rectified linear unit (LReLU), an exponential linear unit (eLU), and a scaled exponential linear unit (SeLU).

[0032] According to a second aspect of the present disclosure, a method for decoding an original speech signal for hybrid adversarial parameter speech synthesis is provided. The method may include (a) receiving a quantized compressed representation of the original linear prediction coding parameters and the residual of the original speech signal estimated by applying a linear prediction coding analysis filter to the original speech signal. The method may further include (b) dequantizing the compressed representation of the original linear prediction coding parameters and the residual. The method may further include (c) inputting the dequantized compressed representation of the residual into the decoder part of the generator to apply an adversarial mapping from the compressed residual domain to the pseudo (first) signal domain. The method may further include (d) outputting a pseudo speech signal by the decoder part of the generator. The method may further include (e) applying a linear prediction coding analysis filter to the pseudo speech signal to obtain a corresponding pseudo residual. And the method may further include (f) reconstructing the original speech signal by applying a linear prediction coding cross synthesis filter to the pseudo residual and the dequantized original linear prediction coding analysis parameters.

[0033] In one embodiment, the order of the linear prediction coding analysis filter used in step (e) may be the same as the order used to estimate the original linear prediction coding parameters.

[0034] In one embodiment, the order of the linear prediction coding analysis filter used in step (e) may be from 16 to 50.

[0035] In one embodiment, the generator may be a generator trained in a generative adversarial network setting.

[0036] In one embodiment, the generative adversarial network setting may include one or more of a geometric setting, a Wasserstein setting, and an energy-based setting.

[0037] In one embodiment, the decoder portion of the generator may include an adversarial generation section.

[0038] In one embodiment, the adversarial generation section may include L layers, wherein there are N filters in each layer, wherein L is a natural number ≥1, and wherein N is a natural number ≥1.

[0039] In one embodiment, in at least one of the L layers of the adversarial generation section, a transposed convolution may be performed followed by a gated tanh unit.

[0040] In one embodiment, the N filters in each of the L layers of the adversarial generation section may be the same size.

[0041] In one embodiment, the N filters in each of the L layers of the adversarial generation section may operate with a stride of 2.

[0042] In one embodiment, an output layer may then follow the last of the L layers of the adversarial generation section.

[0043] In one embodiment, the output layer may include N filters operating with a stride of 1.

[0044] In one embodiment, a 1D convolution operation may be performed in the output layer, followed by a tanh operation.

[0045] In one embodiment, the decoder portion of the generator may further include a context decoding section prior to the adversarial generation section.

[0046] In one embodiment, the context decoding section may include an L=1 layer with N filters (where N is a natural number ≥ 1), followed by one or more blocks of gated tanh units.

[0047] In one embodiment, in the L=1 layer of the context decoding section, the size of the N filters may be 1, and a 1D convolution operation may be performed.

[0048] In one embodiment, the N filters in the L=1 layer of the context decoding section may operate with a stride of 1.

[0049] In one embodiment, the outputs of the one or more blocks of gated tanh units of the context decoding section may be concatenated with a random noise vector (z).

[0050] In one embodiment, the context decoding section may include 10 blocks of gated tanh units.

[0051] According to a third aspect of the present disclosure, a device for encoding an original speech signal for hybrid adversarial parameter speech synthesis is provided. The device may include (a) a receiver for receiving the original speech signal. The device may further include (b) a linear prediction coding analysis filter for applying a linear prediction coding analysis filter to the original speech signal to obtain a corresponding residual. The device may further include (c) an encoder portion of a generator, which is configured to receive the obtained residual at the input of the encoder portion and output a compressed representation of the residual at the output of the encoder portion for encoding the residual. The device may further include (d) a linear prediction coding analysis filter for applying a linear prediction coding analysis filter to the original speech signal to estimate original linear prediction coding parameters. And the device may further include (e) a component for quantizing and transmitting the original linear prediction coding parameters and the compressed representation of the residual.

[0052] According to a fourth aspect of the present disclosure, a device for decoding an original speech signal for hybrid adversarial parameter speech synthesis is provided. The device may include (a) a receiver for receiving a quantized compressed representation of the residual of the original speech signal and the quantized original linear prediction coding parameters estimated by applying a linear prediction coding analysis filter to the original speech signal. The device may further include (b) a component for dequantizing the compressed representation of the original linear prediction coding parameters and the residual. The device may further include (c) a decoder portion of a generator for generating a pseudo speech signal. The device may further include (d) a linear prediction analysis filter for applying a linear prediction coding analysis filter to the pseudo speech signal to obtain a corresponding pseudo residual. And the device may further include (e) a linear prediction coding synthesis filter for reconstructing the original speech signal by applying a linear prediction coding cross synthesis filter to the pseudo residual and the dequantized original linear prediction coding analysis parameter.

[0053] According to a fifth aspect of the present disclosure, a system is provided having a device for encoding an original speech signal for hybrid adversarial parameter speech synthesis (wherein the device is configured to execute a method for encoding an original speech signal for hybrid adversarial parameter speech synthesis) and a device for decoding an original speech signal for hybrid adversarial parameter speech synthesis (wherein the device is configured to execute a method for decoding an original speech signal for hybrid adversarial parameter speech synthesis).

[0054] According to a sixth aspect of the present disclosure, a method for training a generator in a generative adversarial network setting including a generator (which includes an encoder part and a decoder part) and a discriminator is provided. The method may include (a) the step of inputting a compressed representation of the residual of an original speech signal into the generator. The method may further include (b) the step of generating a pseudo speech signal by the generator based on the compressed representation of the residual. The method may further include (c) the step of inputting the pseudo speech signal and the compressed residual of the original speech signal from which the pseudo speech signal is generated into the discriminator one at a time. The method may further include (d) the step of determining by the discriminator whether the pseudo speech signal corresponds to the compressed residual of the original speech signal or to an undefined compressed residual. And the method may further include (e) the step of tuning the parameters of the generator until the discriminator can no longer distinguish whether the pseudo speech signal corresponds to the compressed residual of the original speech signal or to the undefined compressed residual.

[0055] In one embodiment, the generative adversarial network setting may include one or more of a geometric setting, a Wasserstein setting, and an energy-based setting.

[0056] In one embodiment, the determination made by the discriminator may be based on one or more loss functions.

[0057] In one embodiment, the discriminator may include an encoder stage, and the encoder stage may include L layers, with N filters in each layer, where L is a natural number ≥1, and where N is a natural number ≥1.

[0058] In one embodiment, in at least one of the L layers, a 1D convolution operation may be performed, followed by a nonlinear operation including a leaky rectified linear unit (LReLU).

[0059] In one embodiment, the N filters in each of the L layers may be the same size.

[0060] In one embodiment, the N filters in each of the L layers may operate with a stride of 2.

[0061] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising a computer-readable storage medium having instructions suitable for causing a device to execute a method of encoding an original speech signal for hybrid adversarial parameter speech synthesis.

[0062] According to an eighth aspect of the present disclosure, a computer program product is provided, comprising a computer-readable storage medium having instructions suitable for causing a device to execute a method of decoding an original speech signal for hybrid adversarial parameter speech synthesis.

[0063] According to a ninth aspect of the present disclosure, a computer program product is provided, comprising a computer-readable storage medium having instructions suitable for causing a device to execute a method for training a generator in a generative adversarial network setting comprising a generator (which includes an encoder part and a decoder part) and a discriminator. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Example embodiments of the present disclosure will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0065] Figure 1 A flow chart illustrating an example of a method of encoding an original speech signal for hybrid adversarial parametric speech synthesis.

[0066] Figure 2 An example of a method for decoding an original speech signal for hybrid adversarial parameter speech synthesis is described.

[0067] Figure 3 An example of a generator trained in a generative adversarial network setting is described, where the decoder portion includes a context decoding section and an adversarial generation section.

[0068] Figure 4 An example illustrating the architecture of the encoder portion of a generator trained in a generative adversarial network setting.

[0069] Figure 5 An example of a context decoding segment architecture is illustrated.

[0070] Figure 6 Example illustrating the operation of a softmax-gated tanh unit.

[0071] Figure 7 An example illustrating the adversarial generation segment architecture.

[0072] Figure 8 Example illustrating the discriminator architecture. DETAILED DESCRIPTION

[0073] Hybrid Adversarial Parameter Speech Synthesis

[0074] A novel method for deep neural vocoding with fast signal generation is described. This method utilizes a combination of linear predictive coding (LPC) and adversarial generation of speech signals based on a generative adversarial network (GAN) setting. Using GAN as a deep probabilistic generative modeling framework, a pseudo-speech signal can be synthesized from a highly neutral compressed representation of the residual of the original speech signal. In particular, LPC analysis can be used to determine the spectral envelope of the corresponding original signal and the residual of the synthesized pseudo-speech signal. The final natural reconstruction of the original speech signal can then be obtained using the LPC synthesis between the spectral envelope of the corresponding original signal and the residual of the synthesized pseudo-speech signal.

[0075] Hybrid adversarial parameter speech synthesis is particularly capable of fast signal generation. Advantageously, the signal generation process can be performed in parallel rather than sequentially. Thus, the generated signal can be obtained at once rather than in a sample-by-sample manner.

[0076] Overview

[0077] refer to Figure 1 The example of the invention is used to illustrate the method of encoding the original speech signal for hybrid adversarial parameter speech synthesis. In step 101, the original speech signal 1 is received. In step 102, the linear prediction coding analysis filter 2 is used to apply the linear prediction coding analysis filtering to the original speech signal 1 to obtain the corresponding residual of the original speech signal 1. In step 103, the residual of the original speech signal 1 is then input into the encoder part 4 of the generator. The details of the generator will be further described below. The encoder part 4 of the generator encodes the residual of the original speech signal 1. In step 104, a compressed representation of the residual is obtained as an output from the encoder part 4 of the generator. Although compression is not limited, at a sampling rate of 16kHz, the compressed representation of the residual can be 20 parameters per frame of 320 samples of the residual. In step 105, the linear prediction coding analysis filtering is applied to the original speech signal 1 again using the linear prediction coding analysis filter 3 to estimate the linear prediction coding parameters (i.e., the original linear prediction coding parameters) of the original speech signal 1 as the output in step 106.

[0078] Although the orders of the linear prediction coding analysis filters 2 and 3 used to obtain the corresponding residual of the original speech signal 1 and to estimate the original linear prediction coding parameters are not limited, in one embodiment, the order of the linear prediction coding filter 3 used to estimate the original linear prediction coding parameters may be equal to or higher than the order of the linear prediction coding filter 2 used to obtain the residual of the original speech signal 1. In one embodiment, the order of the linear prediction coding filter 2 used to obtain the residual of the original speech signal 1 may be 16. In one embodiment, the order of the linear prediction coding analysis filter 3 used to estimate the original linear prediction coding parameters may be from 16 to 50. Although the order of the linear prediction coding filter 3 used to estimate the original linear prediction coding parameters is generally arbitrary, increasing the number of estimated original linear prediction coding parameters may result in better reconstruction of the signal at the decoder. Although further, the linear prediction coding analysis and synthesis configurations are not limited, these configurations may include optimizations, including one or more of perceptual weight filtering, frequency distortion, and bandwidth extension.

[0079] In step 107 , the compressed representation of the residual obtained in step 104 and the original linear prediction coding parameters estimated in step 106 are then quantized and transmitted.

[0080] Reference now Figure 2 The example of decoding the original speech signal for hybrid adversarial parameter speech synthesis is described. In step 107, the original linear prediction coding parameters estimated by applying the linear prediction coding analysis filter (step 105) to the original signal 1 and the compressed representation of the residual of the same original signal 1 are received. In step 108, the quantized original linear prediction coding parameters are dequantized to reconstruct the original speech signal 1. In step 109, the quantized compressed representation of the residual of the original signal 1 is also dequantized and input into the decoder part 5 of the generator.

[0081] In one embodiment, the decoder portion 5 of the generator may include an adversarial generation section 10. In one embodiment, the decoder portion 5 of the generator may further include a context decoding section 9 before the adversarial generation section 10, such as Figure 3 The decoder part 5 of the generator applies an adversarial generative mapping from the compressed residual domain to the pseudo signal domain. The pseudo signal domain can also be denoted as the first signal domain.

[0082] A pseudo speech signal is obtained as an output from the decoder part 5 of the generator. In step 110, linear prediction coding analysis filtering is applied to the pseudo speech signal using a linear prediction coding analysis filter 6 to obtain a pseudo residual of the pseudo speech signal in step 111. The order of the applied linear prediction coding analysis filter 6 may be the same as the order of the linear prediction coding analysis filter 3 applied to estimate the original linear prediction coding parameters in the encoder. In one embodiment, the order of the linear prediction coding analysis filter 6 applied in step 110 may be from 16 to 50. Then, linear prediction coding cross-synthesis filtering is applied to the dequantized original linear prediction coding parameters obtained in step 108 and the pseudo residual of the pseudo speech signal obtained in step 111 using a linear prediction coding synthesis filter 7 to obtain a reconstructed original speech signal 8 in step 112.

[0083] The above-described method of encoding and decoding the original speech signal for hybrid adversarial parameter speech synthesis can be implemented on a corresponding device (e.g., a corresponding encoder and decoder) for encoding and decoding the original speech signal for hybrid adversarial parameter speech synthesis. Each of the encoder and decoder can be part of a separate device. The encoder and decoder combination can also form a system. The system can also be implemented in a single device.

[0084] Generator

[0085] Reference now Figure 3 The example of FIG. 1 schematically illustrates a generator trained in a generative adversarial network setting. The generator includes an encoder part 4 and a decoder part 5. Although the decoder part 5 may only include the adversarial generation section 10, Figure 3 In the illustrated example, the decoder portion 5 further comprises a context decoding section 9 prior to the adversarial generation section 10 .

[0086] In step 201, the residual of the original speech signal is input to the encoder part 4 of the generator, where the residual is encoded. Then, in step 202, the compressed representation of the residual is input to the decoder part 5 of the generator.

[0087] As stated above, the decoder part 5 of the generator may include a context decoding section 9 before the adversarial generation section 10. This allows the context decoding to be concatenated with the adversarial generation. Thus, the compressed representation of the residual may first enter the context decoding section 9. Then, in step 203, the output of the context decoding section 9 may enter the adversarial generation section 10.

[0088] In the decoder part 5 of the generator, a pseudo speech signal is generated based on the compressed representation of the residual by applying an adversarial mapping from the compressed residual domain to the pseudo (first) signal domain. In step 204, the pseudo speech signal generated by the decoder part 5 of the generator is then obtained as an output from the adversarial generation section 10.

[0089] The respective architectures of the encoder part 4 and the decoder part 5 will now be described in more detail.

[0090] refer to Figure 4 An example of schematically illustrating the layered architecture of the encoder part 4 of the generator.

[0091] In general, the task of the encoder part of the generator is to learn an extremely compressed parameter representation of the residual obtained by applying linear prediction coding analysis filtering to the original speech signal. This compressed parameter representation obtained in the encoding method is input (i.e., used as a conditional prior) for the decoder part of the generator.

[0092] Although the architecture of the encoder part 4 of the generator is not limited, in one embodiment, the encoder part 4 may include L layers, wherein there are N filters in each layer L. L may be a natural number ≥1, and N may be a natural number ≥1. The size of the N filters (also referred to as the kernel size) is not limited and may be selected according to the requirement of encoding the residual of the original speech signal. In one embodiment, in each of the L layers, the size of the N filters may be the same. In one embodiment, the N filters in each of the L layers may operate with a step size of 2.

[0093] exist Figure 4 In the example embodiment of , the residual of the original speech signal may be input to the encoder portion of the generator in a first step 11. The first layer (layer number L=1) 12 of the illustrated encoder layers may include N filters operating with a step size of 2 and a filter size (or kernel size) of 64. The second encoder layer 13 (layer number L=2) illustrated may also include N filters operating with a step size of 2 and a filter size of 64. Although the encoder portion of the generator is not limited with respect to the number of layers L, in Figure 4 In the example embodiment of , the encoder part of the generator comprises a total of 4 layers 12, 13, 14, 15, wherein each of the 4 layers may comprise N filters having a filter size of 64 and operating with a stride of 2. Alternatively, in each of the 4 encoder layers 12, 13, 14, 15, the N filters may operate with a stride of 1 and then be downsampled by a factor of 2.

[0094] In one embodiment, in at least one of the encoder layers, a 1D convolution operation may be performed, followed by a nonlinear operation as an activation, which may include one or more of a parametric rectified linear unit (PReLU), a rectified linear unit (ReLU), a leaky rectified linear unit (LReLU), an exponential linear unit (eLU), and a scaled exponential linear unit (SeLU). Figure 4 In the example of , the nonlinear operation can be based on PReLU. It is also possible to perform a 1D convolution operation in each of the encoder layers, followed by a PreLU as activation. In this case, the input signal is progressively downsampled by a factor of 2.

[0095] The last layer 15 in the encoder layer may then be followed by an output layer or compression layer 16. Although the number N and size of filters in the output layer are not limited, Figure 4 In the example of , the output layer 16 may include N filters of size 65, where the N filters may operate with a stride of 1 (no downsampling). In the output layer, a 1D convolution operation may be performed, followed by a nonlinear operation, which may include one or more of a parametric rectified linear unit (PReLU), a rectified linear unit (ReLU), a leaky rectified linear unit (LReLU), an exponential linear unit (eLU), and a scaled exponential linear unit (SeLU). In this case, in the output layer, a flattening operation may be performed to obtain a channel number of 1.

[0096] Figure 4 The architecture of the encoder part of the generator schematically illustrated in can be summarized as follows:

[0097] 11 / Input: Residual of the original speech signal

[0098] 12 / Encoder layer L=1: in_channels=1, out_channels=32, kernel size=64, stride=2, activation=PreLU

[0099] 13 / Encoder layer L=2: in_channels=32, out_channels=64, kernel size=64, stride=2, activation=PreLU

[0100] 14 / Encoder layer L=3: in_channels=64, out_channels=64, kernel size=64, stride=2, activation=PreLU

[0101] 15 / Encoder layer L=4: in_channels=64, out_channels=128, kernel size=64, stride=2, activation=PreLU

[0102] 16 / Output layer: in_channels=128, out_channels=1, kernel size=65, stride=1, activation=PreLU

[0103] The architectures proposed above represent only examples. Depending on the application, the number of layers in the encoder part can be scaled down or scaled up, respectively.

[0104] Reference now Figure 5 , schematically illustrating the architecture of the context decoding section of the decoder portion of the generator. Context decoding may include two processes. In the first process, a compressed representation of the residual may be input 17 into a layer 18 (layer number L=1) of the context decoding section. The L=1 layer 18 may include N filters operating with a stride of 1 and a filter size of 1. In the L=1 layer 18, a 1D convolution may be performed on the compressed representation of the residual to obtain 64 channels (the output may be represented as an embedding).

[0105] In the second pass, the output of the L=1 layer 18 may be passed through one or more blocks of gated tanh units (GTUs) 19, 20, 21. Although the number of gated tanh units is not limited, in one embodiment, the context decoding segment may include 10 blocks of gated tanh units. In one embodiment, the output of the last block of gated tanh units may be concatenated with a random noise vector (z) 22.

[0106] exist Figure 5 The architecture of the context decoding section of the decoder portion of the generator schematically illustrated in can be summarized as follows:

[0107] 17 / Input: Compressed representation of the residual of the original speech signal

[0108] 18 / layer L=1: in_channels=1, out_channels=64, kernel size=1, stride=1

[0109] 19 / Gated tanh unit

[0110] 20 / Gated tanh unit . . .

[0114] 21 / Gated tanh unit

[0115] 22 / Concatenate with random noise vector (z)

[0116] When providing a context decoding section before the adversarial generation section, the output of the encoder part of the generator can thus be mapped into a different embedding latent space, which has been shown to be better for applying adversarial upsampling by the adversarial generation section to obtain a pseudo speech signal.

[0117] Below, before describing the details of the adversarial generation section, the operation of the gated tanh unit will be described in more detail. In one embodiment, at least one of the gated tanh units of the context decoding section may be softmax gated. In one embodiment, the gated tanh units of the context decoding section may all be softmax gated. In general, all layers of the decoder portion of the generator in which 1D convolution operations are performed may have a softmax gated tanh activation function, where softmax is applied along the channel dimension of the gate output tensor.

[0118] The function defined by such a gating layer can be given by the following equation:

[0119] out = tanh(W f *X)⊙softmax(W g *X)| c (1)

[0120] Where out is the output of the gated 1D convolutional layer, X is the input to the gated 1D convolutional layer, and W f is the weight of the 1D convolution filter, W g are the weights of the 1D convolution gate, * denotes the convolution operation, ⊙ denotes element-wise multiplication, and softmax(.)|c denotes the softmax operation applied along the channel dimension of its input tensor.

[0121] Reference now Figure 6 The example of , schematically illustrates the operation of the softmax gated tanh unit. Figure 6 In the example of , in step 301, the input is fed into the convolutional layer 23 of the softmax gated tanh unit. A 1D convolution operation may be performed in the convolutional layer 23. The convolutional layer 23 may further include N filters operating with a stride of 1. Although the size of the N filters is not limited, Figure 6In the example of , the filter size may be 65. Then, in step 303, the output from the convolutional layer 23 may be divided into two parts. For example, if the output from the convolutional layer 23 is 64 channels, then the 64 channels may be equally divided into 32 channels that advance to the tanh gate 24 and 32 channels that advance to the softmax gate 25. The tanh gate operates on 32 channels along the sample dimension, while the softmax gate operates along the channel dimension. In step 304, the outputs from the tanh gate 24 and the softmax gate 25 may then be element-wise multiplied. The output from the multiplication in step 304 is then input to another convolutional layer 26, in which a 1D convolution may also be performed. The convolutional layer 26 may include N filters of size 1 and operating with a stride of 1. Then, in step 305, the output of the convolutional layer 26 is combined with the residual connection of step 302. The residual connection may prevent the gradients of the deeper layers of the decoder portion of the generator from disappearing. In this case, the residual connection can maintain sufficient gradients during the back-propagation process, resulting in faster convergence. The output shape in step 306 eventually corresponds to the shape of the input in step 301, so no dimensionality change occurs within the gated tanh unit.

[0122] Reference now Figure 7 The example of schematically illustrating the layered architecture of the adversarial generation section of the decoder portion of the generator. Although the architecture of the adversarial generation section of the generator is generally not limited, in one embodiment, the adversarial generation section may include L layers, wherein each layer L has N filters. L may be a natural number ≥ 1, and N may be a natural number ≥ 1. The size of the N filters is not limited, and may be selected based on the requirement of generating a pseudo speech signal from a compressed representation of the residual of the original speech signal. In one embodiment, the size of the N filters in each of the L layers may be the same. In one embodiment, the N filters in each of the L layers may operate with a step size of 2.

[0123] exist Figure 7 In the example of , the output of the context decoding section may be input 27 into the first layer 28 of the adversarial generation section. The illustrated first layer 28 (layer number L=1) may include N filters operating with a stride of 2 and a filter size (or kernel size) of 66. The illustrated second layer 29 (layer number L=2) may also include N filters operating with a stride of 2 and a filter size of 66. Although the adversarial generation section is not limited with respect to the number of layers L, in Figure 7In the example of L, the adversarial generation section may include a total of 4 layers 28, 29, 30, 31, wherein each of the 4 layers may include N filters with a filter size of 66 and operating with a stride of 2. Alternatively, in each of the 4 encoder layers 28, 29, 30, 31, the filter may operate with a stride of 1 and then be downsampled by a factor of 2. In one embodiment, in at least one of the L layers, a transposed convolution operation may be performed followed by a gated tanh unit (GTU). In one embodiment, the gated tanh unit may be a softmax gate.

[0124] The output layer 32 may then follow the last layer 31 in the L layers. Although the number N and size of filters in the output layer are not limited, Figure 7 In the example of , the output layer 32 may include N filters of size 65 and operated with a stride of 1. In the output layer, a 1D convolution may be performed followed by a tanh operation to obtain a pseudo speech signal.

[0125] Figure 7 The architecture of the adversarial generation section schematically illustrated in can be summarized as follows:

[0126] 27 / Input: Output of the context decoding segment concatenated with the random noise vector (z)

[0127] 28 / layer L = 1: in_channels = 128, out_channels = 32, kernel size = 66, stride = 2, GTU (32)

[0128] 29 / Layer L = 2: in_channels = 64, out_channels = 32, kernel size = 66, stride = 2, GTU (32)

[0129] 30 / layer L = 3: in_channels = 64, out_channels = 16, kernel size = 66, stride = 2, GTU (16)

[0130] 31 / layer L = 4: in_channels = 32, out_channels = 32, kernel size = 66, stride = 2, GTU (32)

[0131] 32 / output layer: in_channels=32, out_channels=1, kernel_size=65, stride=1, activation=tanh_discriminator

[0132] Reference now Figure 8The example of schematically illustrating the architecture of the discriminator 39 of the generative adversarial network setting for training the generator. In one embodiment, the discriminator 39 may include L layers 33, 34, 35, 36, 37, each of which may include N filters. L may be a natural number ≥ 1, and N may be a natural number ≥ 1. The size of the N filters is not limited and can be selected according to the requirements of the discriminator 39. In one embodiment, the filter size may be the same in each of the L layers. In one embodiment, the N filters can operate with a stride of 2. In one embodiment, in at least one of the L layers of the discriminator 39, a 1D convolution operation may be performed, followed by a nonlinear operation that may include an LReLU.

[0133] In an embodiment, the discriminator 39 may include an output layer 38, which is followed by the last layer 37 in the L layers. In an embodiment, the output layer 38 may have N filters with a filter size of 32. In an embodiment, the N filters may operate with a stride of 2. Therefore, the output layer 38 may be a one-dimensional convolutional layer that downsamples the hidden activations.

[0134] refer to Figure 8 In the example above, the architecture of the discriminator can be summarized as follows:

[0135] 33 / Discriminator layer L=1: in_channels=2, out_channels=16, filter_size=32, stride=2, activation=leakyReLU

[0136] 34 / Discriminator layer L=2: in_channels=16, out_channels=16, filter_size=32, stride=2, activation=leakyReLU

[0137] 35 / Discriminator layer L=3: in_channels=16, out_channels=32, filter_size=32, stride=2, activation=leakyReLU

[0138] 36 / Discriminator layer L=4: in_channels=32, out_channels=32, filter_size=32, stride=2, activation=leakyReLU

[0139] 37 / Discriminator layer L=5: in_channels=32, out_channels=64, filter_size=32, stride=2, activation=leakyReLU

[0140] 38 / Output layer: in_channels=64, out_channels=32, filter size=32, stride=2

[0141] Output (not shown): Determine whether the input is an original speech signal or a pseudo speech signal

[0142] The architectures proposed above represent examples only. Depending on the application, the number of layers in the discriminator can be scaled down or scaled up, respectively.

[0143] Generative Adversarial Network (GAN) Setup

[0144] In one embodiment, the generator may be a generator trained in a generative adversarial network setting (GAN setting). In one embodiment, the GAN setting may include one or more of a geometric GAN setting, a Wasserstein GAN setting, and an energy-based GAN setting. In one embodiment, the GAN setting may further be a conditional GAN ​​setting that may be set by a condition generator and a condition discriminator.

[0145] The discriminator can be adjusted by inputting the same input as the generator. The discriminator can then contain two input channels, one for the original or pseudo speech signal and the other for the corresponding residual. When the discriminator determines that the input is the original speech signal, the original residual can represent the adjustment because it is the linear prediction coding counterpart of the original speech signal. Vice versa, when the discriminator determines that the input is a pseudo speech signal, the original residual can represent the adjustment because it is the input to the generator that generates the pseudo speech signal. Therefore, the original residual can have the ability to adjust the discriminator.

[0146] The generator may include an encoder part and a decoder part. The decoder part may include an adversarial generation section. The decoder part may further include a context decoding section before the adversarial generation section. In any case, all parts of the generator are jointly trained in the GAN setting. Figure 8 As illustrated in the example of , the identifier may include a layered architecture.

[0147] Training the discriminator and generator in a GAN setting can often be based on game theory that introduces the minimax rule, which is an objective function for solving zero-sum games:

[0148] min G max D V(D,G) (2)

[0149] In the above equation, let V(D, G) denote the value function V of two competing players D and G, where each player attempts to increase its payoff to this function at the expense of the other player. The minimax goal will then be to minimize the maximum payoff obtained by player D as per equation (2).

[0150] The discriminator and generator in the GAN setting can be trained by modeling the objective minimax function accordingly, for example, based on a distance-based adversarial loss function as in the Wasserstein GAN setting, or based on a distance-based divergence loss in terms of a support vector machine that finds a hyperplane separating the original and fake data feature vectors as in the geometric GAN setting.

[0151] In one embodiment, the training of the discriminator and the generator can be based on one or more loss functions, respectively. An example of the discriminator loss function and an example of the generator loss function are described by the following equations (3) and (4):

[0152] Discriminator loss:

[0153]

[0154] Generator loss:

[0155]

[0156] In the above equation, γ = 0.00015 is the regularization multiplier, and is the expected value. In equation (3), the discriminator loss function L D The first half of describes the expected value of the original (real) sample, that is, the distribution P based on the original speech signal x and the corresponding original residual y. The second half of the discriminator loss function (3) describes the expected value generated by the generator based on the corresponding pseudo speech signal G(y). In equation (4), the generator loss function L G The first half of is the weighted reconstruction loss, while the second half is the weighted adversarial loss.

[0157] A method for training a generator in a generative adversarial network setting including a generator and a discriminator may include the following steps. A compressed representation of the residual of an original speech signal may be input into the generator. The residual may be considered as a noise prior. Then, the generator may generate a pseudo speech signal based on the compressed representation of the residual by feeding it to an encoder part and a decoder part. In the next step, the pseudo speech signal and the compressed residual of the original speech signal from which the pseudo speech signal is generated may be input into the discriminator one at a time. Then, the discriminator may determine whether the pseudo speech signal corresponds to the compressed residual of the original speech signal (i.e., the discriminator determines that the input speech signal is the original speech signal) or to an undefined compressed residual (i.e., the discriminator determines that the input speech signal is a pseudo speech signal). The parameters of the generator may then be tuned accordingly until the discriminator is no longer able to distinguish whether the pseudo speech signal corresponds to the compressed residual of the original speech signal or to an undefined compressed residual.

[0158] explain

[0159] Unless otherwise specifically stated, as will be apparent from the following discussion, it should be understood that throughout the public discussion, terms such as "processing," "computing," "calculating," "determining," "analyzing," or the like are utilized to refer to the action and / or process of a computer or computing system or similar electronic computing device manipulating data represented as physical quantities (e.g., electronic quantities) and / or transforming data represented as physical quantities (e.g., electronic quantities) into other data similarly represented as physical quantities.

[0160] In a similar manner, the term "processor" may refer to any device or portion of a device that processes electronic data, such as from registers and / or memory, to transform the electronic data into other electronic data, such as may be stored in registers and / or memory. A "computer" or "computing machine" or "computing platform" may include one or more processors.

[0161] In an example embodiment, the methods described herein may be performed by one or more processors that accept computer-readable (also referred to as machine-readable) code containing a set of instructions that, when executed by the one or more processors, perform at least one of the methods described herein. Any processor capable of executing a set of instructions (sequentially or otherwise) that specifies an action to be taken is included. Thus, an example is a typical processing system that includes one or more processors. Each processor may include one or more of a CPU, a graphics processing unit, and a programmable DSP unit. The processing system may further include a memory subsystem including a main RAM and / or static RAM and / or ROM. A bus subsystem for communication between components may be included. The processing system may further be a distributed processing system with processors coupled by a network. If the processing system requires a display, such a display may be included, for example, a liquid crystal display (LCD) or a cathode ray tube (CRT) display. If manual data entry is required, the processing system also includes an input device, such as one or more of an alphanumeric input unit (e.g., a keyboard), a pointing control device (e.g., a mouse), etc. The processing system may also encompass a storage system (e.g., a disk drive unit). The processing system in some configurations may include a sound output device and a network interface device. Thus, the memory subsystem includes a computer-readable carrier medium that carries a computer-readable code (e.g., software) that includes a set of instructions to cause it to perform one or more of the methods described herein when executed by one or more processors. It should be noted that when the method includes several elements (e.g., several steps), the order of these elements is not implied unless specifically stated. The software may reside in a hard disk, or may reside completely or at least partially in RAM and / or a processor during its execution by a computer system. Thus, the memory and processor also constitute a computer-readable carrier medium that carries the computer-readable code. In addition, the computer-readable carrier medium may form a computer program product or be included in a computer program product.

[0162] In alternative example embodiments, the one or more processors operate as a standalone device, or may be connected (e.g., networked) to other processors. In a networked deployment, the one or more processors may operate as a server or user machine in a server-user network environment, or as a peer machine in a peer-to-peer or distributed network environment. The one or more processors may form a personal computer (PC), tablet PC, personal digital assistant (PDA), cellular phone, network appliance, network router, switch or bridge, or any machine capable of executing a set of instructions (sequentially or otherwise) that specify actions to be taken by the machine.

[0163] It should be noted that the term "machine" shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

[0164] Thus, one example embodiment of each of the methods described herein is in the form of a computer-readable carrier medium carrying a set of instructions, such as a computer program for execution on one or more processors, such as one or more processors as part of a network server arrangement. Thus, as will be appreciated by one skilled in the art, example embodiments of the present disclosure may be embodied as a method, an apparatus such as a special purpose apparatus, an apparatus such as a data processing system, or a computer-readable carrier medium, such as a computer program product. The computer-readable carrier medium carries computer-readable code, which includes a set of instructions that, when executed on one or more processors, cause the processor or processors to implement the method. Accordingly, aspects of the present disclosure may take the form of a method, a fully hardware example embodiment, a fully software example embodiment, or an example embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a carrier medium, such as a computer program product on a computer-readable storage medium, carrying computer-readable program code embodied in the medium.

[0165] The software may further be transmitted or received over a network via a network interface device. Although the carrier medium is a single medium in an example embodiment, the term "carrier medium" should be taken to include a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) storing one or more sets of instructions. The term "carrier medium" should also be taken to include any medium capable of storing, encoding, or carrying a set of instructions for execution by one or more processors and causing the one or more processors to perform any one or more of the methods of the present disclosure. The carrier medium may take a variety of forms, including but not limited to non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical disks, magnetic disks, and magneto-optical disks. Volatile media include dynamic memory, such as main memory. Transmission media include coaxial cables, copper wires, and optical fibers, including wires that comprise a bus subsystem. Transmission media may also take the form of acoustic or optical waves, such as those generated during radio wave and infrared data communications. For example, the term "carrier medium" shall accordingly be taken to include, but not be limited to, solid-state memories, computer products embodied in optical and magnetic media; media carrying propagated signals detectable by at least one processor or one or more processors and representing a set of instructions for implementing a method when executed; and transmission media in a network carrying propagated signals detectable by at least one of the one or more processors and representing the set of instructions.

[0166] It should be understood that in an example embodiment, the steps of the method discussed are performed by an appropriate processor (or processors) of a processing (e.g., computer) system executing instructions (computer readable code) stored in a storage device. It should also be understood that the present disclosure is not limited to any particular implementation or programming technique, and that the present disclosure may be implemented using any suitable technique for implementing the functionality described herein. The present disclosure is not limited to any particular programming language or operating system.

[0167] References throughout this disclosure to "an example embodiment," "some example embodiments," or "example embodiments" mean that a particular feature, structure, or characteristic described in connection with the example embodiment is included in at least one example embodiment of the present disclosure. Thus, the phrases "in an example embodiment," "in some example embodiments," or "in an example embodiment" appearing in various places throughout this disclosure are not necessarily all referring to the same example embodiment. Furthermore, as would be apparent to one of ordinary skill in the art from this disclosure, in one or more example embodiments, the particular features, structures, or characteristics may be combined in any suitable manner.

[0168] As used herein, unless otherwise specified, the use of ordinal adjectives "first," "second," "third," etc. to describe common objects indicates merely that different instances of similar objects are being referred to and is not intended to imply that the objects so described must appear in a given order, either temporally, spatially, hierarchically, or in any other manner.

[0169] In the following claims and the description herein, any of the terms "including" or "it includes" is an open term, which means at least including the following elements / features, but does not exclude other elements / features. Therefore, when used in the claims, the term "including" should not be interpreted as being limited to the components, elements or steps listed thereafter. For example, the scope of expression of a device including A and B should not be limited to a device consisting only of elements A and B. Any of the terms "including" or "it includes" used herein is also an open term, which also means at least including the elements / features after the term, but does not exclude other elements / features. Therefore, including is a synonym for including and refers to including.

[0170] It should be understood that in the above description of the example embodiments of the present disclosure, in order to simplify the present disclosure and facilitate understanding of one or more of the various inventive aspects, various features of the present disclosure are sometimes brought together in a single example embodiment, figure, or description thereof. However, this method of disclosure should not be interpreted as reflecting the intention that the claims require more features than the features explicitly stated in each claim. Specifically, as reflected in the appended claims, the inventive aspects lie in less than all the features of a single aforementioned disclosed example embodiment. Therefore, the claims following the specific embodiment are hereby expressly incorporated into this specific embodiment, wherein each claim stands alone as a separate example embodiment of the present disclosure.

[0171] Furthermore, while some example embodiments described herein include some features included in other example embodiments (but not other features), combinations of features of different example embodiments are intended to be within the scope of the present disclosure and form different example embodiments, as understood by those skilled in the art. For example, in the appended claims, any of the claimed example embodiments may be used in any combination.

[0172] In the description provided herein, many specific details are set forth. However, it should be understood that example embodiments of the present disclosure may be practiced without these specific details. In other instances, well-known methods, structures, and techniques are not shown in detail in order not to obscure the understanding of this description.

[0173] Thus, while what is believed to be the best mode of the present disclosure has been described, those skilled in the art will recognize that other and further modifications may be made thereto without departing from the spirit of the present disclosure, and all such changes and modifications are intended to be within the scope of the present disclosure. For example, any formula given above is merely representative of a procedure that may be used. Functions may be added or deleted from the block diagrams, and operations may be interchanged between functional blocks. Steps may be added or deleted from the described methods within the scope of the present disclosure.

Claims

1. A method for decoding an original speech signal for hybrid adversarial parameter speech synthesis, wherein the method comprises the following steps: (a) receiving quantized original linear prediction coding parameters estimated by applying linear prediction coding analysis filtering to an original speech signal and a quantized compressed representation of a residual of the original speech signal; (b) dequantizing the compressed representation of the original linear predictive coding parameters and the residual; (c) inputting the dequantized compressed representation of the residual into a decoder portion of a generator to apply an adversarial mapping from a compressed residual domain to a pseudo signal domain, wherein the generator is trained in a generative adversarial network setting; (d) outputting a pseudo speech signal by the decoder portion of the generator; (e) applying linear predictive coding analysis filtering to the pseudo speech signal to obtain a corresponding pseudo residual; and (f) reconstructing the original speech signal by applying LPC cross-synthesis filtering to the pseudo residual and the dequantized original LPC analysis parameters.

2. The method according to claim 1, wherein the order of the linear prediction coding analysis filter used in step (e) is the same as the order used to estimate the original linear prediction coding parameters.

3. The method of claim 1 or claim 2, wherein the adversarial network setup includes the generator and the discriminator, and wherein the training of the generator and the discriminator is based on one or more loss functions.

4. A method according to claim 3, wherein the decoder part of the generator includes an adversarial generation section, the adversarial generation section includes L layers, wherein there are N filters in each layer, wherein L is a natural number ≥1 and wherein N is a natural number ≥1, wherein the N filters operate with a stride of 2, and the sizes of the N filters in each of the L layers are the same, and wherein in at least one of the L layers, a transposed convolution is performed followed by a gated tanh unit, and wherein an output layer subsequently follows the last layer of the L layers of the adversarial generation section, wherein the output layer includes N filters operating with a stride of 1, and wherein a 1D convolution operation is performed in the output layer followed by a tanh operation.

5. The method of claim 4, wherein the decoder portion of the generator further comprises a context decoding section preceding the adversarial generation section.

6. The method of claim 5 , wherein the context decoding segment comprises an L=1 layer having N filters, where N is a natural number ≥ 1, followed by one or more blocks of softmax-gated tanh units, wherein the N filters have a size of 1 and the N filters operate with a stride of 1, and wherein a 1D convolution operation is performed in the L=1 layer, and wherein outputs of the one or more blocks of softmax-gated tanh units of the context decoding segment are concatenated with a random noise vector.

7. A method for encoding an original speech signal for hybrid adversarial parameter speech synthesis, wherein the method comprises the following steps: (a) receiving the original speech signal; (b) applying linear predictive coding analysis filtering to the original speech signal to obtain a corresponding residual; (c) inputting the obtained residual into an encoder part of a generator to encode the residual, wherein the generator is trained in a generative adversarial network setting; (d) outputting, by the encoder portion of the generator, a compressed representation of the residual; (e) applying a linear prediction coding analysis filter to the original speech signal to estimate original linear prediction coding parameters; and (f) quantizing and transmitting the original LPC parameters and the compressed representation of the residual, wherein the order of the LPC analysis filter used in step (e) is higher than the order in step (b).

8. The method of claim 7, wherein the order of the linear prediction coding analysis filter used in step (b) is 16, and the order of the linear prediction coding analysis filter used in step (e) is in the range of 16 to 50.

9. The method of claim 7 or claim 8, wherein the adversarial network setup includes the generator and the discriminator, and wherein the training of the generator and the discriminator is based on one or more of a loss function.

10. The method of claim 9, wherein the encoder portion of the generator comprises L layers, wherein there are N filters in each layer, wherein L is a natural number ≥ 1 and wherein N is a natural number ≥ 1, wherein the size of the N filters in each of the L layers is the same, and the N filters operate with a stride of 2, and wherein in at least one of the L layers, a 1D convolution operation is performed followed by a nonlinear operation, the nonlinear operation comprising one or more of a parametric rectified linear unit (PReLU), a rectified linear unit (ReLU), a leaky rectified linear unit (LReLU), an exponential linear unit (eLU), and a scaled exponential linear unit (SeLU).

Images