Socoder techniques

Abstract

The present disclosure relates to an audio generator (10) configured to generate an audio signal (16) from a bitstream (3), the bitstream (3) representing the audio signal (16), the audio signal being subdivided into a sequence of frames, the audio generator (10) comprising: a first data provider (702) configured to provide, for a given frame, first data (15) derived from an input signal (14); a first processing block (40, 50, 50a-50h) configured to receive first data (15) for a given frame and to output first output data (69) in the given frame, where the first processing block (50) comprises: at least one preconditioned learnable layer (710) configured to receive the bitstream (3) or a processed version (112) thereof and to output target data (12) representing an audio signal (16) in the given frame for the given frame; at least one conditioned learnable layer (71, 72, 73) configured to process the target data (12) for a given frame to obtain conditioned feature parameters (74, 75) for the given frame; and a pattern element (77) configured to apply the conditioned characteristic parameter (74, 75) to the first data (15, 59a) or the normalized first data (59, 76 '); wherein the at least one preconditioned learnable layer (710) comprises at least one learnable loop layer.

Description

Technical Field

[0001] Vocoder techniques are presented, and more generally, techniques for generating audio signal representations (eg, bitstreams) and for generating audio signals (eg, at a decoder). Background Art

[0002] The techniques herein are generally construed as involving learnable layers, which may be implemented, for example, via neural networks (e.g., convolutional learnable layers, recurrent learnable layers, etc.).

[0003] In some examples, the present technology is also referred to as Neural End-to-End Speech Codec (NESC). Summary of the invention

[0004] The invention is defined in the independent claims.

[0005] According to one aspect, there is provided an audio generator configured to generate an audio signal from a bitstream, the bitstream representing the audio signal, the audio signal being subdivided into a sequence of frames, the audio generator comprising:

[0006] a first data provider configured to provide, for a given frame, first data derived from an input signal;

[0007] a first processing block configured to receive first data for a given frame and output first output data in the given frame,

[0008] The first processing block includes:

[0009] at least one pre-conditioned learnable layer configured to receive the bitstream or a processed version thereof and output, for a given frame, target data representing an audio signal in the given frame;

[0010] at least one conditional learnable layer configured to process the target data for a given frame to obtain conditional feature parameters for the given frame; and

[0011] a style element configured to apply a conditional feature parameter to the first data or the normalized first data;

[0012] The at least one pre-conditioned learnable layer comprises at least one recurrent learnable layer.

[0013] According to one aspect, an audio generator is provided, configured to generate an audio signal from a bitstream, the bitstream representing the audio signal, the bitstream being subdivided into an index sequence, the audio signal being subdivided into a sequence of frames, the audio generator comprising:

[0014] a quantization index converter configured to convert an index of a bit stream into a code;

[0015] a first data provider configured to provide, for a given frame, first data derived from an input signal from an external or internal source or from a bitstream;

[0016] A first processing block is configured to receive first data for a given frame and output first output data in the given frame, wherein the first processing block includes:

[0017] at least one pre-conditioned learnable layer configured to receive the bitstream or a processed version thereof and output, for a given frame, target data representing an audio signal in the given frame;

[0018] at least one conditional learnable layer configured to process target data for a given frame to obtain conditional feature parameters for the given frame; and

[0019] The style element is configured to apply the conditional feature parameter to the first data or the normalized first data.

[0020] According to an aspect, there is provided an encoder for generating a bitstream, wherein an input audio signal comprising a sequence of input audio signal frames is encoded, each input audio signal frame comprising a sequence of input audio signal samples, the encoder comprising:

[0021] A format definer configured to define a first multi-dimensional audio signal representation of an input audio signal, the first multi-dimensional audio signal representation of the input audio signal comprising at least:

[0022] a first dimension such that a plurality of mutually consecutive frames are ordered according to the first dimension; and

[0023] a second dimension such that the plurality of samples of at least one frame are sorted according to the second dimension;

[0024] A learnable quantizer is configured to associate an index of at least one codebook to each frame of a first multidimensional audio signal representation of an input audio signal or a processed version of the first multidimensional audio signal representation, thereby generating a bitstream.

[0025] According to an aspect, there is provided an encoder for generating a bitstream, wherein an input audio signal comprising a sequence of input audio signal frames is encoded, each input audio signal frame comprising a sequence of input audio signal samples, the encoder comprising:

[0026] A learnable quantizer is configured to associate an index of at least one codebook to each frame of a first multi-dimensional audio signal representation of an input audio signal, thereby generating the bitstream.

[0027] According to an aspect, there is provided an encoder for generating a bitstream encoding an input audio signal comprising a sequence of input audio signal frames, each input audio signal frame comprising a sequence of input audio signal samples, the encoder comprising:

[0028] A format definer configured to define a first multi-dimensional audio signal representation of an input audio signal, the first multi-dimensional audio signal representation of the input audio signal comprising at least:

[0029] a first dimension such that a plurality of mutually consecutive frames are ordered according to the first dimension; and

[0030] a second dimension such that the plurality of samples of at least one frame are sorted according to the second dimension;

[0031] At least one intermediate learnable layer;

[0032] A learnable quantizer is configured to associate an index of at least one codebook to each frame of the first multidimensional or processed version of a first multidimensional audio signal representation of an input audio signal, thereby generating the bitstream.

[0033] According to one aspect, there is provided a method for generating an audio signal from a bitstream, the bitstream representing the audio signal, the audio signal being subdivided into a sequence of frames, the method comprising:

[0034] providing, for a given frame, first data derived from an input signal;

[0035] receiving first data and outputting first output data in a given frame by a first processing block,

[0036] The first processing block includes:

[0037] at least one pre-conditioned learnable layer that receives the bitstream or a processed version thereof and outputs, for a given frame, target data representing an audio signal in a given frame;

[0038] at least one conditional learnable layer, e.g., processing target data for a given frame to obtain conditional feature parameters for the given frame; and

[0039] A style element that applies a conditional feature parameter to the first data or the normalized first data;

[0040] The at least one pre-conditioned learnable layer comprises at least one recurrent learnable layer.

[0041] According to one aspect, a method for generating an audio signal from a bitstream is provided, wherein the bitstream (3) represents the audio signal, the bitstream is subdivided into a sequence of indices, and the audio signal is subdivided into a sequence of frames, the method comprising:

[0042] The quantization index converter step converts the bitstream index to the code;

[0043] A first data provider step, for example providing, for a given frame, first data derived from an input signal from an external or internal source or from a bitstream; and

[0044] the step of using a first processing block to receive first data and output first output data in a given frame,

[0045] The first processing block includes:

[0046] at least one pre-conditioned learnable layer for receiving a bitstream or a processed version thereof and outputting, for a given frame, target data representing an audio signal in a given frame;

[0047] at least one conditional learnable layer, e.g., to process target data for a given frame to obtain conditional feature parameters for the given frame; and

[0048] The style element applies the conditional feature parameter to the first data or the normalized first data.

[0049] According to an aspect, there is provided an audio signal representation generator for generating an output audio signal representation from an input audio signal comprising a sequence of input audio signal frames, each input audio signal frame comprising a sequence of input audio signal samples, the audio signal representation generator comprising:

[0050] A format definer configured to define a first multi-dimensional audio signal representation of an input audio signal, the first multi-dimensional audio signal representation of the input audio signal comprising at least:

[0051] a first dimension, so that a plurality of mutually consecutive frames are sorted according to the first dimension; and

[0052] a second dimension such that the plurality of samples of at least one frame are sorted according to the second dimension,

[0053] At least one learnable layer is configured to process a first multi-dimensional audio signal representation of an input audio signal, or a processed second multi-dimensional audio signal representation of the first multi-dimensional audio signal, to generate an output audio signal representation of the input audio signal.

[0054] According to an aspect, there is provided an audio signal representation generator for generating an output audio signal representation from an input audio signal comprising a sequence of input audio signal frames, each input audio signal frame comprising a sequence of input audio signal samples, the audio signal representation generator comprising:

[0055] a format definer configured to define a first multi-dimensional audio signal representation of an input audio signal;

[0056] a second learnable layer, the second learnable layer being a recurrent learnable layer configured to generate a third multidimensional audio signal representation of the input audio signal by operating along a first direction of the first multidimensional audio signal representation or a processed version thereof, the processed version being the second multidimensional audio signal representation of the input audio signal;

[0057] a third learnable layer, the third learnable layer being a convolutional learnable layer configured to generate a fourth multidimensional audio signal representation of the input audio signal by sliding along a second direction of the first multidimensional audio signal representation of the input audio signal,

[0058] Thereby an output audio signal representation is obtained from the fourth multi-dimensional audio signal representation of the input audio signal.

[0059] According to an aspect, there is provided a method for generating an output audio signal representation from an input audio signal comprising a sequence of input audio signal frames, each input audio signal frame comprising a sequence of input audio signal samples, the audio signal representation generator comprising:

[0060] defining a first multi-dimensional audio signal representation of an input audio signal;

[0061] generating, by the first learnable layer, a second multidimensional audio signal representation of the input audio signal by sliding along a second direction of the first multidimensional audio signal representation of the input audio signal;

[0062] generating, by a second learnable layer, a third multidimensional audio signal representation of the input audio signal by operating along a first direction of a second multidimensional audio signal representation of the input audio signal, the second learnable layer being a recurrent learnable layer;

[0063] generating, by a third learnable layer, a fourth multidimensional audio signal representation of the input audio signal by sliding along a second direction of the first multidimensional audio signal representation of the input audio signal, the third learnable layer being a convolutional learnable layer,

[0064] Thereby an output audio signal representation is obtained from the fourth multi-dimensional audio signal representation of the input audio signal.

[0065] According to one aspect, there is provided an audio generator configured to generate an audio signal from a bitstream, the bitstream representing the audio signal, the audio signal being subdivided into a sequence of frames, the audio generator comprising:

[0066] a first data provider configured to provide first data derived from an input signal for a given frame, wherein the first data has a plurality of channels;

[0067] a first processing block configured to receive first data for a given frame and output first output data in the given frame, wherein the first output data may include a plurality of channels,

[0068] The audio generator further comprises a second processing block configured to receive the first output data or data derived from the first output data as second data for a given frame,

[0069] The first processing block includes:

[0070] at least one pre-conditioned learnable layer configured to receive the bitstream or a processed version thereof and output, for a given frame, target data representing an audio signal in the given frame, wherein a plurality of channels and a plurality of samples are used for the given frame;

[0071] at least one conditional learnable layer configured to process the target data for a given frame to obtain conditional feature parameters for the given frame; and

[0072] a style element configured to apply a conditional feature parameter to the first data or the normalized first data;

[0073] wherein the second processing block is configured to combine the plurality of channels of the second data to obtain an audio signal,

[0074] The at least one pre-conditioned learnable layer comprises at least one recurrent learnable layer. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1a and Figure 1b An example is shown.

[0076] Figure 1c Operation according to an example is shown.

[0077] Figure 2a Figure 2b , Figure 2c The experimental results are shown.

[0078] Figure 3 An example of elements of a decoder is shown.

[0079] Figure 4 An example of an audio generator is shown.

[0080] Figure 5 and Figure 6 Experimental results of a listening test are shown.

[0081] Figure 7 An example of a decoder is shown.

[0082] Figure 8 Examples of encoders and decoders are shown.

[0083] Fig. 9 Operation according to an example is shown.

[0084] Fig.10 An example of a generative adversarial network (GAN) discriminator is shown.

[0085] Fig.11 and 12 An example of a GRU implementation is shown. DETAILED DESCRIPTION

[0086] Figure 1b (in Figure 1a is a simplified version of this, or Figure 8 , which is a more detailed version thereof, shows an example of a vocoder system (or more generally, a system for processing an audio signal). The vocoder system may include, for example, an audio signal representation generator 20 for generating an audio signal representation of an input audio signal 1. The audio signal 1 may be processed by the audio signal representation generator 20. The audio signal representation of the input audio signal 1 may be stored (e.g., for purposes such as processing of the audio signal) or may be quantized (e.g., by a quantizer 300), thereby obtaining a bitstream 3. A decoder 10 (audio generator) may read the bitstream 3 and generate an output audio signal 16.

[0087] Each of the audio signal representation generator 20, the encoder 2 and / or the decoder 10 may be a learnable system and may include at least one learnable layer and / or learnable block.

[0088] The input audio signal 1 (which may be obtained, for example, from a microphone or may be obtained from other sources, such as a storage unit and / or a synthesizer) may be of a type having a sequence of audio signal frames. For example, different input audio signal frames may represent sounds of a fixed time length (for example, 10 milliseconds (ms or milliseconds), but in other examples, different lengths may be defined, for example, 5ms and / or 20ms). Each input audio signal frame may comprise a sequence of samples (for example, at 16 kHz or kilohertz and there will be 160 samples in each frame). In this case, the input audio signal is in the time domain, but in other cases it may be in the frequency domain. However, in general, the input audio signal 1 may be understood as having a single dimension. In Figure 1b (or a more detailed version of it Figure 8), the input audio signal 1 is represented as having five frames, each frame having only two samples (of course, this is for simplicity). For example, the frame numbered t-1 has two samples 0' and 0'. Frame number t in the sequence has samples 1' and 1'. Frame number t+1 has samples 2' and 2'. Frame number t+2 has samples 3' and 3'. Frame number t+3 has samples 4' and 4'. The input audio signal 1 can be provided to a learnable block 200. The learnable block 200 can be a type with a dual path (for example, processing at least one residual). The learnable block 200 can provide a processed version 269 of the input audio signal 1 to a second learnable block 290 (this can be avoided in some cases). Subsequently, the learnable block 200 or the learnable block 290 can provide a processed version of the input audio signal 1 output by it to a quantizer 300. The quantizer 300 can provide a bitstream 3. It will be seen that the quantizer 300 can be a learnable quantizer. In some cases, the output may be provided only by the learnable block 290, to have as output the audio signal representation 269. In some cases, the quantizer 300 may therefore not even be present.

[0089] The learnable block 200 can process the input audio signal 1 (in one of its processed versions) after converting the input audio signal 1 (or its processed version) into a multidimensional representation. Therefore, a format definer 210 can be used. The format definer 210 can be a deterministic block (e.g., a non-learnable block). Downstream of the format definer 210, the processed version 220 (also referred to as the first audio signal representation of the input audio signal 1) output by the format definer 210 can be processed by at least one learnable layer (e.g., 230, 240, 250, 290, 429, 440, 460, 300, see below). At least the learnable layer (e.g., layer 230, 240, 250) inside the learnable block 200 is a learnable layer that processes the first audio signal representation 220 of the input audio signal 1 with its dimensional version (e.g., a two-dimensional version). The learnable layers 429, 440, 460 can also process the multidimensional version of the input audio signal 1. As will be shown, this can be achieved, for example, by a rolling window that moves along a single dimension (time domain) of the input audio signal 1 and generates a multidimensional version 220 of the input audio signal 1. It can be seen that the first audio signal representation 220 of the input audio signal 1 can have a first dimension (intra-frame dimension) such that a plurality of mutually consecutive frames (e.g., immediately following one another) are ordered according to (along) the first dimension. It should also be noted that the second dimension (intra-frame dimension) is such that the samples of each frame are ordered according to (along) the second dimension. Figure 1bAs shown, frame t is then organized along the second direction (intra-frame direction) with two samples 0' and 0'. It can be seen that this sequence of frames t, t+1, t+2, t+3, etc. can be considered along the first dimension, while in the second dimension, the sample sequence is also considered for each frame. The format definer 210 is configured to insert the input audio signal samples of each given frame along the second dimension [e.g., the intra-frame dimension] of the first multi-dimensional audio signal representation of the input audio signal. Additionally or alternatively, the format definer 210 is configured to insert the following additional input audio signal samples along the second dimension [e.g., the intra-frame dimension] of the first multi-dimensional audio signal representation 220 of the input audio signal 1: one or more additional frames immediately preceding the given frame [e.g., in a predefined number, e.g., application-specific, e.g., defined by a user or an application]. The format definer 210 is configured to insert the additional input audio signal samples of one or more additional frames immediately preceding the given frame [e.g., in a predefined number, e.g., application-specific, e.g., defined by a user or an application] along the second dimension of the first multi-dimensional audio signal representation 220 of the input audio signal 1.

[0090] like Figure 1c (In this case, each frame is considered to have nine samples, but it is also possible to Figure 1b with different numbers of samples), there is the possibility to also insert samples of the preceding (immediately preceding) frame and / or samples of the succeeding (immediately succeeding) frame along the second (intra-frame) dimension. Figure 1c In the example of , in the first audio signal representation 220 of the input audio signal 1, the first three samples of frame t are actually occupied by the last three samples of the immediately preceding frame t-1. Alternatively or additionally, the last three samples of frame t in the first audio signal representation 220 of the input audio signal 1 are occupied by the first three samples of the immediately following frame t+1. This is performed frame by frame, so that the first audio signal representation 220 has in each frame the first sample inherited from the last sample of the immediately preceding frame, and the first sample of the immediately following frame as the last sample. It is worth noting that the number of samples of each frame from the input version 1 to the processed version 220 is thus increased. However, it is not always necessary to perform this technique. The number of samples inherited from the immediately preceding frame or the immediately following frame or subsequent frame does not always have to be three (different numbers are also possible, although they are usually smaller than the samples inherited from other samples that are not considered, in total, for more than 50% of the samples of the frames in the version 220) or there may be different numbers of initial samples and / or final samples. In some cases, the initial sample or the final sample is not inherited from the immediately preceding frame or the immediately following frame. In some cases, this technique is not used. Figure 1b (or a more detailed version of it Figure 8), frame t inherits the total number of samples of frame t-1, frame t+1 inherits the total number of samples of frame t, and so on. Although this is only one representation. Nevertheless, downsampling techniques using strided convolutions or interpolation layers are avoided. As will be explained below, the inventors have appreciated that this is advantageous. Even for each frame, a multi-dimensional structure can be defined so that the first audio signal representation 220 has more than two dimensions. This is an example of a dual-path convolutional recurrent learnable layer (e.g., a dual-path convolutional recurrent neural network). There is also an example in section 2.1 of the "Discussion" section below.

[0091] Downstream of the format definer 210, at least one learnable layer 230, 240, 250 may be input by a first audio signal representation 220 of the input audio signal 1. It is noteworthy that in this case, at least one learnable layer 230, 240 and 250 may follow a residual technique. For example, at point 248, a residual value may be generated from the audio signal representation 220. Specifically, the first audio signal representation 220 may be subdivided into a main portion 259a' and a residual portion 259a of the first audio signal representation 220 of the input audio signal. Therefore, the main portion 259a' of the first audio signal representation 220 may not undergo any processing until point 265c, in which the main portion 259a' of the first audio signal representation 220 is added to (added to) a processed residual version 265b' output by at least one learnable layer 230, 240 and 250 such as cascaded to each other. Therefore, a processed version 269 of the input audio signal 1 may be obtained.

[0092] At least one residual learnable layer 230, 240, 250 may include:

[0093] - an optional first learnable layer (230), such as a first convolutional learnable layer, which is a convolutional learnable layer configured to generate a second multidimensional audio signal representation of the input audio signal (1) by sliding along a second direction [such as an intra-frame direction] of the first multidimensional audio signal representation (220) of the input audio signal (1);

[0094] a second learnable layer (240), which may be a recurrent learnable layer (e.g. a gated recurrent learnable layer), configured to generate a third multidimensional audio signal representation of the input audio signal (1) by operating along a first direction [e.g. an intra direction] of the second multidimensional audio signal representation (220) of the input audio signal (1) [e.g. using a 1x1 kernel, e.g. a 1x1 learnable kernel, or another kernel, e.g. another learnable kernel];

[0095] - a third learnable layer (250) [for example, the third learnable layer may be a second convolutional learnable layer], the third learnable layer being a convolutional learnable layer configured to generate a fourth multidimensional audio signal representation (265b') of the input audio signal by sliding along a second direction [for example, an intra-frame direction] of the first multidimensional audio signal representation of the input audio signal [for example, using a 1x1 kernel, for example, a 1x1 learnable kernel].

[0096] It is worth noting that the first learnable layer 230 can be a first convolutional learnable layer. It may have a 1x1 kernel. The 1×1 kernel can be applied by sliding the kernel along the second dimension (i.e., for each frame). The recurrent learnable layer 240 (e.g., a gated recurrent unit, GRU) can be input with the output from the first convolutional learnable layer 230. The recurrent learnable layer (e.g., GRU) can be applied in the first dimension (i.e., by sliding from frame t, to frame t+1, to frame t+2, and so on). As will be explained later, in the recurrent learnable layer 240, each value of the output for each frame can also be based on the previous frame (e.g., the frame immediately preceding, or also the n frames immediately preceding a particular frame; for example, for the output of the recurrent learnable layer 240 for frame t+3 where n=2, the output will take into account the values ​​of the samples for frame t+1 and for frame t+2, but not the values ​​of the samples for frame t). The processed version of the input audio signal 1 output by the recurrent learnable layer 240 may be provided to a second convolutional learnable layer (third learnable layer) 250. The second convolutional learnable layer 250 may have a kernel (e.g., 1×1 kernels) that slides along a second dimension (along a second intra-frame dimension) for each frame. The output 265b′ of the second convolutional learnable layer 250 may then be added, for example, at point 265c (some or other) to a main portion 259a′ of the first audio signal representation 220 of the input audio signal 1 that has bypassed the learnable layers 230, 240, and 250.

[0097] The processed version 269 of the input audio signal 1 may then be provided (as potential 269) to at least one learnable block 290. The at least one convolutional learnable block 290 may provide, for example, a 256-sample version (even though a different number, such as 128, 516, etc., may be used).

[0098] like Figure 8 As shown, at least one convolutional learnable block 290 may include a convolutional learnable layer 429 to perform convolution on the signal 269 (e.g., as output by the learnable block 200). The convolutional learnable layer 429 may be a non-residual learnable layer. The convolutional learnable layer 429 may output a convolutional version 420 of the signal 269 and may also be a processed version of the input audio signal 1.

[0099] At least one convolutional learnable block 290 may include at least one residual learnable layer. At least one convolutional learnable block 290 may include at least one learnable layer (e.g., 440, 460). The learnable layers 440, 460 (or at least one or some of them) may follow a residual technique. For example, at point 448, a residual value may be generated from the audio signal representation or potential representation 269 (or its convolution version 420). Specifically, the audio signal representation 420 may be subdivided into a main portion 459a' and a residual portion 459a of the audio signal representation 420 of the input audio signal 1. Therefore, the main portion 459a' of the audio signal representation 420 of the input audio signal 1 may not be subjected to any processing until point 465, in which the main portion 459a' of the audio signal representation 420 of the input audio signal 1 is added (summed) to the processed residual version 465b' output by at least one learnable layer 440 and 460 cascaded to each other. Thus, a processed version 469 of the input audio signal 1 may be obtained and may represent the output of the audio representation generator 20 .

[0100] The at least one residual learnable layer in the at least one convolutional learnable block 290 may include at least one of the following:

[0101] a first layer (430) configured to generate a residual multidimensional audio signal representation of the input audio signal (1) from the audio signal representation 420 (the first layer 430 may be an activation function, such as a Leaky ReLu, see below);

[0102] - A second learnable layer (440), which is a convolutional learnable layer configured to generate a residual multidimensional audio signal representation of the input audio signal 1 from the audio signal representation output by the first learnable layer (430) by convolution [e.g., kernel 3 may be used].

[0103] - a third layer (450) for generating a residual multidimensional audio signal representation of the input audio signal 1 from the audio signal representation output by the second learnable layer (440) (the learnable layer 450 may be an activation function, such as a Leaky ReLu, see below);

[0104] a fourth learnable layer (460), the fourth learnable layer being a convolutional learnable layer configured to generate a residual multidimensional audio signal representation 456b' of the input audio signal 1 from the residual multidimensional audio signal representation of the input audio signal 1 output by the third learnable layer (450) by convolution [e.g., a kernel 1x1 may be used];

[0105] The output 465b' of the second convolutional learnable layer 460 (fourth learnable layer) may then be added (summed) at point 465 with the main portion 459a' of the audio signal representation 420 (or 269) of the input audio signal 1 which has bypassed layers 430, 440, 450, 460.

[0106] It should be noted that the output 469 may be considered to be the audio signal representation output by the audio signal representation generator 20 .

[0107] Subsequently, a quantizer 300 may be provided in case a bitstream 3 needs to be written. The quantizer 300 may be a learnable quantizer [e.g., a quantizer using at least one learnable codebook], which will be discussed in detail below. The quantizer (e.g., a learnable quantizer) 300 may associate an index of at least one codebook to each frame of a first multidimensional audio signal representation (e.g., 220 or 469) of the input audio signal (1) or a processed version of the first multidimensional audio signal representation, thereby generating a bitstream [the at least one codebook may be, for example, a learnable codebook].

[0108] It is worth noting that the cascade formed by the learnable layers 230, 240, 250 and / or the cascade formed by the layers 430, 440, 450, 460 may include more or fewer layers and different choices may be made. However, it is worth noting that they are residual learnable layers and they are bypassed by the main part 259' of the first audio signal representation 220.

[0109] Figure 7An example of a decoder (audio generator) 10 is shown. A bitstream 3 (obtained at input) may include frames (e.g. encoded as indices, e.g. encoded by encoder 2, e.g. after quantization by quantizer 300). An output audio signal 16 may be obtained. The audio generator 10 may include a first data provider 702. The first data provider 702 may be input with an input signal (input data) 14 (e.g. from an internal source, such as a noise generator or a storage unit, or from an external source, such as an external noise generator or an external storage unit or even data obtained from the bitstream 3). The input signal 14 may be noise (e.g. white noise) or a deterministic value (e.g. a constant). The input signal 14 may have a plurality of channels (e.g. 128 channels, but other numbers of channels are also possible, e.g. a number greater than 64). The first data provider 702 may output first data 15. The first data 15 may be noise, or taken from noise. The first data 15 may be input in at least one first processing block 50 (40). The first data 15 may be independent of the output audio signal 16 (e.g. when obtained from noise and therefore corresponding to the input signal 14), but in some cases they may be obtained from the bitstream 3, such as LPC parameters, or other parameters taken from the bitstream 3; it is worth noting that an advantage of this example is that the first data 15 does not need to be an explicit acoustic feature, and the first data 15 may more easily be noise. At least one first processing block 50 (40) may, for example, condition the first data 15 using conditions obtained by processing the bitstream 3 to obtain first output data 69. The first output data 69 may be provided to a second processing block 45. The audio signal 16 may be obtained from the second processing block (e.g. by PQMF synthesis). The first output data 69 may be in multiple channels. The first output data 69 may be provided to a second processing block 45, which may combine multiple channels of the first output data 69 to provide the output audio signal 16 in one signal channel (e.g. after PQMF synthesis, e.g. in Figure 4 and Fig.10 Indicated by 110, but in Figure 7 not shown).

[0110] As described above, the output audio signal 16 (as well as the original audio signal 1 and its encoded version, the bitstream 3 or its representation 20 or any other processed version thereof, such as 269 or the residual version 259a and 265b', or the main version 259a', and any intermediate version output by the layers 230, 240, 250, or any intermediate version output by any of the layers 429, 430, 440, 450, 460) is generally understood to be subdivided according to a sequence of frames (in some examples, the frames do not overlap with each other, while in other examples they may overlap). Each frame includes a series of samples. For example, each frame can be subdivided into 16 samples (but other resolutions are also possible). As mentioned above, a frame may be 10 ms long (in other cases 5 ms or 20 ms may be used or other time lengths may be used), and the sampling rate may be, for example, 16 kHz (in other cases 8 kHz, 32 kHz or 48 kHz, or any other sampling rate), and a bit rate, for example, 1.6 kbps (kilobits per second) or less than 2 kbps, or less than 3 kbps, or less than 5 kbps (in some cases the choice is left to the encoder 1, which may change the resolution and the signal encoding the resolution). It should also be noted that several frames may be grouped in one single packet of the bitstream 3, for example for transmission or for storage. Although the time length of a frame is generally considered to be fixed, the number of samples per frame may vary and upsampling operations may be performed.

[0111] The decoder (audio generator) 10 can utilize:

[0112] a frame-by-frame branch 10a′ which may be updated for each frame, e.g. using a frame obtained from the bitstream 3 (e.g. a frame may be in the form of an index quantized by a quantizer 300 and / or in the form of a code 112 (e.g. a scalar, a vector or more generally a tensor), e.g. converted from a quantization index converter 313, also called an inverse quantizer or inverse quantizer, or an index to a tensor converter); and / or

[0113] - Sample-by-sample branch 10b'.

[0114] The sample-by-sample branch 10 b ′ may include at least one of blocks 702 , 77 , and 69 .

[0115] like Figure 7As shown, an index may be obtained from a quantization index converter [or converter] 313 to obtain a code (e.g., a scalar, vector, or more general tensor) 112. The code 112 may be multi-dimensional (e.g., two-dimensional, three-dimensional, etc.) and may be understood here as being in the same format (or in a similar or similar format) as the format of the audio signal representation output by the audio signal representation generator 20. The quantization index converter 313 may therefore be understood as performing the inverse operation of the quantizer 300. The quantization index converter 313 may include (e.g., be) a learnable codebook (the quantization index converter 313 may operate deterministically using at least one learnable codebook). The quantization index converter 313 may be trained together with the quantizer, and more generally, with other elements of the encoder 2 and / or the audio generator 10. The quantization index converter 313 may operate in a frame-by-frame manner, for example, by considering a new index for each new frame to be generated. Therefore, each code (scalar, vector, or more general tensor...) 112 has the same structure for each potential representation to be quantized, without sharing exactly the same values, but sharing their approximate values.

[0116] The sample-by-sample branch 10b' may be updated for each sample, for example, at the output sampling rate and / or for each sample at a sampling rate lower than the final output sampling rate, for example using noise 14 or other input obtained from an external or internal source.

[0117] It should also be noted that the bitstream 3 is considered here to encode a single channel signal, and the output audio signal 16 and the original audio signal 1 are also considered to be single channel signals. In the case of a stereo signal or a multi-channel signal, such as a loudspeaker signal or a panoramic sound signal, all the techniques here are repeated for each audio channel (in the stereo case, there are two input audio channels 1, two output audio channels 16, etc.).

[0118] In this article, when we refer to “channels”, we must understand it in the context of convolutional neural networks, where signals are viewed as activation maps with at least two dimensions:

[0119] - a number of samples (e.g., in an abscissa dimension, or e.g., in a time axis); and

[0120] - A number of channels (eg in the ordinate direction, or eg on the frequency axis).

[0121] The first processing block 40 may operate like a conditional network, being provided with data (e.g., scalars, vectors, or more general tensors 112) from the bitstream 3 for generating conditions that modify the input data 14 (input signal). The input data (input signal) 14 (in any of its evolutions) will undergo several processings to arrive at the output audio signal 16, which is intended to be a version of the original input audio signal 1. The two conditions, the input data (input signal) 14 and its subsequently processed version, may be represented as activation maps that undergo a learnable layer (e.g., by convolution). Notably, during its evolution to speech 16, the signal 1 may undergo upsampling (e.g., from one sample 49 to multiple samples, e.g., Figure 4 ), but the number of channels 47 can be reduced (e.g., Figure 4 from 64 or 128 channels to 1 single channel).

[0122] The first data 15 may be obtained, for example, from an input (e.g., noise or a signal from an external signal) or from other internal or external sources (e.g., sample-by-sample branch 10b'). The first data 15 may be considered as an input to the first processing block 40 and may be an evolution of the input signal 14 (or may be the input signal 14). In the context of a conditional neural network (or more generally a conditionally learnable block or layer), the first data 15 may be considered as a potential signal or a previous signal. Basically, the first data 15 is modified according to the conditions set by the first processing block 40 to obtain the first output data 69. The first data 15 may be in multiple channels, for example, in a single sample. Moreover, the first data 15 provided to the first processing block 40 may have a sample resolution, but in multiple channels. Multiple channels may form a set of parameters that may be associated with encoding parameters encoded in the bitstream 3. However, in general, during processing in the first processing block 40, the number of samples per frame increases from a first number to a second higher number (i.e., the sampling rate, also referred to herein as the bit rate, increases from the first sampling rate to the second higher sampling rate). On the other hand, the number of channels may be reduced from the first number of channels to a second fewer number of channels. The conditions used in the first processing block (to be discussed in detail below) can be represented by 74 and 75 and generated by the target data 12, which in turn is generated by the target data 12 obtained from the bitstream 3 (e.g., by the quantization index 313). It will be shown that the conditions (conditioned feature parameters) 74 and 75 and / or the target data 12 can also undergo upsampling to conform to (e.g., adapt to) the dimensions of the version of the target data 12. The unit that provides the first data 15 (from an internal source, an external source, a bitstream 3, etc.) is referred to herein as a first data provider 702.

[0123] from Figure 7It can be seen that the first processing block 40 may include a pre-conditioned learnable layer 710, which may be or include a recurrent learnable layer, such as a recurrent learnable neural network, such as a GRU. The pre-conditioned learnable layer 710 may generate target data 12 for each frame. The target data 12 may be at least two-dimensional (e.g. multi-dimensional): there may be multiple samples for each frame in the second dimension, and multiple channels for each frame in the first dimension. The target data 12 may be in the form of a spectrogram, for example a Mel-spectrogram in the case where the frequency scale is uneven and / or inspired by perceptual principles. In the case where the sampling rate corresponding to the conditioned learnable layer to be fed is different from the frame rate, the target data 12 may be the same for all samples of the same frame, for example at the layer sampling rate. Another upsampling strategy may also be applied. The target data 12 may be provided to at least one conditioned learnable layer, which is indicated here as having layers 71, 72, 73 (see also Figure 3 and below). The conditional learnable layers 71, 72, 73 may generate conditions (some of which may be indicated as β, beta and γ, gamma or numbers 74 and 75), also referred to as conditional feature parameters to be applied to the first data 12, and any upsampled data derived from the first data. The conditional learnable layers 71, 72, 73 may be in the form of a matrix having multiple channels and multiple samples for each frame. The first processing block 40 may include an inverse normalization (or style element) block 77. For example, the style element 77 may apply the conditional feature parameters 74 and 75 to the first data 15. An example may be an element-by-element multiplication of the value of the first data by the condition β (which may operate as a bias) and added to the condition γ (which may operate as a multiplier). The style element 77 may produce the first output data 69 on a sample-by-sample basis.

[0124] The decoder (audio generator) 10 may include a second processing block 45. The second processing block 45 may combine multiple channels of the first output data 69 to obtain the output audio signal 16 (or its precursor audio signal 44', such as Figure 4 shown).

[0125] Now the main reference Fig. 9 The bitstream 3 is subdivided into several frames, which are then encoded in the form of indices (e.g., as obtained from the quantizer 300). From the indices of the bitstream 3, codes (e.g., scalars, vectors, or more general tensors) 112 are obtained by a quantization index converter 313. The first dimension and the second dimension are in Fig. 9The code 112 is shown in (other dimensions may be represented). Each frame is subdivided into multiple samples in the horizontal axis direction (first, intra-frame dimension). Different terms may be "frame index" and "feature map depth", "latent dimension or encoding parameter dimension" for the horizontal axis direction (first direction). In the vertical axis direction (second, intra-frame dimension), multiple channels are provided. The code 112 can be used by a pre-conditioned learnable layer 710 (e.g., a recurrent learnable layer) to generate target data 12, which can also be at least two-dimensional (e.g., multi-dimensional), such as in the form of a spectrogram (e.g., a Mel-spectrogram). Each target data 12 can represent a single frame, and the sequence of frames can evolve over time in the horizontal axis direction (from left to right) along the first intra-frame dimension. For each frame, multiple channels may be located in the vertical axis direction (second, intra-frame dimension). For example, different coefficients will appear in different entries in each column associated with the coefficients associated with the frequency band. The conditional learnable layers 71, 72, 73 generate feature parameters 74, 75 (β and γ). The abscissas of β and y (second intra-frame dimension) are associated with different samples of the same frame, while the ordinates (first intra-frame dimension) are associated with different channels. In parallel, the first data provider 702 may provide first data 15. The first data 15 may be generated for each sample and may have many channels. At the style element 77 (and more generally at the first conditioning block 40), the conditioned feature parameters β and γ (74, 75) may be applied to the first data 15. For example, an element-by-element multiplication may be performed between the columns of the style conditions 74, 75 (conditioned feature parameters) and the first data 15 or its evolution. It will be shown that this process may be repeated multiple times.

[0126] As can be clearly seen from the above, the first output data 69 generated by the first processing block 40 can be obtained as a two-dimensional matrix (or even a tensor with more than two dimensions), wherein the abscissa (first intra-frame dimension) is the sample and the ordinate (second intra-frame dimension) is the channel. By means of the second processing block 45, an audio signal 16 having a single channel and multiple samples (e.g., having a shape similar to the input audio signal 1) can be generated, in particular in the time domain. More generally, at the second processing block 45, the number of samples per frame (bit rate, also referred to as sampling rate) of the first output data 69 can evolve from a second number of samples per frame (second bit rate or second sampling rate) to a third number of samples per frame (third bit rate or third sampling rate) that is higher than the second number of samples per frame (second bit rate or second sampling rate). On the other hand, the number of channels of the first output data 69 can evolve from the second number of channels to a third number of channels that is less than the second number of channels. In other words, the bit rate or sampling rate (third bit rate or third sampling rate) of the output audio signal 16 can be higher than the bit rate (or sampling rate) of the first data 15 (first bit rate or first sampling rate) and the bit rate or sampling rate (second bit rate or second sampling rate) of the first output data 69, while the number of channels of the output audio signal 16 can be lower than the number of channels of the first data 15 (first number of channels) and the number of channels of the first output data 69 (second number of channels).

[0127] Models that process coding parameters frame by frame by concatenating the current frame with the previous frame that is already in that state are also called fluid or streaming models, and can be used as convolutional graphs for real-time convolutions as well as streaming applications such as speech coding.

[0128] Examples of convolutions are discussed below, and it will be appreciated that they may be used in the pre-conditioned learnable layer 710 (e.g., a recurrent learnable layer), any of the at least one conditional learnable layer 71, 72, 73, and more generally in the first processing block 40 (50). In general, an arriving set of conditional parameters (e.g., for a frame) may be stored in a queue (not shown) that is subsequently processed by the first processing block or the second processing block while the first processing block or the second processing block, respectively, processes the previous frame.

[0129] A discussion is now provided regarding operations performed primarily in blocks downstream of a pre-conditioned learnable layer 710 (e.g., a recurrent learnable layer). We consider target data 12 that has been obtained from the pre-conditioned learnable layer 710 and applied to conditional learnable layers 71-73 (conditional learnable layers 71-73 are in turn applied to style elements 77). Blocks 71-73 and 77 may be implemented by a generator network layer 770. The generator network layer 770 may include multiple learnable layers (e.g., multiple blocks 50a-50h, see below).

[0130] Figure 7 (and Figure 4 ) shows an example of an audio decoder (generator) 10 that can decode (e.g., generate, synthesize) an audio signal (output signal) 16 from a bitstream 3, for example according to current technology (also known as StyleMelGAN). The output audio signal 16 can be generated based on an input signal 14 (also known as a potential signal, and it can be noise, such as white noise ("first option"), or it can be obtained from another source). As described above, the target data 12 may include (e.g., be) a spectrogram (e.g., a Mel spectrogram) that provides, for example, a mapping of a sequence of time samples to a Mel scale (e.g., obtained from a pre-conditioned learnable layer 710). The target data 12 and / or the first data 15 are generally processed to obtain speech that is recognizable as natural to a human listener. In the decoder 10, the first data 15 obtained from the input is styled (e.g., at block 77) to have a vector (or more generally a tensor) having acoustic features conditioned by the target data 12. Finally, the output audio signal 16 will be recognizable as speech by a human listener. The input vector 14 and / or the first data 15 (e.g. noise, such as obtained from an internal or external source) can be as follows: Figure 4 The 128x1 vector shown (a single sample, such as a time domain sample or a frequency domain sample, and 128 channels) ( Figure 4 1 is an input signal 14 to be provided to the channel map 30, the first data provider 702 is not shown or considered to be the same as the channel map 30). In other examples, different lengths of input vectors 14 can be used. The input vector 14 can be processed in the first processing block 40 (e.g., under the condition of the target data 12 obtained from the bitstream by the pre-conditioning layer 710). The first processing block 40 may include at least one (e.g., multiple) processing blocks 50 (e.g., 50a...50h). Figure 4 , eight blocks 50a...50h are shown (each of which is also identified as a "TADEResBlock"), although a different number may be selected in other examples. In many examples, processing blocks 50a, 50b, etc. provide a gradual upsampling of the signal, evolving from the input signal 14 to the final audio signal 16 (e.g., at least some of the processing blocks, such as 50a, 50b, 50c, 50d, 50e, increase the sampling rate in such a way that each of them increases the sampling rate (also called bit rate) in the output relative to the sampling rate in its input), while other processing blocks (e.g., 50f-50h) (e.g., downstream of those that increase the sampling rate (e.g., 50a, 50b, 50c, 50d, 50e)) do not increase the sampling rate (or bit rate). Blocks 50a-50h can be understood as forming a single block 40 (e.g. Figure 7, 71, 72, 73, but different numbers are possible). In the first processing block 40, a conditional group of learnable layers (e.g., 71, 72, 73, but different numbers are possible) can be used to process the target data 12 and the input signal 14 (e.g., the first data 15). Thus, the conditional feature parameters 74, 75 (also called gamma, γ and beta, β) can be obtained during training, for example by convolution. The learnable layers 71-73 can therefore be part of the weight layers of the learning network. As described above, the first processing block 40, 50 may include at least one style element 77 (normalization block 77). At least one style element 77 can output the first output data 69 (when there are multiple processing blocks 50, multiple style elements 77 can generate multiple components, which can be added to each other to obtain a final version of the first output data 69). At least one style element 77 can apply the conditional feature parameters 74, 75 to the input signal 14 (potentially) or the first data 15 obtained from the input signal 14.

[0131] The first output data 69 may have multiple channels. The generated audio signal 16 may have one single channel.

[0132] The audio generator (eg decoder) 10 may include a second processing block 45 (in Figure 4 The second processing block 45 may be configured to combine multiple channels ( Figure 4 47 in the figure) to obtain the output audio signal 16 in a single channel, but in a series of samples (in Figure 4 In the example, the sample is indicated by 49).

[0133] "Channel" should not be understood in the context of stereo, but in the context of a neural network (e.g., a convolutional neural network) or more generally a learnable unit. For example, the input signal (e.g., potential noise) 14 can be in 128 channels (in a representation in the time domain) because a channel sequence is provided. For example, when the signal has 40 samples and 64 channels, it can be understood as a matrix with 40 columns and 64 rows, and when the signal has 20 samples and 64 channels, it can be understood as a matrix with 20 columns and 64 rows (other patterning is also possible). Therefore, the generated audio signal 16 can be understood as a single channel signal. If a stereo signal is to be generated, the disclosed technology is simply repeated for each stereo channel to obtain multiple audio signals 16 that are then mixed.

[0134] At least the original input audio signal 1 and / or the generated language 16 can be a sequence of time domain values. In contrast, the output of each (or at least one) of blocks 30 and 50a-50h, 42, 44 can generally have different dimensions (e.g., two-dimensional or other multidimensional tensors). In at least some of blocks 30 and 50a-50e, 42, 44, the signal (14, 15, 59, 69) evolving from the input 14 (e.g., noise or LPC parameters, or other parameters obtained from the bitstream) to the language 16 can be upsampled. For example, at the first block 50a in blocks 50a-50h, 2x upsampling can be performed. Examples of upsampling can include, for example, the following sequence: 1) repetition of the same value, 2) insertion of zeros, 3) another repetition or insertion of zeros + linear filtering, etc.

[0135] The generated audio signal 16 may typically be a single channel signal. If multiple audio channels are required (eg for stereo playback), the claimed process may in principle be iterated multiple times.

[0136] Similarly, the target data 12 may also have multiple channels (e.g., in a spectrogram, such as a mel-spectrogram), as generated by the pre-conditioned learnable layer 710. In some examples, the target data 12 may be upsampled (e.g., by a factor of two, a power of two, a multiple of two, or a value greater than two, such as by a different factor, such as 2.5 or a multiple thereof) to fit the dimensionality of the signal (59a, 15, 69) evolving along subsequent layers (50a-50h, 42), such as to obtain the conditioned feature parameters 74, 75 in a dimensionality appropriate to the dimensionality of the signal.

[0137] If the first processing block 40 is instantiated into a plurality of blocks (e.g., 50a-50h), the number of channels may, for example, be retained in at least some of the plurality of blocks (e.g., from 50e to 50h, and in block 42, the number of channels is unchanged). The first data 15 may have a first dimension or at least one dimension lower than the dimension of the audio signal 16. The first data 15 may have a lower total number of samples in all dimensions than the audio signal 16. The first data 15 may have a lower dimension than the audio signal 16 but a greater number of channels than the audio signal 16.

[0138] The example can be performed according to the paradigm of a generative adversarial network (GAN). The GAN includes a GAN generator 11 ( Figure 4 ) and GAN discriminator 100( Fig.10). The GAN generator 11 attempts to generate an audio signal 16 that is as close to a real audio signal as possible. The GAN discriminator 100 should identify whether the generated audio signal 16 is real or fake. Both the GAN generator 11 and the GAN discriminator 100 can be obtained as neural networks (or by other learnable techniques). The GAN generator 11 should minimize the loss (e.g., by a gradient method or other method) and update the conditioned feature parameters 74, 75 (and / or codebook) by considering the results at the GAN discriminator 100. The GAN discriminator 100 should reduce its own discrimination loss (e.g., by a gradient method or other method) and update its own internal parameters. Therefore, the GAN generator 11 is trained to generate better and better audio signals 16, while the GAN discriminator 100 is trained to identify the real signal 16 from the fake audio signal generated by the GAN generator 11. The GAN generator 11 may include the functionality of the decoder 10, but at least without the functionality of the GAN discriminator 100. Therefore, in most of the foregoing, the GAN generator 11 and the audio decoder 10 may have more or less the same features except for the features of the discriminator 100. The audio decoder 10 may include the discriminator 100 as an internal component. Therefore, the GAN generator 11 and the GAN discriminator 100 may uniformly constitute the audio decoder 10. In an example where the GAN discriminator 100 is not present, the audio decoder 10 may be constituted solely by the GAN generator 11.

[0139] As explained by the wording "a set of conditional learnable layers", the audio decoder 10 can be obtained according to the paradigm of a conditional neural network (e.g., a conditional GAN) (e.g., based on conditional information). For example, the conditional information can consist of target data (or an upsampled version thereof) 12, from which the conditional set of layers 71-73 (weight layers) are trained and conditional feature parameters 74, 75 are obtained. Thus, the style element 77 is conditioned by the learnable layers 71-73. The same applies to the pre-conditioned layer 710.

[0140] Examples at the encoder 2 (or at the audio signal representation generator 20) and / or at the decoder (or more generally the audio generator) 10 may be based on a convolutional neural network. For example, a small matrix (e.g., a filter or kernel) which may be a 3x3 matrix (or a 4x4 matrix, or 1x1, or less than 10x10, etc.) is convolved along a larger matrix (e.g., channel x upsamples the latent or input signal and / or spectrogram and / or spectrogram or upsampled spectrogram or more generally the target data 12), e.g., implying combinations (e.g., products and sums of products; dot products, etc.) between elements of the filter (kernel) and elements of the larger matrix (activation map or activation signal). During training, the elements of the filter (kernel) are obtained (learned) which are the elements that minimize the loss. During inference, the filter (kernel) elements obtained during training are used. An example of convolution may be used at at least one of blocks 71-73, 61b, 62b (see below), 230, 250, 290, 429, 440, 460. It is noteworthy that a three-dimensional tensor (or a tensor with more than three dimensions may be used instead of a matrix. In the case where the convolution is conditional, the convolution need not be applied to a signal evolving from the input signal 14 through the intermediate signal 59a (15), 69, etc. toward the audio signal 16, but may be applied to the target signal 14 (e.g., for generating conditioned feature parameters 74 and 75 to be subsequently applied to the first data 15, or potential, or preceding, or evolving from the input signal toward the speech 16). In other cases (e.g., at blocks 61b, 62b, see below), the convolution may be unconditional and may, for example, be applied directly to the signal 59a (15), 69, etc. evolving from the input signal 14 toward the audio signal 16. Both conditional and unconditional convolutions may be performed.

[0141] In some examples (at the decoder or at the encoder) there may be an activation function downstream of the convolution (ReLu, TanH, softmax, etc.), which may be different depending on the desired effect. ReLu may map the maximum value between 0 and the value obtained at the convolution (in practice, if it is positive, it keeps the same value, and if it is negative, it outputs 0). LeakyReLu may output x if x>0, and 0.1*x if x≤0, where x is the value obtained by the convolution (other values ​​may be used in some cases instead of 0.1, such as a predetermined value within 0.1±0.05). TanH (which may be implemented, for example, at blocks 63a and / or 63b) may provide the hyperbolic tangent of the value obtained at the convolution, for example:

[0142] TanH(x)=(e x -e -x ) / (e x +e -x ),

[0143] Where x is the value obtained at the convolution (e.g., at block 61b, see below). Softmax (e.g., applied at block 64b) can apply an exponential to each of the elements of the convolution result and normalize it by dividing by the sum of the exponentials. Softmax can provide a probability distribution for the entries in the matrix produced by the convolution (e.g., as provided at 62b). After applying the activation function, a pooling step (not shown in the figure) can be performed in some examples, but it can be avoided in other examples. There can also be a softmax gated TanH function, for example, by multiplying the result of the TanH function (e.g., obtained at 63b, see below) with the result of the softmax function (e.g., obtained at 64b, see below) (e.g., obtained at 65b, see below). In some examples, multiple layers of convolution (e.g., a set of conditional learnable layers, or at least one conditional learnable layer) can be one downstream of another and / or in parallel with each other, thereby increasing efficiency. If activation functions and / or application of pooling are provided, they may also be repeated in different layers (or, for example, different activation functions may be applied to different layers) (this may also apply to the encoder).

[0144] At the decoder (or more generally the audio generator) 10, the input signal 14 is processed at different steps to become the generated audio signal 16 (e.g. under the conditions set by the learnable layer or conditional learnable layer group 71-73, and on the parameters 74, 75 learned by the set of conditional learnable layers or learnable layers 71-73). Therefore, the input signal 14 (or its evolved version, i.e. the first data 15) can be understood as being processed in the processing direction (from Figure 4 and Figure 7 14 to 16 in the example) towards becoming a generated audio signal 16 (eg speech). These conditions will be generated substantially based on the target signal 12 and / or the prerequisites in the bitstream 3 and the training (thereby arriving at the most preferred parameter set 74, 75).

[0145] It should also be noted that multiple channels of the input signal 14 (or any evolution thereof) can be considered to have a set of learnable layers and style elements 77 associated therewith. For example, each row of matrices 74 and 75 can be associated to a specific channel of the input signal (or one of its evolutions), such as obtained from a specific learnable layer associated to the specific channel. Similarly, style element 77 can be considered to be formed by multiple style elements (one for each row of the input signal x, c, 12, 76, 76', 59, 59a, 59b, etc.).

[0146] Figure 4 An audio decoder (or more generally an audio generator) 10 (which may embody Figure 6), and it may also include (e.g. be) a GAN generator 11 (see below). Although the target data 12 is obtained from the bitstream 3 through the pre-conditioned learnable layer 710 (see above), Figure 4 Now it is indeed shown that the pre-conditioned learnable layer 710 (eg Figure 7 ). The target data 12 may be a mel-spectrogram (or other tensor) obtained from the preconditioned learnable layer 710 (but they may be other kinds of tensors); the input signal 14 may be potential (previous) noise or a signal obtained from an internal or external source, and the output 16 may be speech. The input signal 14 may have only one sample and multiple channels (denoted as "x" because they may vary, e.g. the number of channels may be 80 or other). The input vector 14 may be obtained in a vector with 128 channels (but other numbers are possible). In the case where the input signal 14 is noise ("first option"), it may have a zero-mean normal distribution and follow the following formula: It could be random noise of dimension 128, mean 0, and the autocorrelation matrix (square 128x128) equal to identity I (different choices can be made). Thus, in the example, noise is used as input signal 14, which could be completely uncorrelated between channels and have a variance (energy) of 1. This can be achieved at every 22528 generated samples (or other numbers may be selected for different examples); thus, the dimension on the time axis may be 1, and the dimension on the channel axis may be 128. In an example, the input signal 14 may be a constant value.

[0147] The input vector 14 can be processed step by step (e.g., at blocks 702, 50a-50h, 42, 44, 46, etc.) to evolve into speech 16 (the evolved signal will be indicated, for example, with different signals 15, 59a, x, c, 76', 79, 79a, 59b, 79b, 69, etc.).

[0148] At block 30, channel mapping may be performed. It may consist of or include a set of simple convolutional layers to change the number of channels, for example from 128 to 64 in this case. Block 30 may thus be learnable (in some examples, it may be deterministic). It can be seen that processing blocks 50a, 50b, 50c, 50d, 50e, 50f, 50g, 50h (together embodying Figure 6At least some of the first processing blocks 50) may increase the number of samples, for example for each frame, by performing upsampling (e.g., up to 2 upsampling). Along blocks 50a, 50b, 50c, 50d, 50e, 50f, 50g, 50h, the number of channels may remain the same (e.g., 64). For example, the sample may be the number of samples per second (or other time unit): we may obtain a sound of 16kHz or higher (e.g., 22Khz) at the output of block 50h. As described above, a sequence of multiple samples may constitute a frame. Each of the blocks 50a-50h (50) may also be a TADEResBlock (TADE, residual block in the context of time adaptive denormalization). It is noteworthy that each block 50a-50h (50) may be conditioned by target data (e.g., code, which may be a tensor, such as a multidimensional tensor, e.g., having 2, 3 or more dimensions) 12 and / or by a bitstream 3. In the second processing block 45 (Figures 1 and Figure 6 ), only a single channel is available, and multiple samples are obtained in a single dimension (see also Fig. 9 ). It can be seen that another TADEResBlock 42 (in addition to blocks 50a-50h) can be used (which reduces the dimensionality to four single channels). Then, a convolutional layer 44 and an activation function (which can be, for example, TanH 46) can be performed. A (pseudo-orthogonal mirror filter) bank 110 can also be applied, thereby obtaining the final output 16 (and possibly being stored, rendered, etc.).

[0149] At least one of the blocks 50a-50h (or each of them, in a specific example) and 42, as well as the encoder layers 230, 240 and 250 (and 430, 440, 450, 460) can be, for example, residual blocks. The residual learnable block (layer) can operate a prediction on a residual component of a signal that evolves from an input signal 14 (e.g., noise) to an output audio signal 16. The residual signal is just a portion (residual component) of the main signal that evolves from the input signal 14 toward the output signal 16. For example, multiple residual signals can be added to each other to obtain the final output audio signal 16. However, other architectures can be used.

[0150] Figure 3An example of one of the blocks 50a-50h (50) is shown. Although when training, they may result in that the blocks 50a-50h (50) may be duplicates of each other. As can be seen, each block 50 (50a-50h) is input with first data 59a, which may be the first data 15 (or an upsampled version thereof, such as data output by the upsampling block 30) or an output from a previous block. For example, block 50b may be input with the output of block 50a; block 50c may be input with the output of block 50b, and so on. In an example, different blocks may operate in parallel with each other, and the results added together. From Figure 3 It can be seen that the first data 59a provided to the block 50 (50a-50h) or 42 is processed and its output is the output data 69 (which will be provided as the input of the subsequent block). As shown by line 59a', the principal component of the first data 59a actually bypasses most of the processing of the first processing block 50a-50h (50). For example, blocks 60a, 900, 60b and 902 and 65b are bypassed by the principal component 59a'. The residual component 59a of the first data 59 (15) can be processed to obtain the residual part 65b' to be added to the adder 65c (which is in Figure 3 , but not shown). Bypassing the main component 59a' and the addition at the adder 65c can be understood as instantiating the fact that each block 50 (50a-50h) processes an operation on a residual signal, which is then added to the main part of the signal. Therefore, each of the blocks 50a-50h can be considered to be a residual block. The addition at the adder 65c does not necessarily need to be performed within the residual block 50 (50a-50h). A single addition of multiple residual signals 65b' (each output by each of the residual blocks 50a-50h) can be performed (for example, at a single adder block in the second processing block 45). Therefore, different residual blocks 50a-50h can operate in parallel with each other. Figure 3In the example of , each block 50 (50a-50h) can repeat its convolution layer twice. The first denormalization block 60a and the second denormalization block 60b can be used in cascade. The first denormalization block 60a may include an instance of the style element 77 to apply the conditional feature parameters 74 and 75 to the first data 59 (15) (or its residual version 59a). The first denormalization block 60a may include a normalization block 76. The normalization block 76 may perform normalization along the channels of the first data 59 (15) (e.g., its residual version 59a). Therefore, a normalized version c (76') of the first data 59 (15) (or its residual version 59a) may be obtained. Therefore, the style element 77 may be applied to the normalized version c (76') to obtain a denormalized (conditioned) version of the first data 59 (15) (or its residual version 59a). The denormalization at the style element 77 can be obtained, for example, by element-wise multiplication of the elements of the matrix (or more generally, the tensor) γ (which embodies the condition 74) and the signal 76' (or another version of the signal between the input signal and the speech), and / or by element-wise addition of the elements of the matrix (or more generally, the tensor) β (which embodies the condition 75) and the signal 76' (or another version of the signal between the input signal and the speech). Thus, a denormalized version 59b (conditioned by the conditioned feature parameters 74 and 75) of the first data 59 (15) (or its residual version 59a) can be obtained.

[0151] Then, a gated activation 900 may be performed on a denormalized version 59b of the first data 59 (e.g., a residual version 59a thereof). Specifically, two convolutions 61b and 62b (e.g., each having a 3x3 kernel and having a dilation factor of 1) may be performed. Different activation functions 63b and 64b may be applied to the results of the convolutions 61b and 62b, respectively. Activation 63b may be TanH. Activation 64b may be softmax. The outputs of the two activations 63b and 64b may be multiplied with each other to obtain a gated version 59c of the denormalized version 59b of the first data 59 (or its residual version 59a). Subsequently, a second denormalization 60b may be performed on the gated version 59c of the denormalized version 59b of the first data 59 (or its residual version 59a). The second denormalization 60b may be similar to the first denormalization, and therefore will not be described here. Subsequently, a second activation 902 may be performed. Here, the kernel may be 3x3, but the dilation factor may be 2. In any case, the dilation factor of the second gated activation 902 may be greater than the dilation factor 900 of the first gated activation. A set of conditioned learnable layers 71-73 (e.g., obtained from pre-conditioned learnable layers) and style elements 77 may be applied (e.g., twice for each block 50a, 50b, ...) to the signal 59a. Upsampling of the target data 12 may be performed at the upsampling block 70 to obtain an upsampled version 12' of the target data 12. The upsampling may be obtained by nonlinear interpolation, and may use, for example, a factor of 2, a power of 2, a multiple of 2, or another value greater than 2. Thus, in some examples, the spectrogram (e.g., a mel-spectrogram) 12' may be made to have the same dimensions (e.g., conforming) as the signal conditioned by the spectrogram (e.g., 76, 76', c, 59, 59a, 59b, etc.). In an example, the first convolution and the second convolution at 61b and 62b downstream of the TADE block 60a or 60b, respectively, may be performed at the same number of elements in the kernel (e.g., 9, e.g., 3x3). However, the second convolution in block 902 may have a dilation factor of 2. In an example, the maximum dilation factor for a convolution may be 2 (two).

[0152] As described above, the target data 12 may be upsampled, for example, so as to conform to the input signal (or a signal evolved therefrom, such as 59, 59a, 76', also referred to as a potential signal or an activation signal). Here, convolutions 71, 72, 73 may be performed (intermediate values ​​of the target data 12 are indicated by 71') to obtain parameters γ (gamma, 74) and β (beta, 75). The convolution at any of 71, 72, 73 may also require a rectified linear unit ReLu or a leaky rectified linear unit leaky ReLu. The parameters γ and β may have the same dimensions as the activation signal (which is processed to evolve from the input signal 14 to the generated audio signal 16, which is represented here as x, 59, 59a or 76' when in normalized form). Therefore, when the activation signal (x, 59, 59a, 76') has two dimensions, γ and β (74 and 75) also have two dimensions, and each of them may be superimposed on the activation signal (the length and width of γ and β may be the same as the length and width of the activation signal). At style element 77, the conditioned feature parameters 74 and 75 are applied to the activation signal (which may be the first data 59a or 59b output by multiplier 65a). However, it should be noted that the activation signal 76' may be a normalized version of the first data 59, 59a, 59b (15) (at example standard block 76), the normalization being in the channel dimension. It should also be noted that the formula (γ*c+β, also in Figure 3 denoted by γ⊙c+β in the example) may be an element-wise product and in some examples is not a convolutional product or a dot product. Convolutions 72 and 73 do not necessarily have activation functions downstream thereof. Parameter γ(74) may be understood as having a variance value, and β(75) may be understood as having a bias value. Note that for each block 50a-50h, 42, the learnable layers 71-73 (e.g., together with the style element 77) may be understood as embodying weight layers. In addition, Figure 4 The block 42 can be instantiated as Figure 3 The block 50 is then, for example, convolutional layer 44 reduces the number of channels to 1, and thereafter, TanH 46 is performed to obtain speech 16. The output 44' of blocks 44 and 46 may have a reduced number of channels (e.g. 4 channels instead of 64), and / or may have the same number of channels as the preceding block 50 or 42 (e.g. 40).

[0153] A PQMF synthesis (see also below) 110 is performed on the signal 44', thereby obtaining the audio signal 16 in one channel.

[0154] In an example, the bitstream (3) may be transmitted (e.g., via a communication medium, such as a wired connection and / or a wireless connection), and / or may be stored (e.g., in a storage unit). Thus, the encoder 3 and / or the audio signal representation generator 20 may include and / or be connected and / or configured to control a transmission unit (e.g., a modem, a transceiver, etc.) and / or a storage unit (e.g., a mass storage device, etc.). To allow storage and / or transmission, there may be other devices between the quantizer 300 and the converter 313 that process the bitstream for the purpose of storage and / or transmission, as well as reading and / or receiving.

[0155] Quantization and conversion from index to code using learnable techniques

[0156] The operation of the quantizer 300 when the quantizer 300 is a learnable quantizer is discussed here, as well as the operation of the quantization index converter 313 (inverse or reverse quantizer) when the quantization index converter 313 is a learnable quantization index converter. Note that the quantizer 300 can be input with a scalar, a vector, or more generally a tensor. The quantization index converter 313 can convert the index on at least one code (which is taken from a codebook, which can be a learnable codebook). It should be noted that in some examples, the learnable quantizer 300 and the quantization index converter 313 can use such deterministic quantization / dequantization, but use at least one learnable codebook.

[0157] Here, the following conventions are used:

[0158] x is the speech (or more generally the input signal 1)

[0159] E(x) is the output of the audio signal generator 20 (e.g., 269), (i.e., x after being processed by the learnable block 200 (DualPathConvRNN) and / or at least one convolutional learnable block 290 (ConvEncoder)), which can be a vector, or more generally a tensor

[0160] An index (e.g., i) that refers to (e.g., points to) a code (e.g., z, r, q) z 、i r 、i q ) is in at least one codebook (e.g. z e 、r e ,q e )middle

[0161] Index (e.g., i z 、i r 、i q ) is written into the bitstream 3 by the learnable quantizer 300 (or more generally by the encoder 2) and read by the quantization index converter 313 (or more generally by the audio decoder 10).

[0162] The primary code (e.g. z) is chosen in such a way as to approximate E(x)

[0163] The first (if any) residual code (e.g. r) is chosen in such a way as to approximate the residual E(x)-z

[0164] The second (if any) residual code (e.g. q) is chosen in such a way as to approximate the residual E(x)-zr

[0165] The decoder 3 (e.g., the quantization index converter 313) reads the index (e.g., i z 、i r 、i q ), obtain the codes (e.g. z, r, q), and reconstruct a tensor (e.g. a tensor representing a frame in the first audio signal representation 220 of the first audio signal 1), for example by summing the codes (e.g. z+r+q) as the tensor 112.

[0166] • Jittering may be added (eg, after obtaining the tensor 112, and / or before the preconditioning layer 710) to avoid potential clustering effects.

[0167] The learnable quantizer (300) of the encoder 2 may be configured to associate an index of a code read from at least one codebook (e.g., a learnable codebook) in the bitstream 3 to each frame of the first multidimensional audio signal representation (e.g., 220) of the input audio signal 1 or another processed version (e.g., 269, 469, etc.) of the input audio signal 1, thereby generating the bitstream 3. The learnable quantizer 300 may associate the code closest to the tensor (e.g., the code that minimizes the distance from the tensor) to each frame (e.g., a tensor) of the first multidimensional audio signal representation (e.g., 220) of the input audio signal 1 or the processed version of the first multidimensional audio signal representation (e.g., output by block 290), thereby writing the index (which is associated in the codebook to the code that minimizes the distance) in the bitstream 3.

[0168] As described above, at least one codebook may be defined based on the residual technique. For example, there may be:

[0169] 1) Main (basic) codebook z e can be defined as having multiple codes such that a particular code z∈z in the codebook e is selected, which is associated with and / or approximates a major portion of the frame E(x) (input vector) output by block 290;

[0170] 2) An optional first residual codebook r with multiple codes can be defined e , so that a specific code r∈r eis selected, which approximates (e.g., best approximates) the residual E(x)-z of the main part of the input vector E(x);

[0171] 3) An optional second residual codebook q with multiple codes can be defined e , so that a specific code q∈q e is chosen, which approximates the first-order residual E(x)-z e -r e ;

[0172] 4) Possible optional further lower ranked residual codebook.

[0173] The codes of each learnable codebook can be indexed according to the index, and the association between each code in the codebook and the index can be obtained through training. Written in the bitstream 3 is the index for each part (main part, first residual part, second residual part). For example, we may have:

[0174] 1) Point to z∈z e The first index i z

[0175] 2) Point to the first residual r∈r e The second index i r

[0176] 3) Points to the second residual q∈q e The third index i r

[0177] Although the codes z, r, q may have the dimensions of the output E(x) of the audio signal representation generator 20 for each frame, the index i z 、i r 、i q It may be an encoded version of them (eg a bit string, eg 10 bits).

[0178] Therefore, at the quantizer 300, there may be multiple residual codebooks such that:

[0179] The second residual codebook q e associating a code (e.g. a scalar, a vector or more generally a tensor) representing a second residual part of the first multidimensional audio signal representation of the input audio signal to an index to be encoded in the audio signal representation,

[0180] The first residual codebook r e associating a code representing a first residual part of a frame of the first multidimensional audio signal representation to an index to be encoded in the audio signal representation,

[0181] The second residual portion of the frame is residual [eg, low ranked] with respect to the first residual portion of the frame.

[0182] At the same time, the audio generator 10 (eg, the decoder, or in particular the quantization index converter 313) may perform the inverse operation. The audio generator 10 may have a learnable codebook that can convert the index (eg, i z 、i r 、i q ) is converted from the code in the learnable codebook to the code (e.g., z, r, q). For example, in the above residual case, for each frame of bitstream 3, the bitstream can present:

[0183] 1) represents the code z∈z e The primary index i z , used to convert the index (code) i z Transformed onto the code z, thus forming the main part z of the tensor (e.g., vector) that approximates E(x)

[0184] 2) represents the code r∈r e The first residual index (second index) i r , for starting from index i r is converted to the code r, thereby forming the first residual part of a tensor (e.g., a vector) approximating E(x)

[0185] 3) represents the code q∈r q The second residual index (third index) i q , for starting from index i q is converted to the code q, thereby forming a second residual part of a tensor (e.g., a vector) similar to E(x)

[0186] Then, a code version (tensor version) 112 of the frame may be obtained, for example as the sum z+r+q. Dithering may then be applied to the obtained sum.

[0187] It should be noted that solutions based on certain types of quantization may also be used without implementing the pre-conditioned learnable layer 710 as an RNN. This may also apply to the case where the pre-conditioned learnable layer 710 does not exist or is a deterministic layer.

[0188] GAN Discriminator

[0189] Fig.10 The GAN discriminator 100 can be used during training to obtain, for example, parameters 74 and 75 to be applied to the input signal 12 (or a processed and / or normalized version thereof). Training can be performed before inference, and the parameters (e.g., 74, 75 and / or at least one learnable codebook) can be stored, for example, in a non-transitory memory and used later (however, in some examples, the parameters 74 or 75 can also be calculated online).

[0190] The role of the GAN discriminator 100 is to learn how to recognize a generated audio signal (e.g., the synthesized audio signal 16 as described above) from a real input signal (e.g., real speech) 104. Therefore, the role of the GAN discriminator 100 is mainly played during the training session (e.g., for learning parameters 72 and 73) and is regarded as the opposite of the role of the GAN generator 11 (which can be regarded as the audio decoder 10 without the GAN discriminator 100).

[0191] In general, the GAN discriminator 100 can be input by both the synthesized audio signal 16 generated by the GAN decoder 10 (and obtained from the bitstream 3, which in turn is generated from the input audio signal 1 by the encoder 2) and the real audio signal (e.g., real speech) 104 acquired by a microphone or from another source, and processes the signals to obtain a metric (e.g., loss) to be minimized. The real audio signal 104 can also be considered as a reference audio signal. During training, the operations explained above for synthesizing speech 16 can be repeated, for example, multiple times, thereby obtaining, for example, parameters 74 and 75.

[0192] In an example, instead of analyzing the entire reference audio signal 104 and / or the entire generated audio signal 16, only a portion thereof (e.g., a portion, a slice, a window, etc.) may be analyzed. A signal portion generated in a random window (105a-105d) sampled from the generated audio signal 16 and from the reference audio signal 104 is obtained. For example, a random window function may be used so that it is not a priori predefined which window 105a, 105b, 105c, 105d will be used. In addition, the number of windows is not necessarily four and may vary.

[0193] Within the window (105a-105d), a PQMF (Pseudo Quadrature Mirror Filter) bank 110 may be applied. Thus, a subband 120 is obtained. Thus, a decomposition (110) of a representation of the generated audio signal (16) or a representation of the reference audio signal (104) is obtained.

[0194] An evaluation block 130 may be used to perform the evaluation. A plurality of evaluators 132a, 132b, 132c, 132d (complex representation with 132) may be used (different numbers may be used). In general, each window 105a, 105b, 105c, 105d may be input to a corresponding evaluator 132a, 132b, 132c, 132d. For each evaluator (132a-132d), the sampling of the random window (105a-105d) may be repeated multiple times. In an example, the number of times the random window (105a-105d) is sampled for each evaluator (132a-132d) may be proportional to the length of the representation of the generated audio signal or the representation of the reference audio signal (104). Thus, each of the evaluators (132a-132d) may receive as input a representation of the generated audio signal (16) or one or several parts (105a-105d) of a representation of the reference audio signal (104).

[0195] Each evaluator 132a-132d may itself be a neural network. In particular, each evaluator 132a-132d may follow the paradigm of a convolutional neural network. Each evaluator 132a-132d may be a residual evaluator. Each evaluator 132a-132d may have parameters (e.g., weights) that are adapted during training (e.g., in a manner similar to one of the manners explained above).

[0196] like Fig.10 As shown, each evaluator 132a-132d also performs downsampling (e.g., by 4 or by another downsampling ratio). For each evaluator 132a-132d, the number of channels can be increased (e.g., by 4, or in some examples by the same number as the downsampling ratio).

[0197] Upstream and / or downstream of the evaluator, convolutional layers 131 and / or 134 may be provided. The upstream convolutional layer 131 may have, for example, a kernel of dimension 15 (e.g., 5x3 or 3x5). The downstream convolutional layer 134 may have, for example, a kernel of dimension 3 (e.g., 3x3).

[0198] During training, a loss function (adversarial loss) 140 may be optimized. The loss function 140 may include a fixed metric between the generated audio signal (16) and the reference audio signal (104) (e.g., obtained during a pre-training step). The fixed metric may be obtained by computing one or more spectral distortions between the generated audio signal (16) and the reference audio signal (104). The distortion may be measured by taking into account the following factors:

[0199] - the magnitude or logarithmic magnitude of the spectral representation of the generated audio signal (16) and the reference audio signal (104), and / or

[0200] - Different time or frequency resolution.

[0201] In an example, the adversarial loss may be obtained by randomly providing and evaluating a representation of a generated audio signal (16) or a representation of a reference audio signal (104) by one or more evaluators (132). The evaluation may include classifying the provided audio signal (16, 132) into a predetermined number of categories, a pre-trained classification level indicating the naturalness of the audio signal (14, 16). The predetermined number of categories may be, for example, "true" versus "false".

[0202] An example of loss can be obtained as follows,

[0203]

[0204] in:

[0205] x is the real voice 104,

[0206] z is a potential input 14 (which may be noise or another input obtained from the bitstream 3),

[0207] s is a tensor representing x (or more generally the target signal 12).

[0208] D(...) is the output of the estimator in terms of probability distribution

[0209] (D(...)=0 means "definitely false", D(...)=1 means "definitely true").

[0210] Spectral reconstruction loss Still used for regularization to prevent adversarial artifacts. The final loss can be, for example:

[0211]

[0212] where each i is the contribution at each evaluator 132a-132d (e.g., each evaluator 132a-132d provides a different D i )and is the pre-training (fixed) loss.

[0213] During training, search The minimum value of can be expressed as

[0214]

[0215] Other types of minimization may be performed.

[0216] In general, the minimum adversarial loss 140 is associated with the optimal parameters (eg, 74 , 75 ) to be applied to the style element 77 .

[0217] 1) It should be noted that the training session and the encoder 2 (or at least the audio signal representation generator 20) can be trained together with the decoder 10 (or more generally the audio generator 10). Therefore, together with the parameters of the decoder 10 (or more generally the audio generator 10), the parameters of the encoder 2 (or at least the audio signal representation generator 20) can also be obtained. In particular, at least one of the following can be obtained by training: the weights of the learnable layers 230, 250 (e.g., kernels);

[0218] 2) The weights of the cyclically learnable layer 240

[0219] 3) The weights of the learnable block 290, including the weights (e.g., kernels) of layers 429, 440, 460

[0220] 4) Codebook (e.g. z e 、r e ,q e At least one of ) is used by the learnable quantizer 300 (in pairs with the codebook of the quantization index converter 313).

[0221] A common way to train the encoder 2 and decoder 10 together with the others is to use a GAN, where the discriminator 100 should distinguish between:

[0222] An audio signal 16 generated from frames in the bitstream 3 actually generated by the encoder 1; and

[0223] An audio signal 16 is generated from frames in a bitstream not generated by the encoder 1 .

[0224] Generation of at least one codebook

[0225] Note in particular that the codebook (e.g., z e 、r e ,q e At least one of ), it should be noted that there may be different methods of defining the codebook.

[0226] During a training session, multiple bitstreams 3 may be generated by the learnable quantizer 300 and obtained by the quantization index converter 313. The index (e.g., i z 、i r 、i q) is written into a bitstream (3) to encode a known frame representing a known audio signal. A training session may comprise evaluating an audio signal 16 generated at the decoder 2 with respect to a known input audio signal 1 provided to the encoder 2: adapting the association of indices of at least one codebook to the frames of the bitstream being encoded [e.g. by minimizing the difference between the generated audio signal 16 and the known audio signal 1].

[0227] In the case of using GAN, the discriminator 100 should discriminate:

[0228] An audio signal 16 generated from frames in the bitstream 3 actually generated by the encoder 1; and

[0229] An audio signal 16 is generated from frames in a bitstream generated by the non-encoder 1 .

[0230] It is worth noting that during a training session, the index length used for each index can be defined (e.g., 10 bits instead of 15 bits). Thus, training can provide at least:

[0231] a plurality of first bitstreams (e.g., generated by encoder 2) having first candidate indices having a first bit length and associated with first known frames representing a known audio signal, the first candidate indices forming a first candidate codebook, and

[0232] A plurality of second bitstreams having second candidate indices having a second bit length and associated with known frames representing the same first known audio signal, the second candidate indices forming a second candidate codebook.

[0233] The first bit length may be higher than the second bit length [and / or the first bit length has a higher resolution but it occupies more frequency band than the second bit length]. The training session may include an evaluation of a generated audio signal obtained from a plurality of first bit streams compared to a generated audio signal obtained from a plurality of second bit streams, thereby selecting a codebook [e.g., such that the selected learnable codebook is a codebook selected between a first candidate codebook and a second candidate codebook] [e.g., may be an evaluation of a first ratio compared to a second ratio, wherein the first ratio is a ratio between a metric measuring the quality of the audio signal generated from the plurality of first bit streams with respect to bit length, and the second ratio is a ratio between a metric measuring the quality of the audio signal generated from the plurality of second bit streams with respect to bit rate (also known as sampling rate), and selecting the bit length that maximizes the ratio] [e.g., this may be repeated for each codebook, e.g., main, first residual, second residual, etc.]. The discriminator 100 may evaluate whether the output signal 16 generated using the second candidate codebook with a low bit length index looks similar to the output signal 16 generated using the pseudo bitstream 3 (e.g., by evaluating and / or the error rate at the discriminator 100), and in the affirmative case, the second candidate codebook with a low bit length index will be selected; otherwise, the first candidate codebook with a high bit length index will be selected.

[0234] Additionally or alternatively, a training session may be performed using:

[0235] a first plurality of first bitstreams having first indices associated with first known frames representing a known audio signal, wherein the first indices have a first maximum number, the first plurality of first candidate indices forming a first candidate codebook; and

[0236] A second plurality of second bitstreams having second indices associated with known frames representing the same first known audio signal, the second plurality of second candidate indices forming a second candidate codebook, wherein the second indices have a second maximum number different from the first maximum number.

[0237] The training session may include an evaluation of a generated audio signal 16 obtained from a first plurality of first bitstreams 3 compared to a generated audio signal 16 obtained from a second plurality of second bitstreams 3, thereby selecting a learnable index [e.g., such that the selected learnable codebook is selected from a first candidate codebook and a second candidate codebook] [e.g., there may be an evaluation of a first ratio compared to a second ratio, the first ratio being a ratio between a metric measuring the quality of the audio signal generated from the first plurality of first bitstreams, the second ratio being a ratio between a metric measuring the quality of the audio signal generated from the second plurality of second bitstreams with respect to bitrate (or sampling rate), and selecting the plurality that maximizes the ratio among the first plurality and the second plurality] [e.g., this may be repeated for each codebook, e.g., primary, first residual, second residual, etc.]. In the second case, different candidate codebooks have different numbers of codes (and indices pointing to the codes), and the discriminator 100 may evaluate whether a low number of codes is desired or a high number of codes is desired (e.g., by evaluating and / or the error rate at the discriminator 100).

[0238] In some cases, it may be possible to decide which resolution to use (e.g., how many low-ranking codebooks to use). This may be achieved, for example, by using a bitstream such as:

[0239] a first plurality of first bitstreams having first indices representing codes obtained from a known audio signal, the first plurality of first bitstreams forming at least one first codebook [eg, at least one primary codebook]; and

[0240] A second plurality of second bit streams, comprising both first indices representing primary codes obtained from a known audio signal and second indices representing residual codes with respect to the primary codes, the second plurality of second bit streams forming at least one first codebook [e.g., at least one primary codebook z e ] and at least one second codebook [e.g. at least one residual codebook r e ].

[0241] The training session may include an evaluation of a generated audio signal obtained from the first plurality of first bitstreams compared to a generated audio signal obtained from a second plurality of second bitstreams. The discriminator 100 may choose to use:

[0242] Only low resolution encoding (eg, only primary code), with only the first plurality [and / or the first candidate codebook z e ] and the second plurality of [ and / or the first candidate codebook as the main codebook, together with at least one second codebook as the residual codebook r e ][eg, such that the selected learnable codebook is selected between the first candidate codebook and the second candidate codebook] (the use of the second plurality may imply the use of a residual codebook that is lower ranked with respect to the first plurality).

[0243] [For example, there may be an evaluation of a first ratio compared to a second ratio, the first ratio being a ratio between metrics measuring the quality of an audio signal generated from a first plurality of first bitstreams, the second ratio measuring a ratio between metrics measuring the quality of an audio signal generated from a second plurality of second bitstreams with respect to bitrate (or sampling rate), and selecting a plurality of the first plurality and the second plurality that maximizes the ratio][For example, this may be repeated for each codebook, e.g., main, first residual, second residual, etc.].

[0244] In some examples, the discriminator 100 will evaluate A threshold of minimum value of and / or error rate may be used to select a low-resolution multiple (e.g., only the primary codebook), or otherwise a second multiple (higher resolution, but high payload in the bitstream) is necessary.

[0245] Recurrent Learnable Layers

[0246] The learnable layer 240 of the encoder (e.g., the audio signal representation generator 20) may be of a recurrent type (the same may apply to the pre-conditioned learnable layer 710). In this case, the output of the learnable layer 240 and / or the pre-conditioned learnable layer 710 for each frame may be conditioned by the output of the previous frame. For example, for each t-th frame, the output of the learnable layer 240 may be f(t, t-1, t-2, ...), where the parameters of the function f() may be obtained by training. The function f() may be linear or non-linear (e.g., a linear function with an activation function). For example, there may be weights W0, W1, and W2 (where W0, W1, and W2 are obtained by training) such that if the output 240 for frame t-1 is F t-1 and the output 240 for frame t-2 is F t-2 , then the output F for frame t t F t =W0*F t-1 +W1*F t-2 +W2*F t-3 , output F for frame t+1 t+1 F t+1 =W0*F t +W1*F t-1 +W2*F t-2 Therefore, the output F of the learnable layer 240 for a given frame t is t It may be conditioned by, for example, at least one previous frame (e.g., t-1, t-2, etc.) before (e.g., immediately before) a given frame t. In some cases, the output value of the learnable layer 240 for a given frame t may be obtained by a linear combination (e.g., by weights W0, W1, and W2) of the previous frame (e.g., immediately before) before the given frame t.

[0247] It is worth noting that each frame may have some samples obtained from the immediately previous frame, and this simplifies the operation.

[0248] In the example, the GRU can be operated in this way. Other types of GRUs can be used. Fig.11 An example of a GRU that may be used is shown (eg, in layer 240 and / or in a pre-conditioned learnable layer 710).

[0249] In general, a recurrent learnable layer (e.g., a GRU, which can be an RNN) can be viewed as a learnable layer with state, so that each time step is conditioned not only by the output, but also by the state of the immediately previous time step. Thus, a recurrent learnable layer can be understood as being spread out in multiple feedforward modules (one feedforward module per time step), in such a way that each feedforward module inherits the state of the immediately preceding feedforward module (while the first feedforward module may input a default state).

[0250] exist Fig.11 In FIG. 1 , a single GRU 1100 is shown. A GRU (or a cascade of GRUs) may form, for example, a learnable layer 240 of an encoder and / or a preconditioned learnable layer 710 of a decoder. Fig.12 Note that a single GRU or recurrent unit 1100 can be in the feed-forward module (1100 t-1 , 1100t, 1100 t+1 etc.), removing its backward path. In this case, the tth module of the GRU is after the (t-1)th module (which accepts its output state as input) and before the (t+1)th module by transmitting its state.

[0251] Alternatively, you can use a cascade of loop modules (such as Fig.12 ), where each GRU or recurrent unit will maintain its own state independently. In this case, the GRUs can be built one after another, and the output of one GRU is passed to the input of the next GRU. Another option might be to connect the states between the cascaded recurrent units also through a mechanism such as attention.

[0252] For example, these relationships may be controlled by formulas such as at least one of the following:

[0253] z t =σ(W z ·(h t-1 , x t ))

[0254] r t =σ(W r ·(h t-1 , x t ))

[0255]

[0256] in:

[0257] t refers to the moment / step, and for an unfolded GRU, also refers to a specific module in the unfolded structure (e.g. t=0 is the first module, the first moment, t=1 is the second moment, etc.);

[0258] x t refers to the input vector of the recurrent module at time t (e.g., for a frame at time t, e.g., with or without samples taken from the (e.g., immediately preceding) frame and / or with or without samples taken from the (e.g., immediately preceding) frame);

[0259] h tRefers to the state and output of the recurrent unit at time t. In the case of unfolding, it will be inherited by the (t+1)th feedforward module (refer to Fig.11 ,h t Reintroduced in feedback as h t-1 , see below; reference Fig.12 ,h t is provided to the immediately following feed-forward module);

[0260] h t-1 refers to the state and output of time step t-1, which is the input of the unit at time t. In the case of unfolded GRU ( Fig.12 ), h t-1 is the input to the t-th feed-forward module (i.e., either the output of the immediately preceding recurrent module or the input to the GRU) (if the t-th module is the first module, then h t-1 will be the default value);

[0261] refers to candidate states and / or outputs of a loop module;

[0262] z t refers to the update gate vector;

[0263] r t Refers to the reset gate vector;

[0264] W.W z , W r and b refer to learnable parameters (e.g., matrices) obtained through training;

[0265] σ (e.g., sigmoid function) and tanH are activation functions (different activation functions can be chosen);

[0266] The operator "*" is an element-wise product;

[0267] The operator "" is a vector / matrix product;

[0268] The comma indicates a concatenation.

[0269] Output h of the tth module / time step t Can be done by t-1 (In updating the gate vector z t The weighted sum of (Weighted in updating the gate vector z t Candidate output The gate vector r can be reset by applying the weight parameter W (e.g., by matrix / vector multiplication) to t and input x t Cascade h t-1The parameter W can be obtained by element-wise product between , preferably followed by application of an activation function (e.g. tanH). z (e.g., by matrix / vector multiplication) applied to h t-1 and input x t Both get the updated gate vector z t , preferably followed by applying an activation function (e.g., a sigmoid function, σ). Reset the gate vector r t By setting the parameter W r (e.g. by matrix / vector multiplication) to h t-1 and input x t , and then apply an activation function (e.g., sigmoid function, σ) to obtain.

[0270] In general:

[0271] Update gate vector [z t ] can be viewed as providing information about how much will be taken from the candidate state and / or output and how much will be taken from the state and / or output of the previous time step [h t-1 ] to obtain information about how much. For example, if z t = 0, the state and / or output used at the current time step is only taken from the state and / or output of the previous time step [h t-1 ]; and if z t = 1, then the state and / or output for the current moment is taken only from the candidate vector]; and / or

[0272] Reset gate vector [r t ] can be understood as giving the state and / or output [h t-1 ]How much information should be reset][If r t = 0, we reset everything and we start from h t-1 Nothing is retained in the t Higher, then we start from h t-1 More in the retention].

[0273] It is worth noting that candidate states and / or outputs Considering the current input x t , and the state and / or output h at time step t-1 t-1 Ignore the current input x t .therefore:

[0274] Update gate vector [z t ] is higher (e.g., z t equal to 1 or close to 1), the fewer states and / or outputs h at time step t-1 t-1 will be considered for generating the current state and / or output h t ,and

[0275] Update gate vector [z t ] is lower (e.g., z t equal to 0 or close to 0), the more the state and / or output h at time step t-1 t-1 will be considered for generating the current state and / or output h t .

[0276] Additionally, when generating candidate states and / or outputs When the gate vector [r t ] Consider:

[0277] Reset gate vector [r t ] is higher (e.g., r t All elements of are 1 or close to 1), the state and / or output h at time step t-1 t-1 Will be used to generate the current state and / or output h t The more relevant the higher the

[0278] Reset gate vector [r t ] is lower (e.g., r t All elements of are 0 or close to 0), the state and / or output h at time step t-1 t-1 Will be used to generate the current state and / or output h t The less relevant.

[0279] In this example, the weight parameters W, W z , W r At least one of the (obtained through training) may be the same for different time instances and / or modules (but in some examples).

[0280] The input of each t-th time step or feed-forward module is usually expressed as x t indicates, but refers to:

[0281] 1) at the GRU 240 of the encoder, at a particular frame in the first audio signal representation 220 of the audio signal 1 (or a processed version thereof, e.g., the output of the convolutional learnable layer 230);

[0282] 2) At the pre-conditioned learnable layer 710 of the decoder, the codes, tensors, vectors, etc. obtained from the bitstream 3 (e.g., output by the quantization index converter 313).

[0283] The output of each t-th time step or feed-forward module can be the state h t Therefore h t (or a processed version thereof) might be:

[0284] 1) At the encoder, the output of the GRU 240 is provided to the convolutional learnable layer 250;

[0285] 2) At the encoder, the output of the pre-conditioned learnable layer 710 (e.g., constituting the target data 15) is provided to the conditioning learnable layers 71-73

[0286] In this discussion, it is often imagined that the state of the output is the same for each time step and / or module. This is why we use the term h t-1 to indicate the state and output of each time step and / or module. However, this is not strictly necessary: ​​the output of each time step and / or module may in principle be different from the state inherited by the following time step and / or module. For example, the output of each time step and / or module may be a processed version of the state of the time step and / or module, or vice versa.

[0287] There are many other ways to make a recurrent learnable layer, and GRU is not the only technique to use. However, it is preferable to have a learnable layer that also considers the state and / or output of the previous moment and / or module for each moment and / or module. Therefore, it should be understood that the vocoder technique is advantageous. The fact that each moment is generated by considering the previous moment greatly facilitates operations such as encoding and decoding (particularly encoding and decoding speech).

[0288] Instead of GRU, we can also use Long / Short Term Memory (LSTM) recurrent learnable layers or “incremental differentiation” as recurrent learnable layers.

[0289] The learnable layers discussed here can be, for example, a neural network (e.g., a recurrent neural network and / or a GAN).

[0290] Generally speaking, in recurrent learnable layers, the dependencies of the previous moment are also trained, and this is a huge advantage of this technique.

[0291] discuss

[0292] Neural networks have proven to be a powerful tool for solving the problem of very low bitrate speech coding. However, designing a robust neural encoder that can operate robustly under realistic conditions remains a major challenge. Therefore, we propose the Neural End-to-End Speech Codec (NESC) (or more generally in this case), a powerful and scalable end-to-end neural speech codec for high-quality wideband speech coding at 3 kbps. The encoder (or more generally in this case) of NESC uses a new architectural configuration that relies on our proposed Dual-PathConvRNN (DPCRNN) layers, while the decoder architecture is based on our previous work Streamwise-StyleMelGAN [1]. Our subjective listening tests show that NESC (or more generally in this case) is particularly robust to unseen conditions and noise, and its computational complexity makes it suitable for deployment on end devices.

[0293] Index terms: Neural speech coding, GAN, quantization

[0294] 1. Introduction

[0295] Very low bitrate speech coding is particularly challenging when using classical techniques. The paradigm commonly adopted is parametric coding, as it produces intelligible speech, but the achievable audio quality is poor and the synthesized speech sounds unnatural. Recent advances in neural networks are filling this gap, enabling speech coding of high-quality speech at very low bitrates.

[0296] We categorize possible approaches to this problem according to the role played by the neural network.

[0297] 1-stage post-filtering: The encoder and decoder are conventional, and a neural network is added after the decoder in a post-processing step to enhance the encoded speech. This enables enhancement of existing communication systems with minimal effort.

[0298] 2-stage neural decoder: The encoder is classical and uses a neural network conditioned on the bitstream to decode the speech. This enables backward compatibility with decoding of existing bitstreams.

[0299] 3-level end-to-end: Both the encoder and decoder are neural networks that are trained jointly. The input to the encoder is the speech waveform, and the quantization may be learned jointly, so the optimal bitstream for the signal is obtained directly.

[0300] Level 1 methods such as [2, 3, 4, 5, 6] are minimally invasive as they can be deployed on existing pipelines. Unfortunately, they still suffer from typical unpleasant artifacts, which are particularly challenging.

[0301] The first published 2-level speech decoder was based on WaveNet [7] and served as a proof of concept. Several subsequent works [8, 9] improved the quality and computational complexity, and

[10] proposed LPCNet, a low-complexity decoder that can synthesize high-quality clear speech at 1.6 kbps. We have shown in previous work [1] that the same bitstream used in LPCNet can be decoded using a feed-forward GAN model, providing significantly better quality.

[0302] All of these models produce high-quality clear speech, but are not 100% robust in the presence of noise and reverberation. Lyra

[11] is the first model to directly address this problem. Its robustness to more general speech patterns is enhanced by using variance conditioning and a new bitstream that is still classically encoded. Overall, the generalization ability and quality of the 2-level models seem to be partially impaired by the limitations of the classical representation of speech at the encoder side.

[0303] Many approaches have been proposed to address this problem from the perspective of a 3-level solution [12, 13, 14, 15], but these models are generally not targeted for very low bit rates.

[0304] The first fully end-to-end approach is Soundstream

[16] , which works at low bitrates and is robust to many different noise perturbations. The core architecture of Soundstream is a U-Net-like convolutional encoder-decoder without skipped cascades and using residual quantization layers in the middle. According to the authors' evaluation, Soundstream is stable in a variety of real-world coding scenarios. In addition, it allows speech synthesis at bitrates ranging from 3kbps to 12kbps. Finally, Soundstream works at 24kHz, implements a noise reduction mode, and can also encode music. Recent work

[17] proposes another 3-level solution using a different set of techniques.

[0305] We propose NESC (or more generally in this case) a new model capable of robustly encoding wideband speech at 3 kbps. The architecture behind NESC (or more generally in this case) is fundamentally different from SoundStream and is the main aspect of the novelty of our approach. The encoder architecture is based on our proposed DPCRNN, which uses a "sandwich structure" of convolutional and recurrent layers to effectively model intra- and intra-frame dependencies. The DPCRNN layers are followed by a series of convolutional residual blocks without downsampling and followed by residual quantization. The decoder architecture consists of a recurrent neural network followed by a Streamwise-StyleMelGAN (SSMGAN[1]) decoder.

[0306] Using data augmentation, we achieve robustness against a variety of different types of noise and reverberation. We extensively test our model using multiple types of signal perturbations and unseen speakers as well as unseen languages. Furthermore, we visualize some of the clustering behaviors that are latent and learned in an unsupervised manner.

[0307] The contributions are particularly as follows:

[0308] We introduce NESC (or more generally in this case) a new end-to-end neural codec for speech.

[0309] We propose the DPCRNN layer, which provides an efficient way to exploit both intra- and inter-frame dependencies for learning latent representations suitable for quantization.

[0310] We analyzed some interesting clustering behaviors exhibited by the quantized potential of NESC.

[0311] We demonstrate the robustness of NESC to multiple types of noise and reverberation scenarios through objective and subjective evaluations.

[0312] 2. Proposed Architecture

[0313] As shown in Figure 1, the proposed model consists of a learned encoder, a learned quantization layer, and a recurrent pre-network, followed by a SSMGAN decoder.

[0314] The encoder architecture may count, for example, 2.09M parameters, while the decoder may have 3.93M parameters. The encoder rarely reuses the same parameters in computations, as we assume this is good for generalization. It can run about 40 times faster than real time on a single thread of a 3.40GHz Intel(R) Core(TM) i7-6700 CPU. Even though the decoder has only twice as many parameters as the encoder, it can run about twice as fast as real time on the same architecture. Our implementation and design are not even optimized for inference speed.

[0315] Our proposed model consists of a learned encoder, a learned quantization layer, and a recurrent pre-network, followed by a SSMGAN decoder ([1]). An overview of the model is shown in Figure 1.

[0316] 2.1. Encoder (or Audio Signal Representation Generator)

[0317] The encoder architecture can rely on our newly proposed DPCRNN, which is inspired by

[18] . This layer consists of or specifically includes: a rolling window operation frame at the definer 210, followed by a 1x1 convolution, a GRU, and finally another 1x1 convolution (230, 240, 250 respectively). The rolling window transformation reshapes the input signal of shape [1, t] into a signal of shape [s, f], where s is the length of the frame and f is the number of frames. We can use 10ms frames, which contain 5ms from the past frame and 5ms of lookahead. For 1s audio at 16kHz, this results in s=80+160+80=320 samples and f=100. Then, the 1x1 convolution layer (e.g. at 230 and / or 250) models the temporal dependencies within each frame, i.e., intra-frame dependencies, while the GRU model (e.g. at 240) the dependencies between different frames, i.e., intra-frame dependencies. This approach allows us to avoid downsampling via strided convolutional or interpolation layers, which was shown in early experiments to strongly affect the final quality of audio synthesized by SSMGAN [1].

[0318] The rest of the encoder architecture (at block 290) consists of (or specifically includes) 4 residual blocks, each of which is a 1d convolution with a kernel size of 3, followed by a 1x1 convolution and activated by a LeakyReLU. The use of DPCRNN allows the temporal dependencies of the signal to be modeled in a compact and efficient manner, so there is no need to use dilation or other tricks to expand the receptive field of the residual blocks.

[0319] Quantification

[0320] The encoder architecture (at block 290) produces a latent vector of dimension 256 for each 10ms packet. This vector is then quantized using a learned residual vectorizer based on a vector quantization VAE (VQ-VAE)

[19] , as shown in

[16] . In short, the quantizer learns multiple codebooks over the vector space of the encoder's latent packets. The first codebook approximates the latent output of the encoder z = E(x) by the nearest entry of codebook ze. The second codebook performs the same operation on the quantized "residual" (i.e., z - ze), and so on for subsequent codebooks. This technique is well known in classical coding and allows the structure of the latent vector space to be used efficiently to encode more points in the latent space than a simple union of the codebooks would allow.

[0321] In NESC (or more generally in this example), we use a residual quantizer with three codebooks (10 bits each) to encode 10ms packets, thus resulting in a total of 3kbps. Although we did not train for this, at inference time it is possible to drop one or two codebooks and still retrieve a distorted version of the output. NESC (or more generally in this example) can then scale to 2kbps and 1kbps.

[0322] 2.3. Decoder

[0323] The decoder architecture we use consists of a recurrent neural network followed by a SSMGAN decoder [1]. We use a single non-causal GRU layer as a pre-net to prepare the bitstream before feeding it to the SSMGAN decoder [1]. This provides better conditioning information for the time-adaptive denormalization layers that form the core of SSMGAN [1]. Apart from the use of a constant prior signal and the conditioning provided by the 256 latent channels, we do not make significant modifications to the SSMGAN decoder. We refer to [1] for more details about this architecture. In short, it is a convolutional decoder based on TADE (also known as FiLM) conditioning and softmax-gated tanh activations. It upsamples the bitstream by a very low upsampling ratio and provides conditioning information at each upsampled layer.

[0324] It outputs four Pseudo Quadrature Mirror Filterbank (PQMF) subbands, which are then synthesized using a synthesis filter. This filter has 50 look-ahead samples, effectively introducing a one-frame delay in our implementation. The total latency of our system is 25ms, with 15ms of encoder and framing latency and 10ms of decoder latency.

[0325] 3. Evaluation

[0326] Experimental setup

[0327] We train NESC (or more generally in this example) on the full LibriTTS dataset

[20] at 16kHz, which includes approximately 260 hours of speech. We enhance the dataset by adding reverberation and background noise. More precisely, we enhance the clear samples from LibriTTS by adding background noise from the DNS Noise Challenge dataset

[21] at random SNRs between 0dB and 50dB, and then convolve them with real or generated room impulse responses (RIRs) from the SLR28 dataset

[22] .

[0328] The training of NESC (or more generally in this case) is very similar to the training of SSMGAN [1] described in [1]. We first pre-train the encoder and decoder together, using the spectral reconstruction loss and MSE loss from

[23] as objectives for about 500k iterations. We then turn on the adversarial loss and discriminator feature loss from

[24] and train for another 700k iterations, beyond which we do not see substantial improvements. The generator is trained on 2s audio clips with a batch size of 64. We use a learning rate of 1·10 -4 The Adam

[25] optimizer is used for pre-training of the generator, and the learning rate is reduced to 5·10 as soon as possible when adversarial training starts. -5 We use a learning rate of 2·10 for the discriminator. -4 Adam optimizer for .

[0329] 3.2. Complexity

[0330] We report the computational complexity estimates in the table below.

[0331] Model Encoder complexity Decoder complexity SSMGAN 0.05GMAC 4.56GMAC SoundStream 5GMAC 5GMAC NESC 0.5GMAC 7GMAC

[0332] Table 1: Complexity estimates

[0333] Our implementation runs faster than a single thread on an Intel(R) Core(TM) i7-6700 CPU at 3.40GHz.

[0334] Potential qualitative statistical analyses

[0335] We provide a qualitative analysis of the latent distribution to better understand its behavior in practice. The quantized latent frames are embedded in a 256-dimensional space, so to plot their distribution we use their t-SNE projections. For each experiment, we first encode 10s of audio using different recording conditions and then label each frame relying on prior information about its acoustic and linguistic features. Each subgraph represents a different set of audio randomly selected from the LibriTTS, VCTK, and NTT datasets. We then look for clusters in the low-dimensional projections. Note that the model was not trained with any clustering objective, so any such behavior shown at inference time is an emergent aspect of the training setting.

[0336] We test for speaker characteristics (e.g., language and gender) as well as acoustic aspects (e.g., voicing and noise). In our first experiment (Figure 2a), we automatically labeled each frame using the VAD algorithm to test for voicing information. We noticed clear clustering of voiced, unvoiced, and quiet frames, with the boundaries consisting of transition frames. We similarly labeled voiced frames according to their quantized pitch values, but this did not show significant clustering behavior. Due to limited space, we do not show images.

[0337] In our second experiment ( Figure 2b ), we test the impact of noise introduced as in 3.1. We again notice a clear split between noisy and clean frames in the latent space, suggesting that the model uses different parts of the latent space for these different modes.

[0338] Finally, we test language and speaker-related features such as gender ( Figure 2c ), and language. In these cases, we did not observe any specific clustering, suggesting that the model is unable to distinguish these macro-level aspects.

[0339] We hypothesize that the mentioned clustering behavior might reflect the compression strategy of the model, which would be consistent with well-known heuristics already used in classical codecs.

[0340] 3.4 Objective score

[0341] We use several objective metrics to evaluate NESC. Such metrics are known to be unreliable for evaluating the quality of neural codecs [7, 10] as they are disproportionately biased towards waveform-preserving codecs. Nevertheless, we report their values ​​for comparison purposes. We consider ViSQOL v3

[29] , POLQA

[30] , and the speech intelligibility measure STOI

[31] .

[0342] Scores are computed on two internally curated test sets, StudioSet and InformalSet, in Tables 1 and 2, respectively. StudioSet consists of 108 multilingual samples from the NTT Telephone Measurement Multilingual Speech Database, totaling approximately 14 minutes of studio-quality recordings. InformalSet consists of 140 multilingual samples scraped from multiple datasets including LibriVox, totaling approximately 14 minutes of recordings. This test set includes samples recorded with an integrated microphone, more natural speech, sometimes with low background noise or reverberation from a small room. NESC (invented) scored the highest among neural coding solutions on all three metrics.

[0343] Table 1: Average objective scores for neural decoders on StudioSet. For all metrics, higher scores are better. The confidence intervals for POLQA and ViSQOL v3 are negligible, while the confidence interval for STOI is less than 0.02.

[0344]

[0345]

[0346] Table 2: Average objective scores of the neural decoders on InformalSet. For all metrics, higher scores are better. The confidence intervals for POLQA and ViSQOL v3 are negligible, while the confidence interval for STOI is less than 0.025.

[0347]

[0348] 3.5. Subjective evaluation

[0349] We test the model only under challenging unseen conditions to evaluate its robustness. To this end, we select a set of test speech samples from the NTT dataset, which includes unseen speakers, languages, and recording conditions. In the test set, "m" stands for male, "f" stands for female, "ar" stands for Arabic, "en" stands for English, "fr" stands for French, "ge" stands for German, "ko" stands for Korean, and "th" stands for Thai.

[0350] We also test the model on noisy speech. To do this, we select the same speech samples as for the clear speech test and apply a similar augmentation strategy as in Section 3.1. We add ambient noise samples (e.g. airport noise, typing noise, etc.) at SNRs between 10dB and 30dB and then convolve them with room impulse responses (RIRs) from small, medium, and large rectangular rooms. More precisely, “ar / f”, “en / f”, “fr / m”, “ko / m”, and “th / f” are convolved with the RIRs of small rooms, and thus the effect of reverberation is not significant for these signals; while the other samples are convolved with the RIRs of medium and large rooms. The augmented datasets are the same as those used in training, as they are large enough for the model to not memorize and overfit.

[0351] We conducted two MUSHRA listening tests to evaluate the NESC quality (or more generally in this example) of clear speech and speech in noise involving 11 professional listeners. The test results for clear speech are shown in Figure 2. Figure 5 The results for noisy speech are shown in Figure 2 and show that NESC (or more generally in this example) is comparable to SSMGAN [1] and Enhanced Voice Service (EVS) in this case. Figure 6 As shown, they demonstrate that SSMGAN [1] is not robust to such scenarios, and show that NESC (or more generally in this case) is comparable to EVS in this case.

[0352] The anchors used for testing were generated using OPUS at 6kbps, since the expected quality is very low at this bitrate. We used EVS with a nominal bitrate of 5.9kbps as a good baseline for the classic codec. To avoid the impact of differently signed CNG frames on the tests, we turned off DTX transmission.

[0353] Finally, our solution was also tested against our previous neural decoder SSMGAN[1] at 1.6kbps. This model produces high-quality speech in clear conditions, but is not robust in noisy real-world environments. SSMGAN[1] was trained on VCTK, so the comparison with NESC (or more generally in this case) is not entirely fair. Early experiments showed that training SSMGAN[1] with noisy data is more challenging than expected. We believe that this problem is caused by SSMGAN[1]'s reliance on pitch information, which can be difficult to estimate in noisy environments. For this reason, we decided to test NESC (or more generally in this case) against the best neural clear speech decoder we had available (i.e. SSMGAN[1] trained on VCTK), and still add it to the noisy speech test as an additional condition to show its limitations.

[0354] Both tests clearly show that NESC (or more generally in this case) is comparable to EVS while having practically half the bitrate. Furthermore, the noise test also shows the limitations of SSMGAN [1] when dealing with noisy and reverberant signals, while also showing how the quality of NESC remains high even under such challenging conditions.

[0355] 4. Conclusion

[0356] We present NESC (or more generally in this case), a new GAN model capable of high-quality and robust end-to-end speech coding. We propose the new DPCRNN as the main building block for efficient and reliable coding. We test our setup with objective quality measurements and subjective listening tests and show that it is robust under various types of noise and reverberation. We present a qualitative analysis of the latent structure, giving a glimpse into the inner workings of our codec. Future work will be devoted to further reducing complexity and improving quality.

[0357] 5. References

[0358] [1] A. Mustafa, J. Buthe, S. Korse, K. Gupta, G. Fuchs, and N. Pia, “A streaming GAN vocoder for very low bitrate wideband speech coding,” in 2021 IEEE Workshop on Audio and Acoustic Signal Processing with Applications (WASPAA), 2021, pp. 66–70.

[0359] [2] Z. Zhao, H. Liu, and T. Fingscheidt, “Enhanced coding of speech with convolutional neural networks,” IEEE / ACM Transactions on Audio, Speech, and Language Processing, vol. 27, No. 4, pp. 663–678, April 2019.

[0360] [3] J. Skoglund and J. Valin, “Improving Opus low-bitrate quality via neural speech synthesis,” INTERSPEECH, 2020.

[0361] [4] S. Korse, K. Gupta, and G. Fuchs, “Enhancing coded speech using a mask-based postfilter,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2020, pp. 6764–6768.

[0362] [5] A. Biswas and D. Jia, “Enhancing audio codecs with generative adversarial networks,” in ICASSP2020 - 2020 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2020, pp. 356–360.

[0363] [6] S. Korse, N. Pia, K. Gupta, and G. Fuchs, “PostGAN: A GAN-based postprocessor to improve coded speech quality,” arXiv preprint arXiv:2201.13093, 2022.

[0364] [7] W. B. Kleijn, F. S. C. Lim, A. Luebs, J. Skoglund, F. Stimberg, Q. Wang, and T. C. Walters, “Low-rate speech coding based on WaveNet,” in ICASSP 2018, IEEE International Conference on Acoustics, Speech and Signal Processing, 2018, pp. 676–680.

[0365] [8]C. A. van den Oord, Y. Li, F. S. Lim, A. Luebs, O. Vinyals, and T. C. Walters, “Low bitrate speech coding using vq-vae and Wavenet decoders,” in ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). IEEE, 2019, pp. 735–739.

[0366] [9] J. Klejsa, P. Hedelin, C. Zhou, R. Fejgin, and L. Villemoes, “High-quality speech coding using SampleRNN,” in ICASSP 2019, IEEE International Conference on Acoustics, Speech, and Signal Processing, 2019, pp. 7155–7159.

[0367]

[10] J. Valin and J. Skoglund, “A 1.6 kb / s real-time wideband neural vocoder using LPCNet,” in INTERSPEECH 2019, 20th Annual Conference of the International Speech Communication Association, 2019, pp. 3406–3410.

[0368]

[11] W. Kleijn, A. Storus, M. Chinen, T. Denton, F. Lim, A. Luebs, J. Skoglund, and H. Yeh, “Generative speech coding with prediction variance regularization,” in ICASSP 2021, IEEE International Conference on Acoustics, Speech, and Signal Processing, 2021.

[0369]

[12] S. Morishima, H. Harashima, and Y. Katayama, “Speech Coding Based on Multilayer Neural Networks,” in IEEE International Conference on Communications, including the Supercomm Technology Conference. IEEE, 1990, pp. 429–433.

[0370]

[13] S. Kankanahalli, “End-to-end optimized speech coding using deep neural networks,” in 2018 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). IEEE, 2018, pp. 2521–2525.

[0371]

[14] K. Zhen, J. Sung, M. S. Lee, S. Beack, and M. Kim, “Cascaded cross-module residual learning for lightweight end-to-end speech coding,” arXiv preprint arXiv:1906.07769, 2019.

[0372]

[15] K. Zhen, M. S. Lee, J. Sung, S. Beack, and M. Kim, “Efficient and scalable neural residual waveform coding with cooperative quantization,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 361–365.

[0373]

[16] N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, “Soundstream: an end-to-end neural audio codec,” in IEEE / ACM Transactions on Audio, Speech, and LAN Instrumentation Processing, pp. 1–1, 2021.

[0374]

[17] X. Jiang, X. Peng, C. Cheng, H. Xu, Y. Zhang, and Y. Lu, “End-to-end neural audio coding for real-time communications,” 2022.

[0375]

[18] Y. Luo, Z. Chen, and T. Yoshioka, “Dual-path RNN: Efficient long sequence modeling for single-channel speech separation in time domain,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Audio Signal Processing (ICASSP), 2020, pp. 46–50.

[0376]

[19] A. van den Oord, O. Vinyals, and K. Kavukcuoglu, “Neural discrete representation learning,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017, pp. 6309–6318.

[0377]

[20] H. Zen, V. Dang, R. Clark, Y. Zhang, R. Weiss, Y. Jia, Z. Chen, and Y. Wu, “Libritts: A text-to-speech corpus from librispeech,” arXiv preprint arXiv:1904.02882, 2019.

[0378]

[21] C. Reddy, H. Dubey, V. Gopal, R. Cutler, S. Braun, H. Gamper, R. Aichner, and S. Srinivasan, “Icassp 2021 Deep Noise Suppression Challenge,” in ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). IEEE, 2021, pp. 6623–6627.

[0379]

[22] D. Povey, “OpenSLR 28 dataset,” https: / / www.openslr.Org / 28 / / .

[0380]

[23] R. Yamamoto, E. Song, and J. Kim, “Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrograms,” in ICASSP 2020, IEEE International Conference on Acoustics, Speech, and Signal Processing, 2020, pp. 6199–6203.

[0381]

[24] K. Kumar, R. Kumar, de T. Boissiere, L. Gestin et al., “MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis,” in Advances in NeurIPS 32, 2019, pp. 14910–14921.

[0382]

[25] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” ICLR, 2015.

[0383]

[29] M. Chinen, F. S. Lim, J. Skoglund, N. Gureev, F. O’Gorman, and A. Hines, “ViSQOLv3: An open source production-ready objective speech and audio metric,” in 2020 12th International Conference on Quality of Multimedia Experience (QoMEX). IEEE, 2020, pp. 1–6.

[0384]

[30] J. Beerends, C Schmidmer, J. Berger, M. Obermann, R. Ullmann, J. Pomy, and M. Keyhl, “Perceptually Objective Listening Quality Assessment (POLQA), the third generation ITU-T standard for end-to-end speech quality measurement Part 1 – Time Alignment.” Audio Engineering Journal SM TERW 61, No. 6, pp. 366–384, June 2013.

[0385]

[31] C.IL Taal, RC Hendriks, R.Heusdens and J.Jensen, “Time-frequency weighted noisy speech intelligibility prediction algorithm”, IEEE Trans Audio Speech Lang. Process, vol. 2125–2136, 2011.

[0386] Further Characterization of the Figures

[0387] Figure 1b and 8 : A high-level architecture of a neural end-to-end speech codec.

[0388] Figure 2a: t-SNE projection of latent frames labeled based on utterance information. Voiced and unvoiced frames are clearly clustered. Each sub-figure represents 10 seconds of speech data.

[0389] Figure 3 : t-SNE projection of latent frames clusters noise frames and clear speech frames. Each sub-graph represents 10s of speech data

[0390] Figure 2b : t-SNE projection of latent frames shows no gender-based clustering. Each subplot represents 10s of speech data.

[0391] Figure 3 c: Listening tests of clear speech showed that NESC was comparable to EVS and SSMGAN.

[0392] Figure 5 :Listening tests of clear speech showed that NESC was comparable to EVS and SSMGAN.

[0393] Figure 6 Listening tests of speech in noise show that NESC is robust under very challenging conditions

[0394] 7. Conclusion

[0395] We propose a new GAN model to NESC that enables high-quality and robust end-to-end speech coding. We propose the new DPCRNN as the main building block for efficient and reliable coding. We show how residual quantization and the SSMGAN decoder can produce high-quality speech signals that are robust to various types of noise and reverberation.

[0396] The question of how to further improve speech quality while reducing the computational complexity of the model remains open.

[0397] 8. Important aspects

[0398] 8.1 Potential applications and benefits of the current example:

[0399] Generate compact yet general and meaningful representations of speech signals, even those recorded in noisy and reverberant environments.

[0400] Applications: processing speech in the latent representation thus generated, such as speech enhancement (e.g. denoising, dereverberation), or disentanglement, separation, modification, suppression of paralinguistic features (speaker ID, emotion, etc.) for applications such as voice conversion, privacy protection, etc.

[0401] Applications in speech transmission: Encode and transmit speech at very low bit rates (or sampling rates) while maintaining natural and good quality, surpassing the coding efficiency of traditional coding schemes.

[0402] 8.2 General Speech Representation

[0403] The main novelty is the adoption of GRU and the use of a dual-path acoustic front-end based on a rolling window. The rolling window operation consists in reshaping a signal in the time domain of shape (1, time length) into overlapping frames of shape (frame length, number of frames). For example, the signal (t0, t1, t2, t3) is passed through a rolling window with a frame length of 2 and an overlap of 1, resulting in a reshaped signal

[0404]

[0405] Each of its length 2 has 3 frames. The temporal dimension along the frame is interpreted as the input channel of a 1x1 convolution, i.e. a convolution with kernel size 1, which models the dependencies within each frame. This is followed by a GRU, which models the dependencies between different frames.

[0406] For more details, see Figure 1b and Figure 8 .

[0407] Existing technologies: HuBERT, wav2vec

[0408] Reference is also made above and below to an audio representation method (or more generally a technique) of generating a latent representation (e.g. 269) from an input audio signal (e.g. 1), the audio signal (e.g. 1) being subdivided in a sequence of frames, the audio representation 200 comprising:

[0409] a rolling window transform 210 that reshapes consecutive samples of the input audio signal segmented into frames into a reshaped input (tensor) of at least 2 dimensions, one (intra-frame) dimension spanning frame indices and another (intra-frame) dimension spanning sample positions within one or more overlapping frames,

[0410] At least one sequence of learnable layers (e.g., 230, 240, 250) to provide an encoded representation (e.g., 269, 469) of an input audio signal (e.g., 1) at a given frame, and accepting as input a reconstructed input (tensor).

[0411] The input audio signal (e.g., 1) may be speech or recorded speech or speech mixed with background noise or room effects. Additionally or alternatively, at least one sequence of learnable layers (e.g., 230, 240, 250) may include a cyclic unit (e.g., 240) (e.g., applied along the intra dimension). Additionally or alternatively, at least one sequence of learnable layers (e.g., 230, 240, 250) may include a convolution 230 (e.g., 1x1 convolution) (e.g., applied along the intra dimension). Additionally or alternatively, at least one sequence of learnable layers (e.g., 230, 240, 250) may include a convolution (e.g., 1x1 convolution) 230, for example followed by a cyclic unit 240, followed by a convolution (e.g., 1x1 convolution) 240.

[0412] 8.3 Application Speech Transmission: Encoders

[0413] The encoder aspect encompasses the novelty of the currently disclosed model by leveraging the speech representation methods disclosed above.

[0414] Prior art: Soundstream[5]

[0415] Here, in particular, an audio encoder (e.g. 2) is disclosed, configured to generate a bit stream (e.g. 3) from an input audio signal (e.g. 1), the bit stream (e.g. 3) representing the audio signal (e.g. 1), the audio signal (e.g. 1) being subdivided into a sequence of frames, the audio encoder comprising:

[0416] a rolling window transform (e.g., 210) that reshapes consecutive samples segmented into frames of the input audio signal into a reshaped input (tensor) of at least 2 dimensions, one (intra-frame) dimension spanning frame indices and another (intra-frame) dimension spanning sample positions within one or more overlapping frames,

[0417] At least one sequence of learnable layers (e.g., 230, 240, 250) to provide an encoded representation of an input audio signal (e.g., 1) at a given frame and accepting as input a reconstructed input (tensor).

[0418] • A quantizer (eg, 300) configured to quantize the latent representation at a given frame.

[0419] Additionally or alternatively, at least one sequence of learnable layers (e.g., 230, 240, 250) may include a cyclic unit (applied along the intra dimension) 240 (e.g., a GRU or LSTM). Additionally or alternatively, at least one sequence of learnable layers (e.g., 230, 240, 250) includes a 1x1 (e.g., 1x1 convolution) (e.g., applied along the intra dimension). Additionally or alternatively, at least one sequence of learnable layers may include a convolution (e.g., 1x1 convolution) 230, followed by a cyclic unit 240, followed by a convolution (e.g., 1x1 convolution) 250. Additionally or alternatively, the quantizer 300 may be a vectorizer. Additionally or alternatively, the quantizer 300 may be a residual or multi-level vectorizer. Additionally or alternatively, the quantizer 300 may be learnable and learned together with at least one learnable layer and / or the codebook used by it is learnable.

[0420] It should be noted that at least one codebook (at the quantizer 300 and / or at the quantization index converter 313) may have a fixed length. In case there are multiple orderings, the encoder may signal in the bitstream which ordering index is encoded.

[0421] 8.4 Application Voice Transmission: Decoder

[0422] The decoder uses features from the published Streamwise-StyleMelGAN (SSMGAN). The decoder side then involves using an RRN (e.g., GRU) as a prenet used before the conditional SSMGAN.

[0423] Prior art: SSMGAN[1]

[0424] An audio decoder (e.g. 10) is disclosed, configured to generate an output audio signal (e.g. 16) from a bit stream (e.g. 3), the bit stream (e.g. 3) representing an audio signal (e.g. 1) intended to be reproduced, the audio signal (e.g. 1) being subdivided into a sequence of frames, the audio decoder (e.g. 10) comprising at least one of the following:

[0425] a first data provider (e.g. 702) configured to provide, for a given frame, first data derived from an external source or an internal source or from a bitstream (e.g. 3),

[0426] At least one pre-conditioned learnable layer based on a recurrent unit (e.g., 710), configured to receive a bitstream (e.g., 3) and, for a given frame, output target data (e.g., 12) representing an audio signal (e.g., 1) in a given frame.

[0427] At least one conditional learnable layer configured to process target data (e.g., 12) for a given frame to obtain conditional feature parameters (e.g., 74, 75) for the given frame.

[0428] A style element (eg, 77) configured to apply the conditional feature parameters (eg, 74, 75) to the first data (eg, 15) or the normalized first data to obtain an output audio signal (eg, 16).

[0429] Final summary

[0430] The above examples are summarized here. Some new functions may also integrate the above examples (eg, by bracket integration, which creates additional embodiments and / or variants).

[0431] As shown in the above example, an audio generator (10) is disclosed, which is configured to generate an audio signal (16) from a bit stream (3), wherein the bit stream (3) represents the audio signal (16), and the audio signal is subdivided into a sequence of frames. The audio generator (10) includes:

[0432] a first data provider (702) configured to provide, for a given frame, first data (15) derived from an input signal (14) [e.g., from an external or internal source or from a bitstream (3)], [wherein the first data (15) may have a single channel or multiple channels; the first data may, for example, be completely independent of the target data and / or of the bitstream, while in other examples the first data may have some relationship with the bitstream in that it may be obtained from the bitstream, e.g., from an LPC parameter of the bitstream, or other parameters obtained from the bitstream];

[0433] A first processing block (40, 50, 50a-50h) configured to receive first data (15) for a given frame and output first output data (69) in the given frame, [wherein the first output data (69) may include a single channel or multiple channels],

[0434] [For example, the audio generator further comprises a second processing block (45) configured to receive the first output data (69) or data derived from the first output data (69) as the second data for a given frame],

[0435] The first processing block (50) comprises:

[0436] at least one pre-conditioned learnable layer (710) configured to receive the bitstream (3) or a processed version thereof (112) and, for a given frame, output target data (12) representing an audio signal (16) in the given frame [e.g., multiple channels and multiple samples for the given frame];

[0437] at least one conditional learnable layer (71, 72, 73) configured to process the target data (12) for a given frame to obtain feature parameters (74, 75) conditioned on the given frame; and

[0438] a style element (77) configured to apply a conditional feature parameter (74, 75) to the first data (15, 59a) or the normalized first data (59, 76');

[0439] [wherein the second processing block (45), if present, may be configured to combine the plurality of channels (47) of the second data (69) to obtain the audio signal (16)],

[0440] Wherein the at least one pre-conditioned learnable layer (710) comprises at least one recurrent learnable layer [e.g., a gated recurrent learnable layer, such as a gated recurrent unit GRU]

[0441] [For example, configured to obtain the audio signal (16) from the first output data (69) or a processed version of the first output data (69)].

[0442] The audio generator (10) may be configured to obtain the audio signal (16) from the first output data (69) or a processed version of the first output data (69).

[0443] The audio generator (10) may cause the first data (15) to have a plurality of channels, wherein the first output data (69) includes the plurality of channels (47),

[0444] The audio generator also includes a second processing block (45) configured to receive the first output data (69) or data derived from the first output data (69) as second data for a given frame, the output target data (12) having multiple channels and multiple samples for the given frame, wherein the second processing block (45) is configured to combine the multiple channels (47) of the second data (69) to obtain the audio signal (16).

[0445] As shown in the above example, an audio generator (10) is disclosed, which is configured to generate an audio signal (16) from a bit stream (3), wherein the bit stream (3) represents the audio signal (16), and the audio signal is subdivided into a sequence of frames. The audio generator (10) includes:

[0446] a first data provider (702) configured to provide, for a given frame, first data (15) derived from an input signal (14), [e.g. from an external or internal source or from a bitstream (3)], [wherein the first data (15) may have a single channel or multiple channels; the first data may, for example, be completely independent of the target data and / or the bitstream, while in other examples the first data may have some relationship to the bitstream in that it may be obtained from the bitstream, e.g. from an LPC parameter of the bitstream, or other parameters obtained from the bitstream];

[0447] A first processing block (40, 50, 50a-50h) configured to receive first data (15) for a given frame and output first output data (69) in the given frame, wherein the first output data (69) may include a plurality of channels (47),

[0448] The audio generator further comprises a second processing block (45) configured to receive, for a given frame, the first output data (69) or data derived from the first output data (69) as second data,

[0449] The first processing block (50) comprises:

[0450] at least one pre-conditioned learnable layer (710) configured to receive the bitstream (3) or a processed version thereof (112) and, for a given frame, output target data (12) representing an audio signal (16) in the given frame [e.g., multiple channels and multiple samples for the given frame];

[0451] at least one conditional learnable layer (71, 72, 73) configured to process the target data (12) for a given frame to obtain conditional feature parameters (74, 75) for the given frame; and a style element (77) configured to apply the conditional feature parameters (74, 75) to the first data (15, 59a) or the normalized first data (59, 76');

[0452] wherein the second processing block (45), if present, may be configured to combine the plurality of channels (47) of the second data (69) to obtain the audio signal (16),

[0453] Wherein at least one pre-conditioned learnable layer (710) comprises at least one recurrent learnable layer [e.g., a gated recurrent learnable layer, such as a gated recurrent unit, a GRU, or an LSTM]

[0454] [For example, the audio generator may be configured to obtain the audio signal (16) from the first output data (69) or a processed version of the first output data (69)].

[0455] The audio generator may cause the recurrent learnable layer to include at least one gated recurrent unit (GRU).

[0456] The audio generator can make the recurrent learnable layer include at least one long short-term memory, LSTM, and recurrent learnable layer.

[0457] The audio generator can configure the recurrent learnable layer to generate an output [target data (12)] for a given time moment by considering the output [target data (12)] and / or the state at a previous [e.g., immediately previous] time moment, wherein the correlation between the output [target data (12)] and / or the state at a previous [e.g., immediately previous] time moment is obtained through training.

[0458] The audio generator may cause a recurrent learnable layer to operate along a sequence of time steps, each time step having at least one state such that each time step is conditioned by the output and / or state of the previous time step [the state of the previous time step may be the output] [may be, e.g. Fig.11 As shown, the steps and / or the output of each step is recursively provided to a subsequent time step, such as the time step immediately following it] [or, as Fig.12 As shown, there may be multiple feed-forward modules, each of which provides a state and / or output to a subsequent module, such as an immediately subsequent module] [In some examples, it can be understood that Fig.12 The implementation is like Fig.11 [In examples, the parameters of different moments and / or feedforward modules may generally be different from each other, but in some examples they may be the same].

[0459] The audio generator may also include multiple feed-forward modules, each feed-forward module providing a state and / or output to an immediately subsequent module.

[0460] The audio generator can be configured such that the recurrent learnable layer generates states and / or outputs [h t ][For a specific t-th state or module]:

[0461] By updating the gate vector [z t ][whose elements may have values ​​between 0 and 1, or another value between 0 and c, where c>0] weighting the candidate states and / or outputs to generate a first weighted addend; and

[0462] Update the gate vector z by a vector complementary to 1 (i.e. its components are complementary to 1) t The state and / or output [h t-1 ] are weighted to generate a second weighted addend; and

[0463] Add the first addend to the second addend

[0464] [Update gate vector [zt ] provides information about how much to take from the candidate state and / or output and how much to take from the state and / or output at the previous time step [h t-1 ] to obtain information; for example, if z t = 0, the state and / or output used at the current time step is only taken from the state and / or output of the previous time step [h t-1 ]; and if z t =1, then the state and / or output for the current moment is taken only from the candidate vector].

[0465] The audio generator can be configured such that the recurrent learnable layer generates states and / or outputs [h t ]:

[0466] The weighted version of the candidate state and / or output is combined with the state and / or output h at the previous time step by means of the complementary weight vectors t-1 The weighted versions of are added together.

[0467] The audio generator may have a recurrent learnable layer configured to generate candidate states and / or outputs by applying at least weight parameters [W] obtained through training to:

[0468] Reset gate vector [r t ] and the state and / or output of the previous time step [h t-1 ], concatenated with the input [xt] for the current moment;

[0469] Optionally followed by an activation function (e.g. tanH)

[0470] [Reset gate vector [r t ] gives the state and / or output of the previous time step [h t-1 ]How much information should be reset][If r t = 0, we reset everything and start from h t-1 Nothing is retained in the t Higher, then we start from h t-1 More in the retention].

[0471] The audio generator may also be configured to apply an activation function after applying the weight parameter W. The activation function may be TanH.

[0472] The audio generator may be such that the recurrent learnable layer is configured to generate candidate states and / or outputs in at least the following manner:

[0473] The weight vector is obtained by the weight parameter W obtained by training, which is affected by the following two factors:

[0474] For the current input [xt ]and

[0475] The weight of the previous time step is reset in the gate vector [r t ] on the state and / or output [h t-1 ], [Reset gate vector gives the state and / or output of the previous time step [h t-1 ]How much information should be reset][If r t = 0, we reset everything, and we start from h t-1 Nothing is retained in the t Higher, then we start from h t-1 More in the retention].

[0476] The audio generator can be configured such that the recurrent learnable layer is configured by setting the parameter [W z ] is applied to the following cascade to generate the update gate vector [z t ]:

[0477] Loop module [h t-1 ] input [h t-1 ] Cascaded to

[0478] For the current input [x t ][e.g., input to at least one pre-conditioned learnable layer (710)],

[0479] An activation function (eg, a sigmoid function, σ) is optionally then applied.

[0480] The audio generator can be configured to z Then the activation function is applied.

[0481] The audio generator can be such that the activation function is a sigmoid function, σ.

[0482] The audio generator can reset the gate vector r t is obtained by setting the weight parameter W r Obtained by applying to the concatenation of:

[0483] The state and / or output h at the previous time step t-1 and

[0484] For the current input x t .

[0485] The audio generator can be configured to r Then the activation function is applied.

[0486] The audio generator can be such that the activation function is a sigmoid function, σ.

[0487] The audio generator (10) may include a quantization index converter (313) [also known as an index to code converter, inverse quantizer, inverse quantizer, etc.] configured to convert an index of a bitstream (13) to a code [e.g., according to an example, the code may be a scalar, a vector, or more generally a tensor] [e.g., according to a codebook, e.g., a tensor may be multi-dimensional, such as a matrix or a generalization thereof to multiple dimensions, such as three dimensions, four dimensions, etc.] [e.g., the codebook may be learnable or may be deterministic] [e.g., the codebook 112 may be provided to a pre-conditioned learnable layer (710)].

[0488] As shown in the above example, an audio generator (10) is disclosed, which is configured to generate an audio signal (16) from a bit stream (3), wherein the bit stream (3) represents the audio signal (16), the bit stream (3) is subdivided into an index sequence, and the audio signal is subdivided into a sequence of frames, and the audio generator (10) includes:

[0489] A quantization index converter (313) [also referred to as an index to code converter, inverse quantizer, inverse quantizer, etc.] is configured to convert an index of the bitstream (13) to a code [e.g., according to an example, the code may be a scalar, a vector, or more generally a tensor] [e.g., according to a codebook, e.g., a tensor may be multi-dimensional, such as a matrix or a generalization thereof to multiple dimensions, such as three dimensions, four dimensions, etc.] [e.g., the codebook may be learnable or may be deterministic],

[0490] The first data provider (702) is configured to provide, for a given frame, first data (15) derived from an input signal (14) from an external or internal source or from a bitstream (3), [wherein the first data (15) may have a single channel or multiple channels][;

[0491] a first processing block (40, 50, 50a-50h) configured to receive first data (15) for a given frame and output first output data (69) in the given frame, [wherein the first output data (

[69] ) may include a single channel or multiple channels (47)], and

[0492] [There may be a second processing block (45) configured to receive the first output data (69) or data derived from the first output data (69) as second data for a given frame],

[0493] The first processing block (50) comprises:

[0494] at least one pre-conditioned learnable layer (710) configured to receive a bitstream (3) or a processed version thereof (112) [e.g., the processed version (112) may be output to a converter (313) via quantization indices], and, for a given frame, output target data (12) representing an audio signal (16) in a given frame [e.g., a plurality of channels and a plurality of samples for a given frame];

[0495] at least one conditional learnable layer (71, 72, 73) configured to process the target data (12) for a given frame to obtain conditional feature parameters (74, 75) for the given frame; and a style element (77) configured to apply the conditional feature parameters (74, 75) to the first data (15, 59a) or the normalized first data (59, 76');

[0496] [Wherein the second processing block (45), if present, may be configured to combine the plurality of channels (47) of the first output data or the second output data (69) to obtain the audio signal (16)]

[0497] [For example, configured to obtain the audio signal (16) from the first output data (69) or a processed version (16) of the first output data (69)].

[0498] The audio generator may cause the first data to have a plurality of channels, the first output data to include a plurality of channels, the target data to have a plurality of channels,

[0499] Also included is a second processing block (45) configured to combine the plurality of channels (47) of the first output data to obtain an audio signal (16).

[0500] The audio generator may cause at least one codebook to be learnable.

[0501] The audio generator may cause the quantization index converter (313) to use at least one codebook to convert the index encoded in the bitstream [e.g., codebook z e 、r e ,q e , where index i z Denotes the code z approximating E(x) and taken from codebook z e , index i r Approximate E(x)-z and take it from codebook r e , index q e Approximate E(x)-zr and take it from codebook i q ] is associated to a code, such as a scalar, a vector, or more general tensor, representing one frame, several frames, or a portion of a frame of the audio signal to be generated.

[0502] The audio generator may make at least one codebook [e.g. e 、re ,q e ] is or includes a basic codebook [e.g. z e ], associating an index [e.g., z] encoded in the bitstream (3) to a code [e.g., a scalar, vector, or more general tensor] representing a major part of the frame [e.g., latent].

[0503] The audio generator may cause at least one codebook to be [or include] a residual codebook [eg, a first residual codebook, such as r e , and may be a second residual codebook, such as q e , and possibly even lower ordered residual codebooks; additional codebooks are possible] that associates an index encoded in the bitstream to a code [e.g., a scalar, vector, or more generally a tensor] representing a residual [e.g., error] portion of a frame [e.g., where the audio generator is also configured to reconstruct the frames, e.g., by adding a basis portion to one or two or more residual portions for each frame].

[0504] The audio generator can define multiple residual codebooks so that

[0505] The second residual codebook associates an index encoded in the bitstream to a code (scalar, vector, tensor, etc.) representing the second residual portion of the frame, and

[0506] The first residual codebook associates an index encoded in the bitstream to a code representing a first residual part of a frame,

[0507] Wherein the second residual portion of the frame is residual [eg, low ranked] relative to the first residual portion of the frame.

[0508] The audio generator can cause the bitstream (3) to signal whether the index associated with the residual frame is encoded, and the quantization index converter (313) is correspondingly configured to read (e.g., only) the encoded index according to the signal notification [and, in the case of different orderings, the bitstream can signal which indexes of which ordering are encoded, and / or at least one codebook (313) correspondingly reads, for example, only the encoded index according to the signal].

[0509] The audio generator may be such that at least one codebook is a fixed length codebook [eg at least one codebook has a number of bits between 4 and 20, such as between 8 and 12, such as 10].

[0510] The audio generator can be configured to perform dithering on the code.

[0511] The audio generator may be such that a training session is performed by receiving a plurality of bitstreams having indices associated with known codes, representing known audio signals, the training session comprising an evaluation of the generated audio signal with respect to the known audio signal, thereby adjusting an association of an index of at least one codebook with a frame of the encoded bitstream [e.g. by minimizing a difference between the generated audio signal and the known audio signal] [e.g. using a GAN].

[0512] The audio generator may enable a training session to be performed by receiving at least:

[0513] a plurality of first bitstreams having first candidate indices, the first candidate indices having a first bit length and being associated with first known frames representing a known audio signal, the first candidate indices forming a first candidate codebook, and

[0514] a plurality of second bit streams having second candidate indices, the second candidate indices having a second bit length and being associated with known frames representing the same first known audio signal, the second candidate indices forming a second candidate codebook,

[0515] wherein the first bit length is higher than the second bit length [and / or the first bit length has a higher resolution but it occupies more frequency band than the second bit length],

[0516] The training session includes an evaluation of generated audio signals obtained from the plurality of first bitstreams compared to generated audio signals obtained from the plurality of second bitstreams, thereby selecting a codebook [e.g., such that the selected learnable codebook is a selected codebook between a first candidate codebook and a second candidate codebook] [e.g., there may be an evaluation of a first ratio versus a second ratio, wherein the first ratio is a ratio between metrics measuring the quality of the audio signals generated from the plurality of first bitstreams with respect to bit length, and the second ratio is a ratio between metrics measuring the quality of the audio signals generated from the plurality of second bitstreams with respect to bit rate (sampling rate), and selecting a bit length that maximizes the ratio] [e.g., this may be repeated for each codebook, e.g., primary, first residual, second residual, etc.].

[0517] The audio generator can enable a training session to be performed by receiving:

[0518] a first plurality of first bitstreams having first indices associated with first known frames representing a known audio signal, wherein the first indices have a first maximum number, the first plurality of first bitstreams forming a first candidate codebook; and

[0519] a second plurality of second bitstreams having second indices associated with known frames representing the same first known audio signal, the second plurality of second candidate indices forming a second candidate codebook, wherein a second maximum number of second indices is different from the first maximum number,

[0520] The training session includes an evaluation of a generated audio signal obtained from a first plurality of first bitstreams compared to a generated audio signal obtained from a second plurality of second bitstreams, thereby selecting a learnable index [e.g., such that the selected learnable codebook is selected from a first candidate codebook and a second candidate codebook] [e.g., there may be an evaluation of a first ratio versus a second ratio (sampling rate), wherein the first ratio is a ratio between a metric measuring the quality of the audio signal generated from the first plurality of first bitstreams, and the second ratio is a ratio between a metric measuring the quality of the audio signal generated from the second plurality of second bitstreams with respect to the bit rate (sampling rate), and selecting the one that maximizes the ratio between the first plurality of bitstreams and the second plurality of second bitstreams] [e.g., this can be repeated for each codebook, e.g., primary, first residual, second residual, etc.].

[0521] The audio generator may enable a training session to be performed by receiving:

[0522] The first plurality of first bit streams have codes representing codes obtained from a known audio signal, and the first plurality of first bit streams with first indices form at least one first codebook [e.g., at least one main codebook z e ];as well as

[0523] A second plurality of second bit streams, comprising first indices representing primary codes obtained from a known audio signal and second indices representing residual codes with respect to the primary codes, the second plurality of second bit streams forming at least one first codebook [e.g., at least one primary codebook z e ] and at least one second codebook [e.g. at least one residual codebook r e ];

[0524] The training session includes an evaluation of a generated audio signal obtained from a first plurality of first bitstreams compared to a generated audio signal obtained from a second plurality of second bitstreams, thereby performing a comparison between the first plurality of [and / or first candidate codebooks z e ] and the second plurality [ and / or the first candidate codebook z e As the main codebook, together with the residual codebook r e [e.g., such that the selected learnable codebook is selected from the first candidate codebook and the second candidate codebook] [e.g., a first ratio between metrics measuring the quality of audio signals generated from the first plurality of first bitstreams can be evaluated compared to a second ratio between metrics measuring the quality of audio signals generated from the second plurality of second bitstreams with respect to bit rate (sampling rate), and the plurality that maximizes the ratio can be selected from the first plurality and the second plurality] [e.g., this can be repeated for each codebook, e.g., main, first residual, second residual, etc.].

[0525] The audio generator may be configured so that a bit rate (sampling rate) of the audio signal (16) is greater than a bit rate (sampling rate) of the target data (12) of the first data (15) and / or the second data (69).

[0526] The audio generator also includes a second processing block (45) configured to increase the bit rate (sampling rate) of the second data (69) to obtain the audio signal (16) [and / or wherein the second processing block (45) is configured to reduce the number of channels of the second data (69) to obtain the audio signal (16)].

[0527] The audio generator may cause the first processing block (50) to be configured to upsample the first data (15) from a number of samples for a given frame to a second number of samples for the given frame that is greater than the first number of samples.

[0528] The audio generator may include a second processing block (45) configured to upsample second data (69) obtained from the first processing block (40) from a second number of samples for a given frame to a third number of samples for the given frame that is greater than the second number of samples.

[0529] The audio generator may be configured to reduce the number of channels of the first data (15) from a first number of channels to a second number of channels of the first output data (69), the second number of channels being lower than the first number of channels.

[0530] The audio generator also includes a second processing block (45) configured to reduce the number of channels of the first output data (69) obtained from the first processing block (40) from the second number of channels of the audio signal (16) to a third number of channels, wherein the third number of channels is smaller than the second number of channels.

[0531] The audio generator can cause the audio signal (16) to be a single channel audio signal.

[0532] The audio generator may be configured to obtain an input signal (14) from a bitstream (3, 3b).

[0533] The audio generator may be configured to obtain an input signal from noise (14).

[0534] The audio generator may cause at least one pre-conditioned learnable layer (710) to be configured to provide target data (12) as a spectrogram or a decoded spectrogram.

[0535] The audio generator is such that at least one conditional learnable layer or a group of conditional learnable layers includes one or at least two convolutional layers (71-73).

[0536] The audio generator causes the first convolution layer (71-73) to be configured to convolve the target data (12) or the upsampled target data using a first activation function to obtain first convolution data (71').

[0537] The audio generator may enable the first activation function to be a leaky rectified linear unit function, namely, a Leaky ReLu function.

[0538] The audio generator is such that at least one conditional learnable layer or a group of conditional learnable layers (71-73) and the style element (77) are weight layers in a residual block (50, 50a-50h) of a neural network including one or more residual blocks (50, 50a-50h).

[0539] The audio generator causes the audio generator (10) to further include a normalization element (76) configured to normalize the first data (59a, 15).

[0540] The audio generator causes the audio generator (10) to further include a normalization element (76) configured to normalize the first data (59a, 15) in the channel dimension.

[0541] The audio generator causes the audio signal (16) to be a speech audio signal.

[0542] The audio generator causes the target data (12) to be upsampled by a factor of a power of 2 or another factor such as 2.5 or a multiple of 2.5.

[0543] The audio generator causes the target data (12) to be upsampled (70) by non-linear interpolation.

[0544] The audio generator causes the first processing block (40, 50, 50a-50k) to further include:

[0545] another set of learnable layers (62a, 62b) configured to process data derived from the first data (15, 59, 59a, 59b) using a second activation function (63b, 64b),

[0546] The second activation function (63b, 64b) is a gated activation function.

[0547] The audio generator enables another set of learnable layers (62a, 62b) to include one or two or more convolutional layers.

[0548] The audio generator makes the second activation function (63a, 63b) a softmax gated hyperbolic function, namely the tangent TanH.

[0549] The audio generator is such that the first activation function is a leaky rectified linear unit (leaky ReLu) function.

[0550] The audio generator causes the convolution operation (61b, 62b) to run with a maximum dilation factor of 2.

[0551] The audio generator includes eight first processing blocks (50a-50h) and one second processing block (45).

[0552] The audio generator causes the first data (15, 59, 59a, 59b) to have its own dimension lower than that of the audio signal (16).

[0553] The audio generator may cause the target data (12) to be a spectrogram.

[0554] The audio signal (16) may be a single channel audio signal.

[0555] As shown in the above example, an audio signal representation generator (2, 20) is disclosed for generating an output audio signal representation (3, 469) from an input audio signal (1) comprising a sequence of input audio signal frames, each input audio signal frame comprising a sequence of input audio signal samples, the audio signal representation generator comprising:

[0556] A format definer (210) configured to define a first multi-dimensional audio signal representation (220) of an input audio signal (1), the first multi-dimensional audio signal representation (220) of the input audio signal comprising at least:

[0557] a first dimension [e.g., an intra-frame dimension] such that a plurality of mutually consecutive frames [e.g., immediately subsequent frames] are ordered according to the first dimension; and

[0558] a second dimension [e.g., an intra-frame dimension] such that a plurality of samples of at least one frame are ordered according to the second dimension [the format definer may be configured to order samples that are consecutive to one another, e.g., immediately subsequent samples, one after another according to the second dimension],

[0559] At least one learnable layer (230, 240, 250, 290, 300) is configured to process a first multidimensional audio signal representation (220) of an input audio signal (1) or a processed version of the first multidimensional audio signal representation to generate an output audio signal representation (3, 469) of the input audio signal (1).

[0560] The audio signal representation generator may cause the format definer (210) to be configured to interpolate the input audio signal samples of each given frame along a second dimension (eg, an intra-frame dimension) of the first multi-dimensional audio signal representation of the input audio signal.

[0561] The audio signal representation generator may cause the format definer (210) to be configured to insert the input audio signal samples of each given frame along a second dimension (eg, an intra-frame dimension) of the first multidimensional audio signal representation (220) of the input audio signal (1).

[0562] The audio signal representation generator may cause the format definer (210) to be configured to insert additional input audio signal samples of one or more additional frames immediately following a given frame along a second dimension [e.g. an intra-frame dimension] of the first multidimensional audio signal representation (220) of the input audio signal (1) [e.g. in a predefined number, e.g. application-specific, e.g. defined by a user or the application].

[0563] The audio signal representation generator may cause the format definer (210) to be configured to insert additional input audio signal samples of one or more additional frames immediately preceding the given frame along a second dimension of the first multidimensional audio signal representation (220) of the input audio signal (1) [e.g. in a predefined number, e.g. application-specific, e.g. defined by a user or the application].

[0564] The audio signal representation generator may be such that the at least one learnable layer comprises at least one recurrent learnable layer (240) [eg, a GRU].

[0565] The audio signal representation generator may cause at least one recurrent learnable layer (240) to operate along a first dimension [eg, an intra dimension].

[0566] The audio signal representation generator may further include at least one first convolutional learnable layer (230) [e.g., having a convolutional kernel, which may be a learnable kernel and / or may be a 1x1 kernel] between the format definer (210) and the at least one recurrent learnable layer (240) [e.g., a GRU or LSTM],

[0567] The audio signal representation generator may cause, in at least one first convolutional learnable layer (230) [first learnable layer], a kernel to slide along a second direction [e.g., an intra-frame direction] of a first multidimensional audio signal representation (220) of an input audio signal (1).

[0568] The audio signal representation generator may also include at least one convolutional learnable layer (250) [e.g., having a convolutional kernel, which may be a learnable kernel and / or may be a 1x1 kernel] located downstream of the at least one recurrent learnable layer (240) [e.g., a GRU or LSTM].

[0569] The audio signal representation generator may cause, in at least one convolutional learnable layer (250) [first learnable layer], a kernel to slide along a second direction [e.g., an intra-frame direction] of a first multidimensional audio signal representation (220) of an input audio signal (1).

[0570] The audio signal representation generator may be such that at least one or more of the at least one learnable layer is a residual learnable layer.

[0571] The audio signal representation generator may cause at least one learnable layer (230, 240, 250) to be a residual learnable layer [e.g., bypassing (259') at least one learnable layer (230, 240, 250) a main portion of the first multidimensional audio signal representation (220) of the input audio signal, and / or applying at least one learnable layer (230, 240, 250) to at least a residual portion (259a) of the first two-dimensional audio signal representation (220) of the input audio signal (1)].

[0572] The audio signal representation generator may cause a recurrent learnable layer to operate along a sequence of time steps, each time step having at least one state such that each time step is conditioned by [e.g., immediately following] the previous time step [the state of the previous time step may be the output] [may be, e.g. Fig.11 As shown, the steps and / or the output of each step is recursively provided to a subsequent time step, such as an immediately subsequent time step] [or, as Fig.12 As shown, there may be multiple feed-forward modules, each of which provides a state and / or output to a subsequent module, such as an immediately subsequent module] [In some examples, it can be understood that Fig.12 Implementations such as Fig.11 An expanded (e.g., developed) version of an implementation of the invention] [In examples, parameters of different moments and / or feedforward modules may generally be different from each other, but in some examples they may be the same].

[0573] The audio signal representation generator may be such that the output of the steps and / or each step is recursively provided to a subsequent time step.

[0574] The audio signal representation generator may include a plurality of feed-forward modules, each feed-forward module providing a state and / or output to a subsequent module.

[0575] The audio signal representation generator can enable a recurrent learnable layer to generate an output [target data (12)] for a given time moment by considering the output [target data (12)] and / or the state at the previous [e.g., immediately previous] time moment, wherein the correlation of the output and / or state at the previous [e.g., immediately previous] time moment is obtained through training.

[0576] As shown in the above example, an audio signal representation generator (2, 20) is disclosed for generating an output audio signal representation (3, 469) from an input audio signal (1) comprising a sequence of input audio signal frames, each input audio signal frame comprising a sequence of input audio signal samples, the audio signal representation generator (2, 20) comprising:

[0577] [e.g. deterministic] a format definer (210) configured to define a first multi-dimensional audio signal representation (220) of an input audio signal (1) [e.g. as above];

[0578] [an optional first learnable layer (230), e.g., a first convolutional learnable layer, which is a convolutional learnable layer configured to generate a second multidimensional audio signal representation (220) of the input audio signal (1) by sliding along a second direction [e.g., an intra-frame direction] of the first multidimensional audio signal representation (220) of the input audio signal (1);]

[0579] a second learnable layer (240) being a recurrent learnable layer configured to generate a third multidimensional audio signal representation of the input audio signal (1) by operating [e.g. using a 1x1 kernel, e.g. a 1x1 learnable kernel, or another kernel] along a first direction [e.g. an intra direction] of the second multidimensional audio signal representation (220) of the input audio signal (1);

[0580] a third learnable layer (250) [e.g., may be a second convolutional learnable layer], which is a convolutional learnable layer configured to generate a fourth multidimensional audio signal representation (265b') of the input audio by sliding [e.g., using a 1x1 kernel, e.g., a 1x1 learnable kernel] along a second direction [e.g., an intra-frame direction] of the first multidimensional audio signal representation of the input audio signal,

[0581] An output audio signal representation (269) is thereby obtained from the fourth [or second or third] multidimensional audio signal representation (265b') of the input audio signal (1) [for example, after adding the fourth multidimensional audio signal representation (265b') together with the main part of the multidimensional audio signal representation (220) of the input audio signal (1), or after block 290 and / or quantization block 300].

[0582] The audio signal representation generator may further include a first learnable layer (230) which is a convolutional learnable layer configured to generate a second multidimensional audio signal representation of the input audio signal (1) by sliding along a second direction of the first multidimensional audio signal representation (220) of the input audio signal (1).

[0583] The audio signal representation generator may cause applying the first learnable layer along a second dimension of the first multi-dimensional audio signal representation of the input audio signal.

[0584] The audio signal representation generator may be such that the first learnable layer is a residual learnable layer.

[0585] The audio signal representation generator may be such that at least the second learnable layer (240) and the third learnable layer (250) are residual learnable layers [e.g., a major portion of the first multidimensional audio signal representation (220) of the input audio signal bypasses (259') the first learnable layer (230), the second learnable layer (240) and the third learnable layer (250), and / or the first learnable layer (230), the second learnable layer (240) and the third learnable layer (250) are applied to at least a residual portion (259a) of the first two-dimensional audio signal representation (220) of the input audio signal (1)].

[0586] The audio signal representation generator may cause applying a first learnable layer [(eg, by sliding a kernel)] along a second dimension of the first multidimensional audio signal representation of the input audio signal.

[0587] The audio signal representation generator may cause applying a third learnable layer (eg, by a sliding kernel) along a second dimension of a third multidimensional audio signal representation of the input audio signal.

[0588] The audio signal representation generator may further comprise an encoder [and / or a quantizer] to encode the bitstream from the output audio signal representation.

[0589] The audio signal representation generator may further comprise at least one further learnable block (290) downstream of the at least one learnable block (230) [and / or upstream of the quantizer, the quantizer may be a learnable quantizer, e.g. a quantizer using a learnable codebook] to generate a fifth audio signal representation (469) of the input audio signal (1) from a fourth [or first, or second, or third, or another] multidimensional audio signal representation (269) of the input audio signal (1) [and / or from an output audio signal representation (3, 469) of the input audio signal (1)], having a plurality of samples [e.g. 256, or at least between 120 and 560] for each frame [e.g. 10ms, or 5ms, or 20ms] [the learnable block may be, e.g. a non-residual learnable block, and it may have a kernel which may be a learnable kernel, e.g. a 1x1 kernel].

[0590] The audio signal representation generator may be such that at least one further learnable block (290) downstream of the at least one learnable block (230) [and / or upstream of the quantizer] comprises:

[0591] At least one residual learnable layer [e.g., a main portion (459a') of the audio signal representation (429) bypasses (459') the first layer (430) [e.g., an activation function, such as Leaky ReLU] [the first bypassed layer 430 may therefore be a non-learnable activation function], at least one of the second learnable layer (440), the third layer (450) [e.g., an activation function, such as Leaky ReLU] and the fourth learnable layer (450) [e.g., not followed by an activation function] and / or at least one of the first layer (430), the second learnable layer (440), the third layer (450) and the fourth learnable layer (450) is applied to at least the residual portion (459a) of the audio signal representation (359a) of the input audio signal (1).

[0592] The audio signal representation generator may be such that at least one further learnable block (290) downstream of the at least one learnable block (230) [and / or upstream of the quantizer] comprises:

[0593] At least one convolutional learnable layer.

[0594] The audio signal representation generator may be such that at least one further learnable block (290) downstream of the at least one learnable block (230) [and / or upstream of the quantizer] comprises:

[0595] At least one learnable layer activated by an activation function (e.g. ReLu or Leaky ReLu).

[0596] The audio signal representation generator can make the activation function be ReLu or Leaky ReLu.

[0597] The audio signal representation generator may cause the format definer (210) to be configured to define a first multi-dimensional audio signal representation (220) of the input audio signal (1), the first multi-dimensional audio signal representation (220) of the input audio signal comprising at least:

[0598] a first dimension [e.g., an intra-frame dimension] such that a plurality of mutually consecutive [e.g., immediately adjacent] frames are ordered according to the first dimension; and

[0599] a second dimension [e.g., an intra-frame dimension] such that a plurality of samples of at least one frame are ordered according to the second dimension [the format definer may be configured to order mutually consecutive samples, e.g., immediately subsequent samples, one after another according to the second dimension].

[0600] As shown in the above example, an encoder (2) is disclosed, which includes an audio signal representation generator (20) and a quantizer (300) to encode a bitstream (3) from an output audio signal representation (269).

[0601] The encoder (2) may cause the quantizer (300) to be a learnable quantizer (300) [e.g., a quantizer using at least one learnable codebook], configured to associate an index of at least one codebook to each frame of the first multidimensional audio signal representation (290) of the input audio signal (1) or a processed version of the first multidimensional audio signal representation, thereby generating a bitstream [the at least one codebook may, for example, be a learnable codebook].

[0602] As shown in the above example, an encoder (2) for generating a bitstream (3) is disclosed, wherein an input audio signal (1) comprising a sequence of input audio signal frames is encoded, each input audio signal frame comprising a sequence of input audio signal samples, and the encoder (2) comprises:

[0603] A format definer (210) configured to define [e.g. generate] a first multi-dimensional audio signal representation (220) of an input audio signal, the first multi-dimensional audio signal representation of the input audio signal comprising at least:

[0604] a first dimension [e.g., an intra-frame dimension] such that a plurality of mutually consecutive frames [e.g., immediately following] are ordered according to the first dimension; and

[0605] a second dimension [e.g., an intra-frame dimension] such that a plurality of samples of at least one frame are ordered according to the second dimension [the format definer may be configured to order mutually consecutive samples, such as immediately subsequent samples, one after the other according to the second dimension],

[0606] Optionally, at least one intermediate layer [e.g. a deterministic layer and / or at least one learnable layer, such as a recurrent learnable layer, e.g. a GRU or LSTM)] to provide at least one processed version of the first multidimensional audio signal representation of the input audio signal;

[0607] A learnable quantizer [e.g., a quantizer using a learnable codebook, while the quantization itself may be deterministic or learnable] associates an index into at least one codebook to each frame of a processed version of a first multidimensional audio signal representation of a first multidimensional or input audio signal, thereby generating a bitstream.

[0608] As shown in the above example, an encoder for generating a bitstream is disclosed, wherein an input audio signal comprising a sequence of input audio signal frames is encoded, each input audio signal frame comprising a sequence of input audio signal samples, and the encoder comprises:

[0609] A learnable quantizer is configured to associate an index of at least one codebook to each frame of a first multidimensional audio signal representation of an input audio signal, thereby generating a bitstream.

[0610] As shown in the above example, an encoder is disclosed for generating a bit stream to encode an input audio signal comprising a sequence of input audio signal frames, each input audio signal frame comprising a sequence of input audio signal samples, the encoder comprising:

[0611] A format definer configured to define a first multi-dimensional audio signal representation of an input audio signal, the first multi-dimensional audio signal representation of the input audio signal comprising at least:

[0612] a first dimension [e.g., an intra-frame dimension] such that a plurality of mutually consecutive frames [e.g., immediately following] are ordered according to the first dimension; and

[0613] a second dimension [e.g., an intra-frame dimension] such that a plurality of samples of at least one frame are ordered according to the second dimension [the format definer may be configured to order mutually consecutive samples, such as immediately subsequent samples, one after the other according to the second dimension],

[0614] at least one intermediate learnable layer [e.g. such as a recurrent learnable layer, e.g. a GRU or LSTM, which may be residual, and which may be cascaded with the at least one convolutional learnable layer] to provide at least one processed version of the first multidimensional audio signal representation of the input audio signal;

[0615] A learnable quantizer associates an index of at least one codebook [eg, a learnable codebook] to each frame of a processed version of a first multidimensional audio signal representation of a first multidimensional or input audio signal, thereby generating a bitstream.

[0616] The encoder may cause the learnable quantizer [or quantizer] to associate the indexes [eg, i z 、i r 、i q , index i z Denotes the approximation E(x) and is taken from a codebook [e.g., a learnable codebook] z e Code z, index i r represents the approximation E(x)-z and is taken from a codebook [e.g., a learnable codebook] r e Code r, index i q Denotes the approximation E(x)-zr and is taken from a codebook [e.g., a learnable codebook]q e code q].

[0617] The encoder may make at least one codebook [eg, a learnable codebook] [eg, z e 、r e ,q e ] includes at least one base codebook [e.g., a learnable codebook] [e.g., z e], associating a multidimensional tensor [or other type of code, such as a vector] of a first multidimensional audio signal representation of the input audio signal to an index to be encoded in the audio signal representation [e.g., i z ].

[0618] The encoder may cause at least one codebook [eg, a learnable codebook] to include at least one residual codebook [eg, a learnable codebook] [eg, a first residual codebook, eg, r e , and may be a second residual codebook, such as q e , possibly even a lower-ordered residual codebook], associating a multidimensional tensor of a first multidimensional audio signal representation of an input audio signal to an index to be encoded in the audio signal representation.

[0619] The encoder may define multiple residual codebooks [e.g., learnable codebooks] such that:

[0620] a second residual codebook [e.g. a second residual learnable codebook], associating a multidimensional tensor representing a second residual part of the first multidimensional audio signal representation of the input audio signal to an index to be encoded in the audio signal representation,

[0621] a first residual codebook [e.g. a second residual learnable codebook] associating a multidimensional tensor representing a first residual part of a frame of a first multidimensional audio signal representation to an index to be encoded in the audio signal representation,

[0622] Wherein the second residual portion of the frame is residual [eg, low ranking] with respect to the first residual portion of the frame.

[0623] The encoder can be configured to signal in the bitstream (3) whether the index associated to the residual frame is encoded, and the quantization index (313) is correspondingly encoded according to the signal [and, in the case of different orderings, the bitstream can signal which indices of which ordering are encoded, and / or at least one codebook [e.g., a learnable codebook] (313) for example reads only the encoded indices correspondingly according to the signal].

[0624] The encoder may cause at least one codebook [eg a learnable codebook] to be a fixed-length codebook [eg at least one codebook has a number of bits between 4 and 20, such as between 8 and 12, such as 10].

[0625] The encoder may further include [e.g. in an intermediate layer or downstream of the intermediate layer, but upstream of the quantizer] at least one further learnable block (290) downstream of the at least one learnable block (230) [and / or upstream of the quantizer, the quantizer may be a learnable quantizer, e.g. a quantizer using a learnable codebook] to generate a fifth audio signal representation of the input audio signal (1) from a fourth multidimensional audio signal representation (269) or another version of the input audio signal (1), having a plurality of samples [e.g. 256, or at least between 120 and 560] for each frame [e.g. 10ms, or 5ms, or 20ms] [the learnable block may be, for example, a non-residual learnable block, and it may have a kernel which may be a learnable kernel, e.g. a 1x1 kernel].

[0626] The encoder may cause at least one further learnable block (290) located downstream of the at least one learnable block (230) [and / or upstream of the quantizer] to include:

[0627] At least one residual learnable layer [e.g., a main portion (459a') of the audio signal representation (429) bypasses (459') at least one of the first learnable layer (430), the second learnable layer (440), the third learnable layer (450) and the fourth learnable layer (450), and / or at least one of the first learnable layer (430), the second learnable layer (440), the third learnable layer (450) and the fourth learnable layer (450) is applied to at least a residual portion (459a) of the audio signal representation (359a) of the input audio signal (1)].

[0628] The encoder may cause at least one further learnable block (290) located downstream of the at least one learnable block (230) [and / or upstream of the quantizer] to include:

[0629] At least one convolutional learnable layer.

[0630] The encoder may cause at least one further learnable block (290) located downstream of the at least one learnable block (230) [and / or upstream of the quantizer] to include:

[0631] At least one learnable layer activated by an activation function (e.g. ReLu or Leaky ReLu).

[0632] The encoder may be such that a training session is performed by generating a plurality of bitstreams having candidate indices associated with known frames representing a known audio signal, the training session comprising decoding of the bitstreams and evaluation of an audio signal generated by the decoding with respect to the known audio signal, thereby adjusting the association of indices of at least one codebook [e.g. a learnable codebook] with frames of the encoded bitstream [e.g. by minimizing the difference between the generated audio signal and the known audio signal] [e.g. using a GAN].

[0633] The encoder may cause a training session to be performed by receiving at least:

[0634] a plurality of first bitstreams having first candidate indices, the first candidate indices having a first bit length and being associated with first known frames representing a known audio signal, the first candidate indices forming a first candidate codebook, and

[0635] a plurality of second bitstreams having second candidate indices, the second candidate indices having a second bit length and being associated with known frames representing the same first known audio signal, the second candidate indices forming a second candidate codebook,

[0636] wherein the first bit length is higher than the second bit length [and / or the first bit length has a higher resolution but it occupies more frequency band than the second bit length],

[0637] The training session includes an evaluation of a generated audio signal obtained from a plurality of second bitstreams compared to a generated audio signal obtained from a plurality of first bitstreams, thereby selecting a codebook [e.g., such that the selected learnable codebook is a codebook selected between a first candidate codebook and a second candidate codebook] [e.g., there may be an evaluation of a first ratio compared to a second ratio, the first ratio being a ratio between metrics measuring the quality of the audio signals generated from the plurality of first bitstreams with respect to bit length, the second ratio being a ratio between metrics measuring the quality of the audio signals generated from the plurality of second bitstreams with respect to bit rate (sampling rate)), and selecting a bit length that maximizes the ratio] [e.g., this may be repeated for each codebook, e.g., primary, first residual, second residual, etc.].

[0638] The encoder may cause a training session to be performed by receiving:

[0639] a first plurality of first bitstreams having first indices associated with first known frames representing a known audio signal, wherein the first indices are in a first maximum number, the first plurality of first candidate indices forming a first candidate codebook; and

[0640] a second plurality of second bitstreams having second indices associated with known frames representing the same first known audio signal, the second plurality of second candidate indices forming a second candidate codebook, wherein the second indices are within a second maximum number different from the first maximum number,

[0641] The training session includes an evaluation of generated audio signals obtained from a first plurality of first bitstreams compared to generated audio signals obtained from a second plurality of second bitstreams, thereby selecting a learnable index [e.g., such that the selected learnable codebook is selected among a first candidate codebook and a second candidate codebook] [e.g., there may be an evaluation of a first ratio between a metric measuring the quality of audio signals generated from the first plurality of first bitstreams compared to a second ratio between a metric measuring the quality of audio signals generated from the second plurality of second bitstreams with respect to a bit rate (sampling rate), and selecting the plurality among the first plurality and the second plurality that maximizes the ratio] [e.g., this may be repeated for each codebook, e.g., primary, first residual, second residual, etc.].

[0642] In the learnable layer 240 of the encoder, it may have a recurrent learnable layer (e.g., GRU), which in some examples may be configured to generate an output (e.g., provided to the convolutional layer 250) by considering the output and / or state at the previous [e.g., immediately previous] time moment (e.g., for a given time moment), where the correlation of the output [target data (12)] and / or the state at the previous [e.g., immediately previous] time moment can be obtained through training.

[0643] The recurrent learnable layer of the learnable layer 240 can operate along a series of time steps, each time step having at least one state, such that each time step is conditioned by the output and / or state of the previous time step [the state of the previous time step can be the output] [can be, for example Fig.11 As shown, the output of the step and / or each step is recursively provided to a subsequent time step, such as the time step immediately following it] [or, as Fig.12 As shown, there may be multiple feed-forward modules, each of which provides a state and / or output to a subsequent module (eg, an immediately subsequent module) [In some examples, Fig.12 The implementation can be understood as Fig.11 An expanded version of an implementation of ][In examples, the parameters of different time moments and / or feedforward modules may generally be different from each other, but in some examples they may be the same].

[0644] The GRU of the learnable layer 240 may also include multiple feed-forward modules, each of which provides a state and / or output to an immediately subsequent module.

[0645] The GRU of the learnable layer 240 may be configured to generate states and / or outputs in the following manner:

[0646] [h t ][For a specific tth state or module]:

[0647] By updating the gate vector [z t] weights the candidate states and / or outputs [elements of which may have values ​​between 0 and 1, or another value between 0 and c, where c>0] to generate a first weighted addend; and

[0648] The gate vector z is updated by a vector whose components are complemented to 1. t The state and / or output [h t-1 ] are weighted to generate a second weighted addend; and

[0649] Add the first addend to the second addend

[0650] [Update gate vector [z t ] can provide information about how much to take from the candidate state and / or output and how much to take from the state and / or output at the previous time step [h t-1 ] to obtain information; for example, if z t = 0, the state and / or output at the current time step is only taken from the state and / or output of the previous time step [h t-1 ]; and if z t =1, then the state and / or output at the current moment is only taken from the candidate vector].

[0651] The GRU of the learnable layer 240 may be configured such that the recurrent learnable layer generates states and / or outputs [h t ]:

[0652] By complementing the weighted vectors with each other, adding the weighted version of the candidate state and / or the output with the weighted version of the state and / or the output h of the previous time step t-1 .

[0653] The GRU of the learnable layer 240 may be configured to generate candidate states and / or outputs by applying at least the weight parameters [W] obtained through training to:

[0654] Reset gate vector [r t ] and the state and / or output of the previous time step [h t-1 ], and the element-by-element product of the input [x t ]cascade;

[0655] Optionally followed by an activation function (e.g. tanH)

[0656] [Reset gate vector [r t ] provides information about the state and / or output of the previous time step [h t-1 ]How much information should be reset][If r t = 0, we reset everything and do not start from h t-1 retains any content, and if r tHigher, then we start from h t-1 Keep more].

[0657] The GRU of the learnable layer 240 may also be configured to apply an activation function after applying the weight parameter W. The audio generator may be such that the activation function is TanH.

[0658] The GRU of the learnable layer 240 may be configured to generate candidate states and / or outputs by at least:

[0659] The vector conditioned by:

[0660] For the current input [x t ]and

[0661] Weighted to reset gate vector [r t ] on the previous time step and / or output [h t-1 ], [The reset gate vector gives information about the state and / or output of the previous time step [h t-1 ]The information that should be reset][If r t = 0, we reset everything, and we do not start from h t-1 retains any content, and if r t Higher, then we start from h t-1 Keep more]

[0662] The GRU of the learnable layer 240 can be configured by setting the parameter [W z ] is applied to the following cascade to generate the update gate vector [z t ]:

[0663] Loop module [h t-1 ] input [h t-1 ] Cascaded to

[0664] For the current input [x t ][e.g., input to at least one pre-conditioned learnable layer (710)],

[0665] An activation function (eg, a sigmoid function, σ) is optionally then applied.

[0666] When the parameter W is applied z After that, an activation function can be applied. The activation function is the sigmoid function, σ.

[0667] Reset gate vector r t The weight parameter W r Applied to the concatenation of the following two to obtain:

[0668] The state and / or output h at the previous time step t-1and

[0669] For the current input x t .

[0670] When the parameter W is applied r After that, an activation function can be applied. The activation function is the sigmoid function σ.

[0671] The audio generator may enable a training session to be performed by receiving:

[0672] a first plurality of first bit streams having first indices representing codes obtained from a known audio signal, the first plurality of first bit streams forming at least one first codebook [e.g. at least one primary codebook z e ];as well as

[0673] A second plurality of second bit streams, comprising first indices representing primary codes obtained from a known audio signal and second indices representing residual codes with respect to the primary codes, the second plurality of second bit streams forming at least one first codebook [e.g., at least one primary codebook z e ] and at least one second codebook [e.g. at least one residual codebook r e ];

[0674] The training session comprises an evaluation of generated audio signals obtained from a first plurality of first bitstreams compared to generated audio signals obtained from a second plurality of second bitstreams,

[0675] Thus in the first plurality [and / or the first candidate codebook z e ] and the second plurality [ and / or the first candidate codebook z e As the main codebook, together with the residual codebook r e at least one second codebook of] [e.g., such that the selected learnable codebook is selected among the first candidate codebook and the first candidate codebook] [e.g., there may be an evaluation of a first ratio between metrics measuring the quality of audio signals generated from a first plurality of first bitstreams compared to a second ratio between metrics measuring the quality of audio signals generated from a second plurality of second bitstreams with respect to a bit rate (sampling rate), and selecting the plurality of them that maximizes the ratio] [e.g., this may repeat the codebooks, such as main, first residual, second residual, etc., for each bitstream].

[0676] As shown in the above examples, a method for training an audio signal generator [e.g., a decoder] is disclosed that may include a training session that includes generating a plurality of bitstreams having candidate indices associated with known frames representing a known audio signal, the training session including decoding of the bitstreams and evaluating an audio signal generated by the decoding with respect to the known audio signal, thereby adjusting the association of at least one codebook index with a frame of the encoded bitstream [e.g., by minimizing the difference between the generated audio signal and the known audio signal] [e.g., using a GAN].

[0677] As shown in the above example, a method for training an audio signal generator [e.g., a decoder] as described above is disclosed, which may include a training session, including generating a plurality of bitstreams having candidate indices associated with known frames representing a known audio signal, the training session including providing the audio signal generator with a bitstream not provided by the encoder, thereby obtaining the index to be used [e.g., obtaining a codebook] by optimizing a loss function.

[0678] As shown in the above examples, a method for training an audio signal generator [e.g., a decoder] as described above is disclosed, which may include a training session, the training session including generating multiple output audio signal representations of a known input audio signal, the training session including evaluating the multiple output audio signal representations [e.g., bitstreams] with respect to the known input audio signal and / or minimizing a loss function, thereby adjusting parameters of at least one or more learnable layers that optimize the loss function.

[0679] As shown in the above examples, a method for training an audio signal representation generator (or encoder) as described above is disclosed, which may include a training session, the training session including receiving a plurality of bitstreams having indices associated with known frames representing known audio signals, the training session including an evaluation of the generated audio signal with respect to the known audio signal, thereby adjusting the association of at least one codebook index with the frame of the encoded bitstream and / or optimizing a loss function [e.g., by minimizing the difference between the generated audio signal and the known audio signal] [e.g., using a GAN].

[0680] As shown in the above example, a method for training an audio signal representation generator (or encoder) as described above and an audio signal generator [e.g., decoder] as described above is disclosed, which may include:

[0681] providing a plurality of audio signals (1) to an audio signal representation generator, thereby obtaining audio signal representations and / or bitstreams (3), and generating, at the audio signal generator (10), output signal (16) representations from the audio signals and / or bitstreams (3);

[0682] providing a plurality of audio signal representations and / or bitstreams (3) not generated by the audio signal representation generator (20) to the audio signal generator (10), and generating, at the audio signal generator (10), output signals (16) from the audio signal representations and / or bitstreams (3);

[0683] A loss function associated to an output signal (16) from the audio signal representation and / or bitstream (3) is evaluated compared to the output signal (16) from the audio signal representation and / or bitstream (3), thereby obtaining parameters of the learnable layers and blocks of the audio signal generator (10) and the audio signal representation generator by minimizing the loss function.

[0684] As shown in the above example, a method for generating an audio signal (16) from a bit stream (3) is disclosed, wherein the bit stream (3) represents the audio signal (16), and the audio signal is subdivided into a sequence of frames. The method may include:

[0685] providing, for a given frame, first data (15) derived from an input signal (14) [e.g. from an external or internal source or from a bitstream (3)], [wherein the first data (15) may have a single channel or multiple channels];

[0686] By a first processing block (40, 50, 50a-50h), receiving [e.g., for a given frame] first data (15) and outputting first output data (69) in the given frame, [wherein the first output data (69) may include a single channel or multiple channels (47)],

[0687] [For example, the method further comprises receiving, by a second processing block (45), for example for a given frame, the first output data (69) or data derived from the first output data (69) as the second data,]

[0688] Wherein, the first processing block (50) comprises:

[0689] At least one pre-conditioned learnable layer (710) receives the bitstream (3) or a processed version thereof (112) and outputs, for a given frame, target data (12) representing an audio signal (16) in a given frame [e.g., multiple channels and multiple samples for a given frame];

[0690] At least one conditional learnable layer (71, 72, 73) processes the target data (12), for example for a given frame, to obtain conditional feature parameters (74, 75) for the given frame; and

[0691] A style element (77) that applies conditional feature parameters (74, 75) to the first data (15, 59a) or the normalized first data (59, 76');

[0692] [wherein the second processing block (45), if present, may combine the plurality of channels (47) of the second data (69) to obtain the audio signal (16)],

[0693] Wherein the at least one pre-conditioned learnable layer (710) comprises at least one recurrent learnable layer [e.g., a gated recurrent learnable layer, such as a gated recurrent unit, a GRU, or an LSTM]

[0694] [For example, the audio signal (16) is obtained from the first output data (69) or a processed version of the first output data (69)].

[0695] As shown in the above example, a method for generating an audio signal (16) from a bit stream (3) is disclosed, wherein the bit stream (3) represents the audio signal (16), the bit stream (3) is subdivided into a sequence of indexes, and the audio signal is subdivided into a sequence of frames. The method may include:

[0696] The quantization index converter step (313) [also referred to as an index to code converter step, an inverse quantizer step, an inverse quantizer step, etc.] converts the index of the bitstream (13) to a code [e.g., according to an example, the code may be a scalar, a vector or more generally a tensor] [e.g., according to a codebook, e.g., a tensor may be multi-dimensional, such as a matrix or a generalization thereof in multiple dimensions, such as three dimensions, four dimensions, etc.] [e.g., the codebook may be learnable or may be deterministic],

[0697] The first data provider step (702) provides, for example for a given frame, first data (15) derived from an input signal (14) from an external or internal source or from a bitstream (3), [wherein the first data (15) may have a single channel or multiple channels]

[0698] The steps of using a first processing block (40, 50, 50a-50h), for example for a given frame to receive first data (15) and output first output data (69) in the given frame [wherein the first output data (69) may include a single channel or multiple channels (47)], and

[0699] [There may be a second processing block (45), for example for a given frame to receive the first output data (69) or data derived from the first output data (69) as second data],

[0700] The first processing block (50) comprises:

[0701] at least one pre-conditioned learnable layer (710) to receive a bitstream (3) or a processed version thereof (112) and to output, for a given frame, target data (12) of an audio signal (16) represented in the given frame [e.g., multiple channels and multiple samples for the given frame];

[0702] at least one conditional learnable layer (71, 72, 73), e.g., processing the target data (12) for a given frame to obtain conditional feature parameters (74, 75) for the given frame; and

[0703] A style element (77) that applies conditional feature parameters (74, 75) to the first data (15, 59a) or the normalized first data (59, 76');

[0704] [Wherein the second processing block (45), if present, may combine the plurality of channels (47) of the first output data or the second output data (69) to obtain the audio signal (16)]

[0705] [For example, the audio signal (16) is obtained from the first output data (69) or a processed version (16) of the first output data (69)].

[0706] As shown in the above example, a method for generating an output audio signal representation (3, 469) from an input audio signal (1) comprising a sequence of input audio signal frames, each input audio signal frame comprising a sequence of input audio signal samples, an audio signal representation generator (2, 20) is disclosed.

[0707] This can include:

[0708] defining a first multi-dimensional audio signal representation (220) of the input audio signal (1) [e.g., same as above];

[0709] generating, by a first learnable layer (230), [e.g., a first convolutional learnable layer, which is a convolutional learnable layer], a second multidimensional audio signal representation of the input audio signal (1) by sliding along a second direction [e.g., an intra-frame direction] of the first multidimensional audio signal representation (220) of the input audio signal (1);

[0710] generating, by a second learnable layer (240), the second learnable layer (240) being a recurrent learnable layer, a third multidimensional audio signal representation of the input audio signal (1) by operating along a first direction [e.g. an intra direction] of the second multidimensional audio signal representation (220) of the input audio signal (1) [e.g. using a 1x1 kernel, e.g. a 1x1 learnable kernel or another kernel];

[0711] by a third learnable layer (250) [which may be, for example, a second convolutional learnable layer], which is a convolutional learnable layer [e.g., using a 1x1 kernel, e.g., a 1x1 learnable kernel] that generates a fourth multidimensional audio signal representation (265b′) by sliding along a second direction [e.g., an intra-frame direction] of the first multidimensional audio signal representation of the input audio signal,

[0712] An output audio signal representation (469) is thereby obtained from the fourth multidimensional audio signal representation (265b') of the input audio signal (1) [e.g. after adding the fourth multidimensional audio signal representation (265b') to the main part of the multidimensional audio signal representation (220) of the input audio signal (1), or after block 290 and / or quantization block 300].

[0713] The non-transitory storage unit storing instructions may, when executed by a processor, cause the processor to perform the method as described above.

[0714] Further examples

[0715] In general, an example can be implemented as a computer program product with program instructions, which are operable to perform one of the methods when the computer program product is run on a computer. The program instructions can be stored, for example, on a machine-readable medium. Other examples include a computer program stored on a machine-readable carrier for performing one of the methods described herein. In other words, therefore, an example of a method is a computer program with program instructions, which are used to perform one of the methods described herein when the computer program is run on a computer. Therefore, another example of the method is a data carrier medium (or a digital storage medium, or a computer-readable medium) including a computer program recorded thereon for performing one of the methods described herein. A data carrier medium, a digital storage medium, or a recording medium is tangible and / or non-transient, rather than an intangible and temporary signal. Therefore, another example of the method is a data stream or signal sequence representing a computer program for performing one of the methods described herein. The data stream or signal sequence can be transmitted, for example, via a data communication connection (e.g., via the Internet). Yet another example includes a processing device, such as a computer or a programmable logic device, that performs one of the methods described herein. Further examples include a computer having a computer program for performing one of the methods described herein installed thereon. Another example includes a device or system for transmitting (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The device or system may, for example, include a file server for transmitting the computer program to the receiver. In some examples, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some examples, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, these methods may be performed by any appropriate hardware device. The above examples are merely illustrative of the above principles. It should be understood that modifications and changes to the arrangements and details described herein will be apparent. Therefore, it is intended to be limited by the scope of the claims, rather than by the specific details presented by the description and explanation of the examples herein. Even if they appear in different drawings, the same or equivalent elements or elements having the same or equivalent functions are represented by the same or equivalent reference numerals in the following description.

[0716] In addition, further examples are defined by the attached claims (examples are also in the claims). It should be noted that any examples defined by the claims can be supplemented by any details (features and functions) described in the following chapters. In addition, the examples described in the above paragraphs can be used alone and can also be supplemented by any features in another chapter or any features included in the claims. The text in parentheses and square brackets is optional and defines further embodiments (those further defined by the claims). In addition, it should be noted that the various aspects described herein can be used alone or in combination. Therefore, details can be added to each of the individual aspects without adding details to another aspect. It should also be noted that the present disclosure explicitly or implicitly describes the features of mobile communication devices and receivers and mobile communication systems. According to certain implementation requirements, the examples can be implemented in hardware. The implementation can be performed using a digital storage medium, such as a floppy disk, a digital versatile disk (DVD), a Blu-ray disc, a compact disk (CD), a read-only memory (ROM), a programmable read memory-dedicated memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory, on which electronically readable control signals are stored, which cooperate (or can cooperate) with a programmable computer system to perform the corresponding method. Therefore, the digital storage medium can be computer-readable. In general, an example can be implemented as a computer program product with program instructions, and when the computer program product is run on a computer, the program instructions are operable to perform one of the methods. The program instructions can be stored, for example, on a machine-readable medium. Other examples include a computer program for performing one of the methods described herein stored on a machine-readable carrier. In other words, therefore, an example of a method is a computer program with program instructions, which, when the computer program is run on a computer, is used to perform one of the methods described herein. Therefore, another example of the method is a data carrier medium (or a digital storage medium, or a computer-readable medium), which includes a computer program recorded thereon for performing one of the methods described herein. A data carrier medium, a digital storage medium, or a recording medium is tangible and / or non-transient, rather than an intangible and transient signal. Yet another example includes a processing unit, such as a computer or a programmable logic device, that performs one of the methods described herein. A further example includes a computer having a computer program for performing one of the methods described herein installed thereon. A further example includes a device or system for transmitting (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The device or system may, for example, include a file server for transmitting the computer program to the receiver. In some examples, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein.In some examples, a field programmable gate array can cooperate with a microprocessor to perform one of the methods described herein. In general, these methods can be performed by any appropriate hardware device. The above examples are illustrations of the above principles. It should be understood that modifications and changes to the arrangements and details described herein will be apparent. Therefore, it is intended to be limited by the scope of the attached patent claims rather than by the specific details presented by the description and explanation of the examples herein.

Claims

1. An audio signal representation generator (2, 20) for generating an output audio signal representation (3, 469) from an input audio signal (1) comprising a sequence of input audio signal frames, each input audio signal frame comprising a sequence of input audio signal samples, the audio signal representation generator comprising: A format definer (210) configured to define a first multi-dimensional audio signal representation (220) of the input audio signal (1), the first multi-dimensional audio signal representation (220) of the input audio signal comprising at least: a first dimension, such that a plurality of mutually consecutive frames are sorted according to the first dimension; and a second dimension such that a plurality of samples of at least one frame are ordered according to the second dimension, and at least one learnable layer (230, 250, 290, 300) is configured to process the first multidimensional audio signal representation (220) of the input audio signal (1) or a processed version of the first multidimensional audio signal representation to generate the output audio signal representation (3, 469) of the input audio signal (1).

2. An audio signal representation generator according to claim 1, wherein the format definer (210) is configured to interpolate input audio signal samples for each given frame along the second dimension of the first multidimensional audio signal representation of the input audio signal.

3. An audio signal representation generator according to claim 1 or 2, wherein the format definer (210) is configured to insert additional input audio signal samples of one or more additional frames immediately following the given frame along the second dimension of the first multidimensional audio signal representation (220) of the input audio signal (1).

4. An audio signal representation generator according to any one of claims 1 to 3, wherein the format definer (210) is configured to insert additional input audio signal samples of one or more additional frames immediately preceding the given frame along the second dimension of the first multidimensional audio signal representation (220) of the input audio signal (1).

5. The audio signal representation generator according to any one of claims 1-4, wherein the at least one learnable layer comprises at least one recurrent learnable layer (240).

6. The audio signal representation generator according to claim 5, wherein the at least one recurrent learnable layer (240) comprises a gated recurrent unit (GRU).

7. An audio signal representation generator according to claim 5 or 6, wherein the at least one recurrent learnable layer (240) operates along the first dimension.

8. The audio signal representation generator according to any of the preceding claims, further comprising at least one first convolutional learnable layer (230) located between the format definer (210) and the at least one recurrent learnable layer (240).

9. The audio signal representation generator according to claim 8, wherein in the at least one first convolutional learnable layer (230), a kernel slides along a second direction of the first multidimensional audio signal representation (220) of the input audio signal (1).

10. The audio signal representation generator according to any of the preceding claims, further comprising at least one convolutional learnable layer (250) located downstream of the at least one recurrent learnable layer (240).

11. The audio signal representation generator according to claim 10, wherein in the at least one convolutional learnable layer (250), a kernel slides along the second direction of the first multidimensional audio signal representation (220) of the input audio signal (1).

12. An audio signal representation generator according to any one of the preceding claims, wherein at least one or more of the at least one learnable layer is a residual learnable layer.

13. An audio signal representation generator according to claim 12, wherein at least one learnable layer (230, 240, 250) is a residual learnable layer, a main part of the first multidimensional audio signal representation (220) of the input audio signal bypasses (259') the at least one learnable layer (230, 240, 250), and / or the at least one learnable layer (230, 240, 250) is applied to at least a residual part (259a) of the first two-dimensional audio signal representation (220) of the input audio signal (1).

14. An audio signal representation generator according to any one of claims 5-13, wherein the recurrent learnable layer (240) operates along a series of time steps, each time step having at least one state, such that each time step is conditioned by the output and / or state of the previous time step.

15. The audio signal representation generator according to claim 14, wherein: The output of the step and / or each step is recursively provided to the following time step.

16. An audio signal representation generator according to claim 14 or 15, comprising a plurality of feed-forward modules, each feed-forward module providing the state and / or output to a subsequent module.

17. An audio signal representation generator according to any one of claims 5-16, wherein the recurrent learnable layer (240) generates an output for a given time instant by taking into account the output and / or state at a previous time instant, wherein the correlation of the output and / or state at the previous time instant is obtained by training.

18. An audio signal representation generator according to any of the preceding claims, the format definer being initially configured to order mutually consecutive samples one after the other according to the second dimension.

19. An audio signal representation generator (2, 20) for generating an output audio signal representation (3, 469) from an input audio signal (1) comprising a sequence of input audio signal frames, each input audio signal frame comprising a sequence of input audio signal samples, the audio signal representation generator (2, 20) comprising: a format definer (210) configured to define a first multi-dimensional audio signal representation (220) of the input audio signal (1); a second learnable layer (240), the second learnable layer being a recurrent learnable layer configured to generate a third multidimensional audio signal representation of the input audio signal (1) by operating in a first direction of the first multidimensional audio signal representation (220) or a processed version thereof, the processed version being a second multidimensional audio signal representation of the input audio signal (1); a third learnable layer (250), the third learnable layer being a convolutional learnable layer configured to generate a fourth multidimensional audio signal representation (265b') of the input audio signal by sliding along the second direction of the third multidimensional audio signal representation of the input audio signal, Thereby an output audio signal representation (269) is obtained from said fourth multidimensional audio signal representation (265b') of said input audio signal (1).

20. The audio signal representation generator according to claim 19, further comprising a first learnable layer (230), which is a convolutional learnable layer configured to generate a second multidimensional audio signal representation of the input audio signal (1) by sliding along a second direction of the first multidimensional audio signal representation (220) of the input audio signal (1).

21. The audio signal representation generator of claim 20, wherein the first learnable layer is applied along a second dimension of the first multi-dimensional audio signal representation of the input audio signal.

22. An audio signal representation generator according to claim 21, wherein the first learnable layer is a residual learnable layer.

23. An audio signal representation generator according to any one of claims 19-22, wherein at least the second learnable layer (240) or the third learnable layer (250) is a residual learnable layer.

24. The audio signal representation generator according to any of claims 19-23, wherein the third learnable layer is applied along a second dimension of the third multidimensional audio signal representation of the input audio signal.

25. An audio signal representation generator according to any one of claims 19-24, further comprising an encoder or quantiser for encoding a bit stream from the output audio signal representation.

26. An audio signal representation generator according to any one of claims 19-25, further comprising at least one further learnable block (290) located downstream of the at least one learnable block (230) to generate a fifth audio signal representation (469) of the input audio signal (1) from the fourth multidimensional audio signal representation (269) of the input audio signal (1), wherein multiple samples are used for each frame.

27. The audio signal representation generator according to claim 26, wherein the at least one further learnable block (290) downstream of the at least one learnable block (230) comprises: At least one residual learnable layer, a second learnable layer (440), a third layer (450), and a fourth learnable layer (450).

28. An audio signal representation generator according to claim 26 or 27, wherein the at least one further learnable block (290) downstream of the at least one learnable block (230) comprises: At least one convolutional learnable layer.

29. An audio signal representation generator according to any one of claims 26 to 28, wherein the at least one further learnable block (290) downstream of the at least one learnable block (230) comprises: At least one learnable layer activated by an activation function.

30. An audio signal representation generator according to claim 29, wherein the activation function is ReLu or LeakyReLu.

31. An audio signal representation generator according to any one of claims 19 to 30, wherein the format definer (210) is configured to define a first multidimensional audio signal representation (220) of the input audio signal (1), the first multidimensional audio signal representation (220) of the input audio signal comprising at least: A first dimension, such that a plurality of mutually consecutive frames are sorted according to the first dimension; as well as The second dimension causes the plurality of samples of at least one frame to be sorted according to the second dimension.

32. An encoder (2) comprising an audio signal representation generator (20) according to any one of the preceding claims and a quantizer (300) for encoding a bit stream (3) from an output audio signal representation (269).

33. The encoder (2) according to claim 32, wherein the quantizer (300) is a learnable quantizer (300) configured to associate an index of at least one codebook to each frame of the first multidimensional audio signal representation (290) of the input audio signal (1), or a processed version of the first multidimensional audio signal representation, thereby generating a bitstream.

34. The encoder of claim 33, wherein the learnable quantizer uses at least one codebook corresponding to the index i to be encoded in the bitstream z 、i r 、i q association, where index i z Denotes the code z that approximates E(x) and is taken from codebook z e , index i r represents the approximation E(x)-z and is taken from the codebook r e The code r, and index i q represents the approximation E(x)-zr and is taken from the codebook q to be encoded in the bitstream e The code for q.

35. The encoder according to any of claims 33-34, wherein the at least one codebook comprises at least one base codebook, the at least one base codebook associating a multidimensional tensor of the first multidimensional audio signal representation of the input audio signal to an index to be encoded in the bitstream.

36. The encoder according to any of claims 33 to 35, wherein the at least one codebook comprises at least one residual codebook associating a multidimensional tensor of the first multidimensional audio signal representation of the input audio signal to an index to be encoded in the bitstream.

37. The encoder according to any one of claims 33 to 36, wherein a plurality of residual codebooks are defined such that: a second residual codebook for associating a multidimensional tensor representing a second residual part of the first multidimensional audio signal representation of the input audio signal to an index to be encoded in the audio signal representation, a first residual codebook for associating a multidimensional tensor representing a first residual part of a frame of a first multidimensional audio signal to an index to be encoded in a representation of the audio signal, Wherein the second residual portion of the frame is residual with respect to the first residual portion of the frame.

38. Encoder according to any of claims 32-37, configured to signal in the bitstream (3) whether an index associated to a residual frame is encoded or not.

39. An encoder according to any one of claims 33 to 38, wherein at least one codebook is a fixed length codebook.

40. An encoder according to any one of claims 32-39, further comprising at least one further learnable block (290) located downstream of the at least one learnable block (230), for generating a fifth audio signal representation of the input audio signal (1) from the fourth multidimensional audio signal representation (269) or another version of the input audio signal (1), wherein multiple samples are used for each frame.

41. The encoder of claim 40, wherein the at least one further learnable block (290) downstream of the at least one learnable block (230) comprises: At least one residual learnable layer.

42. The encoder according to claim 40 or 41, wherein the at least one further learnable block (290) downstream of the at least one learnable block (230) comprises: At least one convolutional learnable layer.

43. The encoder according to any one of claims 40 to 42, wherein the at least one further learnable block (290) downstream of the at least one learnable block (230) comprises: At least one learnable layer activated by an activation function (e.g. ReLu or Leaky ReLu).

44. A method for generating an output audio signal representation (3, 469) from an input audio signal (1) comprising a sequence of input audio signal frames, each input audio signal frame comprising a sequence of input audio signal samples, the audio signal representation generator (2, 20) comprising: defining a first multi-dimensional audio signal representation (220) of the input audio signal (1); generating, by a first learnable layer (230), a second multidimensional audio signal representation (220) of the input audio signal (1) by sliding along a second direction of the first multidimensional audio signal representation (220) of the input audio signal (1); generating, by a second learnable layer (240), a third multidimensional audio signal representation of the input audio signal (1) by operating along a first direction of the second multidimensional audio signal representation (220) of the input audio signal (1), the second learnable layer (240) being a recurrent learnable layer; generating, by a third learnable layer (250), a fourth multidimensional audio signal representation (265b') of the input audio signal by sliding along the second direction of the first multidimensional audio signal representation of the input audio signal, the third learnable layer (250) being a convolutional learnable layer, The output audio signal representation (469) is thereby obtained from the fourth multidimensional audio signal representation (265b') of the input audio signal (1).

45. A non-transitory storage unit storing instructions which, when executed by a processor, cause the processor to perform the method according to claim 44.

46. ​​An audio generator (10) configured to generate an audio signal (16) from a bit stream (3), the bit stream (3) representing the audio signal (16), the audio signal being subdivided into a sequence of frames, the audio signal generator (10) comprising: A first data provider (702) configured to provide first data (15) derived from an input signal (14) for a given frame, wherein the first data (15) has a plurality of channels; a first processing block (40, 50, 50a-50h) configured to receive the first data (15) for the given frame and output first output data (69) in the given frame, wherein the first output data (69) can include a plurality of channels (47), The audio generator further comprises a second processing block (45) configured to receive, for the given frame, the first output data (69) or data derived from the first output data (69) as second data, The first processing block (50) comprises: at least one pre-conditioned learnable layer (710) configured to receive the bitstream (3) or a processed version thereof (112) and output, for the given frame, target data (12) representing the audio signal (16) in the given frame, wherein a plurality of channels and a plurality of samples are used for the given frame; at least one conditional learnable layer (71, 72, 73) configured to process the target data (12) for the given frame to obtain conditional feature parameters (74, 75) for the given frame; and a style element (77) configured to apply conditional feature parameters (74, 75) to the first data (15, 59a) or the normalized first data (59, 76'); wherein the second processing block (45) is configured to combine the plurality of channels (47) of the second data (69) to obtain the audio signal (16), wherein the at least one pre-processing learnable layer (710) comprises at least one recurrent learnable layer.

47. An audio generator according to claim 46, wherein the recurrent learnable layer is configured to generate an output for a given moment by considering the output and / or state at a previous moment, wherein the output is target data (12), wherein the correlation of the output and / or state at the previous moment is obtained through training.

48. An audio generator according to any one of claims 46-47, wherein the recurrent learnable layer operates along a series of time steps, each time step having at least one state, such that each time step is conditioned by the output and / or state of the previous time step.

49. The audio generator of claim 48, further comprising a plurality of feed-forward modules, each feed-forward module providing the state and / or output to an immediately subsequent module.

50. An audio generator according to any one of claims 46-49, wherein the recurrent learnable layer is configured to generate a state and / or output h for a particular t-th state or module by: t : By updating the gate vector z t weighting the candidate states and / or outputs to generate a first weighted addend; By updating the gate vector z t A vector of complementary ones representing the state and / or output h at the previous time step t-1 performing weighting to generate a second weighted addend; and Add the first addend to the second addend.

51. An audio generator according to any one of claims 48-50, wherein the recurrent learnable layer is configured to generate states and / or outputs h by: t : By means of mutually complementary weight vectors, the state and / or output h of the previous time step is added t-1 A weighted version of the candidate states and / or a weighted version of the output.

52. An audio generator according to claim 50 or 51, wherein the recurrent learnable layer is configured to generate the candidate states and / or outputs by applying at least weight parameters W obtained through training to: Reset gate vector r t and the state and / or output h at the previous time step t-1 element-wise product between the input x at the current time t cascade.

53. The audio generator of any one of claims 50-52, wherein the recurrent learnable layer is configured to generate the candidate states and / or outputs by at least: The vector conditioned by the following two is weighted by the weight parameter W obtained by training: For the current input x t and Weighted to reset gate vector r t The state and / or output h at the previous time step t-1 .

54. An audio generator according to any one of claims 50-53, wherein the recurrent learnable layer is configured to generate z Apply the following cascade to generate the update gate vector z t : The loop module h t-1 The input h t-1 Cascaded to The input x for the current moment t .

55. An audio generator according to any one of claims 50-54, wherein the weight parameter W is r The reset gate vector r is obtained by applying the cascade of the following two: The state and / or output h of the previous time step t-1 as well as The input x for the current moment t .

56. The audio generator (10) according to any one of claims 46-55, comprising a quantization index converter (313) configured to convert an index of the bit stream (13) into a code.

57. The audio generator according to any one of claims 46-56, configured so that the bit rate of the audio signal (16) is greater than the bit rate of the target data (12) and / or the first data (15) and / or the second data (69).

58. An audio generator according to any one of claims 46-57, configured to obtain the input signal (14) from the bit stream (3, 3b).

59. An audio generator according to any one of claims 46-57, configured to obtain the input signal from noise (14).

60. An audio generator according to any one of claims 46-59, wherein the at least one pre-processing learnable layer (710) is configured to provide the target data (12) as a spectrogram or a decoded spectrogram.

61. An audio generator according to any one of claims 46-60, wherein the at least one learnable layer or a set of conditional learnable layers comprises one or at least two convolutional layers (71-73).

62. An audio generator according to any one of claims 46-61, wherein the first convolution layer (71-73) is configured to convolve the target data (12) or the upsampled target data using a first activation function to obtain first convolution data (71').

63. An audio generator according to any one of claims 46 to 62, wherein the first data (15, 59, 59a, 59b) has a lower intrinsic dimension than the audio signal (16).

64. An audio generator according to any one of claims 46-63, wherein the target data (12) is a spectrogram.

65. An audio generator according to any one of claims 46-64, wherein the audio signal (16) is a monophonic audio signal.

Images