Optimized quantization of potential space in neural audio coding

Abstract

The invention relates to a method for encoding an audio signal, the method comprising analyzing (E401) the audio signal by means of an analysis neural network to obtain a potential representation of the audio signal, and wherein, the potential representation is encoded by:-applying (E404) an orthogonal linear transformation (T) to a potential vector (z) representing the potential representation of the audio signal or to a pre-processed potential vector (z ') to obtain a transformed potential vector (y); -quantizing (E405) the obtained transformed potential vector (y). The invention also relates to a corresponding method for decoding an audio signal, a coding device and a decoding device.

Description

TECHNICAL FIELD

[0001] The present invention relates to the general field of encoding and decoding of audio signals. In particular, the present invention relates to optimized quantization of a latent space (or of a neural network transformed signal) for encoding and decoding of audio signals. BACKGROUND

[0002] The present document focuses on lossy encoding (or lossy compression) of audio and speech signals.

[0003] In a lossy compression system, the overall goal is to find a signal representation that enables encoding at the lowest bit rate possible, given a quality, or conversely, to achieve the best quality possible, given a bit rate, taking into account constraints such as complexity, amount of storage or algorithmic delay.

[0004] Historically, two approaches have been used in this field:

[0005] • Encoding of speech signals, which mainly uses time-domain methods based on (short-term and long-term) linear prediction to model the speech signal - in addition to the parameters of the model, the unmodeled part (innovation or residual prediction) is also encoded. The auditory perception is usually taken into account by using perceptual weighting. The CELP model (CELP stands for Code Excited Linear Prediction) remains the reference paradigm for "classic" speech coding at low bit rate and high quality.

[0006] • Encoding of audio signals, which is mainly based on the use of sub-band based frequency domain or transform methods to obtain coding gains, and on the limits of psychoacoustic properties and auditory perception. Coding with critical decimation by MDCT (MDCT stands for Modified Discrete Cosine Transform) remains the reference technology for "classic" audio coding at low bit rate.

[0007] Since then, speech coding and audio coding have converged with the help of "universal" coding models that combine, in particular, time-domain (CELP) and frequency-domain (MDCT) methods.

[0008] The state of the art is very rich in terms of "classic" speech and audio coding techniques, and a large number of standardized or proprietary codecs have been developed that integrate various technological advances, have various applications and meet various constraints. The latest standards such as USAC, 3D MPEG-H Audio, EVS and Opus fully demonstrate the latest advances and currently represent the state of the art in terms of "classic" coding.

[0009] Artificial intelligence-based methods, particularly those using neural networks (such as WaveNet and VQ-VAE, which stands for Vector Quantization Variational Autoencoder), have driven the rise of neural network-based speech and audio coding methods. Deep learning has shown great potential in other areas related to audio coding, such as text-to-speech, image and video coding, and the analysis and processing of audio signals in general.

[0010] A characteristic of neural methods applied to speech and audio is that they typically operate directly in the time domain (i.e., on waveforms). However, frequency domain models also exist.

[0011] The following distinction can be made between the two neural methods for encoding speech and audio:

[0012] • Features are encoded and fed into a decoding stage that uses a generative model to decode based on synthesis: parameters (e.g., Mel spectrograms) are estimated and encoded, and a neural network-based decoder (e.g., WaveNet) reconstructs the waveform based on this encoded representation.

[0013] • End-to-end coding, inspired by the VQ-VAE method, decomposes the codec into an analysis part, a quantizer, and a synthesis part for end-to-end training. The principle is to use discrete latent coding to represent the distribution of the (high-dimensional) input.

[0014] Neural audio coding has reached a level of maturity sufficient to enable the deployment of neural codecs in dialogue applications.

[0015] Here, as a method for representing neural audio coding, codecs called "SoundStream" and "EnCodec" are mentioned, which are described in the following documents:

[0016] N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, “Soundstream: An end-to-end neural audio codec,” IEEE / ACM Transactions on Audio, Speech, and Language Processing, 2021.

[0017] A. Défossez et al., “High fidelity neural audio compression”, arXiv preprint, arXiv:2210.13438, 2022.

[0018] They perform at low bitrates (around 3 kbit / s to 6 kbit / s) on par with "classic" speech and audio coding methods (like EVS and Opus) operating at higher bitrates.

[0019] The two codecs, SoundStream and EnCodec, are very similar - here the principle of the EnCodec method is described only (for the case of mono signals, i.e. one-dimensional signals).

[0020] Figure 1 The encoding principle according to the EnCodec method (for the case of mono signals, i.e. one-dimensional signals) is shown. The design of this neural network-based codec can be divided into two steps: training and inference. In the training phase, the parameters of the model are trained on training data. The inference phase (or inference) refers to the deployment of the model and its implementation in real situations.

[0021] In the inference phase, i.e. during the use of the "trained" codec, the encoding (block 100) of a signal frame of 320 samples of the input signal x sampled at a frequency of 24 kHz is divided into two blocks:

[0022] - the analysis part (block 101), in which a neural network converts the input signal x into a latent representation z (also called "latent space"). This latent space represents the important features of the input signal in a compact form.

[0023] For the EnCodec example, the architecture of the analysis part will be described below with reference to Figure 9 and Fig. 10.

[0024] - the quantization / encoding part (block 102), in which the latent representation z is quantized / encoded, optionally with entropy coding (language model) in the case of EnCodec, thus generating a bitstream I consisting of multiplexed quantization indices.

[0025] Also in the inference phase, the decoding (block 110) of the reconstructed audio signal is divided into two blocks:

[0026] - inverse quantization / decoding of the latent space from the bitstream I (block 111);

[0027] - synthesis part with a synthesis neural network (block 112). The architecture of this part is symmetrical to the analysis part (in reverse order), with the convolutions replaced by transposed convolutions; this architecture is described below with reference to Figure 9 and Fig. 10. The synthesis generates an approximation of the signal x, denoted by: .

[0028] Note that the full implementation of EnCodec (block 100 and block 110) is given at https: / / github.com / facebookresearch / encodec.

[0029] The quantization used in EnCodec is a multi-stage vector quantization called "Residual VQ" or even RVQ (for "Residual Vector Quantization").

[0030] During the training phase, the RVQ quantization dictionary is learned and stored; the selected codewords are updated with a moving average (with a forgetting factor), while the unused inputs are replaced by candidates drawn from the current batch of the training database. During the gradient backpropagation phase of the analysis / synthesis network, the quantization is bypassed. In training, a variable number of stages (called "residual stages") is chosen to train a single model representing multiple bitrates. Moreover, during training, the codec also includes an additional metrics computation block (block 121) and - according to GAN principles (GAN stands for Generative Adversarial Network) - a complementary neural network called "discriminator" (block 120) which is used to provide various cost functions for training. The details related to training are not given here as they are outside the scope of the present invention, which focuses on improving a neural network-based codec applied to audio signals under the assumption that the analysis network and the synthesis network have already been trained (as indicated by the shaded part of Figure 1

[0031] Figure 2 The operation of the multi-stage quantization (block 102 without entropy coding) is illustrated, and Figure 3 The associated decoding (block 111 without entropy decoding) is illustrated.

[0032] Depending on the number of bits R per frame, the operation employs a cascaded series of K quantizers (blocks 201, 202,..., 20K); where the signal z of the latent space is encoded by the first stage, then the quantization error e of the previous stage is computed (blocks 251, 252, etc.) to obtain the input of the current stage. k

[0033] In EnCodec described in the above article, for an input signal of dimension D = 128, the bitrate of one stage is 10 bits. For a 24 kHz mono input, EnCodec defines N = 1024 stages of input (10 bits per frame of 320 samples at 24 kHz, i.e. 0.75 kbit / s), and can combine up to 32 stages (up to a bitrate of 24 kbit / s). The RVQ quantization involves storing K = 32 dictionaries (blocks 001 to 00K), one per stage; each dictionary C k ​​of dimension D = 128 and size N = 1024. The quantization of the k-th stage is simply performed by searching the nearest neighbor of e k to all codewords of the dictionary C k .

[0034] By default, the number of successive stages of the combination is predefined: K = 2, 4, 8, 16 or 32 stages for a bitrate of 1.5, 3, 6, 12 or 24 kbit / s (where R = 20, 40, 80, 160 or 320 bits / frame). The index i k is multiplexed to form the bitstream I of the current frame (block 200).

[0035] During decoding (as shown in Figure 3 , the number of stages K is determined according to the bitrate R. The bitstream I is demultiplexed and the indices are separated (block 300), and each stage (block 301, block 302,..., block 30K) is obtained by reading the stored codeword from the associated dictionary C k (block 001 to block 00K) and decoding it, and all the thus decoded codewords are successively added (block 352,..., block 35K) to obtain the decoded signal of the latent space in the current frame .

[0036] The RVQ method for quantizing the latent space has some advantages: the implicit structure of the latent space is implicitly learned by vector quantization (instead of being explicitly extracted), and the multi-stage method enables a (scalable) hierarchical coding with improved stages, where the number of these improved stages depends on the target bitrate.

[0037] However, this method leads to many problems. It is known to the person skilled in the art that multi-stage (unstructured) quantization is suboptimal, especially in terms of signal-to-noise ratio of the quantization; and this problem gets worse as the number of stages increases. In addition, multi-stage quantization requires a non-negligible storage: N x D storage values are needed for each (10-bit) stage, where N = 1024 is the size of the dictionary (in number of codewords) and D is the dimension of the latent space (D = 128 in EnCodec).

[0038] Finally, in terms of 10-bit bitrate, the complexity of searching the nearest neighbor (which must be performed in an exhaustive way by minimizing the squared error between the input e j and all codewords c k (i) increases linearly with the number of stages and the bitrate.

[0039] There is therefore a need to optimize the techniques used in the field of neural audio coding / decoding for encoding a latent space. SUMMARY

[0040] The present invention aims at improving the prior art.

[0041] To this end, the invention relates to a method for encoding an audio signal, the method comprising analyzing the audio signal by means of an analysis neural network to obtain a latent representation of the audio signal, and wherein the latent representation is encoded via the following steps:

[0042] - applying a linear transformation to the latent vector representing the latent representation of the audio signal or to the pre-processed latent vector to obtain a transformed latent vector;

[0043] - quantizing the obtained transformed latent vector.

[0044] Applying a transformation to the latent vector enables to obtain a transformed vector whose components are decorrelated; and for this transformed vector, the quantization does not require as complex a search for the nearest neighbor nor as much storage space as the methods of the prior art. In one embodiment where the multi-stage quantization method is replaced by a direct quantization, the method also enables to obtain a better encoding quality at high bitrates than with a vector quantization method employing several quantization stages.

[0045] In one embodiment, the pre-processed latent vector is obtained by subtracting from the latent vector a correction vector representing the mean value of the latent vector or a latent vector resulting from at least one first direct quantization.

[0046] The correction vector allows to adapt the latent vector to the transformation implemented by removing the mean value which can disturb the obtained transformed vector and make the transformation suboptimal.

[0047] In one particular embodiment, the pre-processed latent vector is further obtained after scaling the components of the pre-processed latent vector.

[0048] This scaling allows a further adaptation of the latent vector prior to the transformation step.

[0049] In one particular embodiment, the transformation further comprises reducing the components to be quantized by deleting some components of the latent vector.

[0050] In this embodiment, the order of importance of the parameters of the transformation enables to truncate some components of the latent vector to be transformed, thereby reducing the number of components to be quantized, without however degrading the quality of the quantization.

[0051] In one possible embodiment, the quantization indices corresponding to the bit budget used in the at least one first direct quantization are encoded.

[0052] The indices enable to adapt to the remaining coding bitrate.

[0053] In one embodiment, the quantization of the transformed latent vector is performed via gain-shape vector quantization.

[0054] This quantization method is simple and requires little storage space.

[0055] In another embodiment, the quantization of the transformed latent vector is performed via at least one step of vector quantization using a transformed quantization dictionary.

[0056] This embodiment of quantization enables the encoder and decoder according to the invention to be compatible with the prior art EnCodec encoder or decoder.

[0057] The invention also relates to a method for decoding an audio signal, the method comprising synthesizing a decoded latent vector via a synthesis neural network to obtain a decoded audio signal, and wherein the latent vector is decoded via the steps of:

[0058] - receiving a bitstream containing at least some of the quantization indices of the transformed latent vector;

[0059] - decoding the transformed latent vector based on the received quantization indices;

[0060] - applying a linear transformation to the decoded transformed latent vector to obtain a decoded latent vector.

[0061] In one embodiment, the decoded latent vector is an intermediate latent vector to which a correction vector is added to obtain the decoded latent vector, the correction vector representing the mean of the latent vector or a decoded latent vector resulting from at least one first direct inverse quantization.

[0062] The decoding method has the same advantages as the encoding method described above.

[0063] In order to obtain a predefined vector size, the decoding method further comprises a step of adding additional components to the decoded latent vector. This will fill the elements removed during encoding.

[0064] The invention relates to an encoding device comprising processing circuitry for implementing the steps of the encoding method as described above.

[0065] The application also relates to a decoding device comprising processing circuitry for implementing the steps of the decoding method as described above.

[0066] The application relates to a computer program comprising instructions for implementing the encoding method or the decoding method as described above when executed by a processor.

[0067] Finally, the application relates to a processor-readable storage medium storing a computer program comprising instructions for executing the encoding method or the decoding method as described above. BRIEF DESCRIPTION OF DRAWINGS

[0068] Other characteristics and advantages of the application will become more clearly apparent from reading the following description of particular embodiments, given by way of simple illustrative and non-limiting example, and from examining the annexed drawings in which:

[0069] [ Figure 1 ] illustrates one example of a neural network-based encoder and decoder with quantization of the latent space;

[0070] [ Figure 2 ] illustrates one example of encoding by multi-stage (unstructured) vector quantization through the latent space according to the prior art;

[0071] [ Figure 3 ] illustrates one example of decoding by multi-stage (unstructured) vector quantization through the latent space according to the prior art;

[0072] [ Figure 4a ] illustrates in the form of a flowchart the steps of an encoding method according to one main embodiment of the application;

[0073] [ Figure 4b ] illustrates in the form of a flowchart the steps of a decoding method according to one main embodiment of the application;

[0074] [ Figure 5 ] illustrates an encoding device and an encoding method according to a first embodiment of the application.

[0075] [ Figure 6 ] illustrates a decoding device and a decoding method according to a first embodiment of the application.

[0076] [ Figure 7 ] illustrates an encoding device and an encoding method according to a second embodiment of the application.

[0077] [ Figure 8 ] illustrates a decoding device and a decoding method according to a second embodiment of the application.

[0078] Figure 9 Figure 1 illustrates an example of an embodiment of the analysis and synthesis parts of one example of a neural audio codec used according to the application;

[0079] Figure 10a Figure 2 illustrates an example of an input / output format of the analysis part of one example of a neural audio codec used according to the application;

[0080] Figure 10b Figure 3 illustrates an example of an input / output format of the analysis part of one example of a neural audio codec used according to the application;

[0081] Figure 11 Figure 4 illustrates an example of a structural embodiment of an encoding device and a decoding device according to one embodiment of the application. DETAILED DESCRIPTION

[0082] Figure 4a Figure 5 illustrates, in the form of a flowchart, the main steps of a method for encoding an audio signal according to the application.

[0083] In a first step E401, an audio signal x is analyzed to obtain a latent representation z of this audio signal by a neural network of analysis (block 101 ) such as described with reference to Figure 1 The architecture of this neural network-based analysis step can be as described with reference to Figure 1 and is described in details in Figure 9 and in Figure 10. In variants, the example of implementation of block 101 of EnCodec can be replaced by other neural network-based architectures (e.g. the analysis part of the SoundStream codec, the analysis part of a self-encoder, etc.).

[0084] Here, in the described embodiment, the obtained latent space has been illustrated in the form of vectors and is represented by a latent vector z.

[0085] For example, in the case of a matrix representation, the latent space can also be represented by a set of latent vectors. In the subsequent parts of the description, the text will make reference to the processing of a latent vector. It will be understood that, for a matrix representation, the processing can be performed on the set of latent vectors, for example via successive processing of each vector, or by converting the matrix into a one-dimensional vector by concatenating it by rows or by columns.

[0086] This pre-processing of the latent vector can be implemented in step E403.

[0087] To this end, in E402 a correction vector is obtained, and in E403 this correction vector is subtracted from the latent vector z to obtain a pre-processed latent vector z’.​​​​

[0088] The correction vector can be fixed or adaptive. It represents the average of the latent vector (obtained from values ​​pre-stored in memory (Inf)) or the latent vector generated by at least one initial direct quantization.

[0089] The following will refer to Figure 5 and Figure 7 Describe the various ways to obtain this correction vector.

[0090] The latent vectors are corrected to fit the transformation performed in step E404. In some scenarios, this step may be omitted, and the transformation can then be applied to the unmodified latent vectors.

[0091] In step E404, a linear transformation is applied to the unprocessed latent vector z or the latent vector that was preprocessed and thus modified in step E403. Therefore, the transformed latent vector y is obtained at the end of step E404.

[0092] The following is for reference. Figure 5 and Figure 7 Describe the applications of this transformation.

[0093] In step E405, the transformed latent vector is quantized.

[0094] Therefore, multiple embodiments are possible, and references have been made to... Figure 5 and Figure 7 It has been described.

[0095] At the end of this quantization step, depending on the quantization method implemented, a bit stream I is obtained containing the quantization indices ((i1, i2, ...) or (m, b)) of the transformed potential vector.

[0096] Figure 4b The main steps of a method for decoding audio signals are shown.

[0097] Based on the quantization index ((i1, i2, ...) or (m, b)) of the transformed latent vector received in bitstream I, the transformed latent vector is subjected to... The decoding step is performed in step E411. An example of the embodiment will be referred to... Figure 6 and Figure 8 Describe it.

[0098] In step E412, a linear transformation is applied to the decoded and transformed latent vector to obtain the decoded latent vector. In cases where the latent vector is preprocessed during encoding, the decoded latent vector is then processed in E414. the intermediate vector at this time and the correction vector obtained in step E413 The modification step on the decoded latent vector is performed by combining to form a decoded latent vector .

[0099] In step E415, a neural network-based synthesis (block 112) is performed on the decoded latent vector as described with reference to Figure 1 to obtain a decoded audio signal In variants, the example of implementation of block 112 of EnCodec can be replaced by other neural network-based architectures (e.g. the synthesis part of the SoundStream codec, the synthesis part of an autoencoder, etc.).

[0100] Figure 5 A first embodiment of a device for encoding and a first embodiment of a method for encoding an audio signal are illustrated. This figure in particular illustrates a module for encoding a latent representation (or latent space) (500) contained in an audio signal encoder (block 100) such as described with reference to Figure 1 In some variants, the latent space can be obtained using methods other than the analysis neural network of EnCodec. Without loss of generality, consider that the input audio signal is divided into successive time frames (or one-dimensional vectors) of length L (giving the dimension of the input vector). The latent space is here considered as a vector of dimension D in the current frame. Thus, as an example of embodiment, consider here L = 320 samples and D = 128. In some variants, the values of L and D will likely be different and vary between frames.

[0101] Thus, Figure 1 the block for quantizing / encoding the latent space (block 102) of EnCodec is replaced by the module for encoding the latent space (block 500) described here.

[0102] Thus, block 500 receives as input the latent representation z delivered by the neural network-based analysis module (block 101) of EnCodec. Figure 1

[0103] Hereinafter, the terms "latent space" and "latent representation" will designate the same representation in the form of a set of vectors z (or, for simplicity, latent vectors z), i.e. a representation of an audio signal comprising a plurality of variables data (also called latent variables) which can be discrete or non-discrete.

[0104] The dimension of the input of this encoding module (block 500) is here denoted [D, T], where D is the dimension of the latent space in each frame and T is the number of frames.

[0105] ​Moreover, to simplify the notation, the case of encoding a given frame is considered, with T = 1 frame. The case T > 1 is related to an extension of the embodiments of the application, in which a plurality of frames is jointly encoded. In the following, the input of the block 500 will be considered as having dimension [D, 1].

[0106] It is here assumed that the latent representation takes the form of a vector, and is represented by a latent vector z of dimension [D, 1]. In case the latent space does not take the form of a vector, it can for example take the form of a matrix, which can then be provisioned to be converted into a vector form.

[0107] As will be seen hereafter, the encoding method according to the application consists in transforming the latent space so that the various components of the latent space are as much as possible decorrelated.

[0108] In a first step, a modified latent vector is obtained by subtracting in the block 512 from the latent vector z the correction vector delivered by the block 510 (using the data stored in the block 540); this involves a vector of dimension [D, 1] of the latent variable.

[0109] In a preferred embodiment, when the input x of the encoder is a signal representing silence (zero value, or random low noise, such as the background noise of a sound capture system or "idle noise"), the vector corresponds to a (stable) value of z. In this case, the block 540 directly stores this predetermined vector, and the block 510 only reads this vector.

[0110] In a variant, the vector may be pre-learned and stored as a reference Figure 2 and Figure 3 The dictionary C1 (block 001) of the codec described is the average (or centroid) of the quantized dictionary in which it is stored. This quantized dictionary C1 is stored in a memory in the block 540, or more advantageously, only the centroid of this dictionary is stored .

[0111] Thus, in one embodiment, the vector is determined in the following way:

[0112] which corresponds to the centroid of the first dictionary C1, this first dictionary corresponding to the block 001 as illustrated in Figure 2 and Figure 3 .

[0113] In a variant, a dictionary of the following stage can be included; for example, in the case of the first 2 dictionaries C1 and C2 (block 001 and block 002):

[0114]

[0115] In this case, the quantization dictionaries C1 and C2 are stored in the memory in block 540, or more advantageously, only the centroids of these 2 dictionaries .

[0116] The advantage of blocks 510 and 512 is to adapt the latent vector z before applying the transform T of block 513. To this end, one seeks to remove the component based on the mean of the latent space. In the absence of such removal, the presence of this mean can make the transform in 513 subsequent less good and can impair the transformed vector obtained.

[0117] This vector is defined here as fixed and does not require the transmission of additional information.

[0118] Then, block 513 performs a linear transform of PCA / KLT type, for example (PCA stands for Principal Component Analysis, KLT stands for Karhunen-Loeve Transform).

[0119] We now review these two techniques.

[0120] Karhunen-Loeve Transform (KLT)

[0121] The Karhunen-Loeve Transform (KLT) of a random vector z centered on 0 and with covariance matrix is defined by:

[0122]

[0123] where is the matrix of eigenvectors obtained by eigen decomposition of , the eigenvectors being by convention column vectors,

[0124]

[0125] where is the diagonal matrix whose coefficients are the eigenvalues. The matrix contains the eigenvectors (columns) of , so that

[0126]

[0127] The KLT can be seen as a basis transform, since the product expresses the vector in a basis given by the eigenvectors.

[0128] The inverse transform is given by:

[0129]

[0130] KLT can The components are decorrelated; the transformed vector The variance is eigenvalues.

[0131] Principal Component Analysis (PCA)

[0132] Principal component analysis (PCA) is a dimensionality reduction technique that produces orthogonal variables and maximizes the variance of the projected variables (or is equivalent to minimizing the reconstruction error).

[0133] Although the PCA method given below, like KLT, is based on eigenvalue decomposition, it makes the estimated covariance matrix... It is based on dimensions of observation vectors , The calculation results are as follows:

[0134]

[0135] And it is assumed that these vectors are centered:

[0136]

[0137] Will Perform eigenvalue decomposition The form allows for the calculation of principal components: .

[0138] PCA is a matrix The transformation is performed to project the data onto a new basis so that the variance of the variable after the projection is maximized.

[0139] Note that it can also be formed from a matrix of size D × K. signal The singular value decomposition (SVD) yields the PCA. In this case, it can be written as:

[0140]

[0141] Among them, matrix , and Specify the left singular vector, singular value, and right singular vector.

[0142] It can be shown , which corresponds to The diagonalization of the PCA. Therefore, the projection vector of the PCA corresponds to... The column vectors, and the projection gives As a result.

[0143] Without loss of generality, KLT and PCT are considered here to represent the same transform method and analysis method. The following text will make reference to linear transformations.

[0144] Thus, block 513 implements a step of linear transformation and truncation operation of the latent vector z' modified by applying a rotation matrix U of size D x D defined below (pre-computed and stored in a memory 533) to obtain a transformed latent vector y, such that:

[0145] y = trunc(U. z')

[0146] where trunc(.) designates a truncation operation that keeps only the first D T components. In a preferred embodiment, for example D T = 80 (compared to D = 128).

[0147] The transform step (block 513) does not require a bit budget.

[0148] In a variant, by introducing a pre-normalization by standard deviation of the D different components, the covariance matrix would possibly be replaced by a correlation matrix, giving a correlation matrix.

[0149] In a variant, by performing the matrix multiplication with the matrix only, it is possible to combine the matrix multiplication by with the truncation from D dimensions to D T dimensions.

[0150] In a first embodiment described here, the rotation matrix U is pre-computed and obtained from at least one of the quantization dictionaries RVQ (block 001).

[0151] The covariance matrix R of the (centered) first dictionary C1 is obtained for example as follows:

[0152]

[0153] In some variants, it is also possible to use the following stage, and for example desirable:

[0154] In other variants, it is possible to use an audio database defining a set of input frames of a signal x, and to record a sequence of values of the signal z' to determine R as the covariance (or correlation) matrix of this signal z'. This matrix R does not need to be stored, it only serves as intermediate information for determining the linear transformation to be applied in the reasoning steps according to the application.

[0155] ​By applying principal component analysis (PCA) or an equivalent Kalman-Louis transform (KLT) to R, eigenvalues ​​are obtained from the covariance matrix R by estimating the covariance matrix and performing eigenvalue decomposition (EVD). And the eigenvector matrix U, such that

[0156] In one particular embodiment, the feature values ​​are sorted in descending order, wherein In practice, this property can be obtained, for example, by applying singular value decomposition (SVD) to matrix R or by sorting the obtained eigenvalues ​​and reordering matrix U.

[0157] According to the present invention, in this first embodiment, RVQ quantization is modified by utilizing decorrelation of the latent space from the z-transformation to y. Therefore, it is possible to use the transformed dictionary C'1, C'2, ..., C' K Replace the predefined dictionaries C1, C2, ..., C at each stage. K Truncation of the latent space can be performed on these transformed dictionaries. Specifically, since the transformation U is performed in order of importance of the eigenvalues ​​of the transformation matrix, the resulting components are also ordered in order of importance. This allows some components to be truncated and only the remaining parts to be represented without significantly degrading the quality of the quantization.

[0158] In one example of the embodiment, dimension D = 128 is changed to a truncated dimension D. T = 80.

[0159] Therefore, according to the present invention, the new quantization dictionary is defined as follows:

[0160]

[0161] The trunc(.) operator specifies a truncation operation, which retains only the first D elements. T Each component.

[0162] Therefore, through modules 501, 502, ..., 50k and modules 551, 552, ... (and reference) Figure 2 The quantization modules 201, 202, ..., 20k and 251, 252... described therein use transformed quantization dictionaries C'1, C'2, ..., C' in memory stored in blocks 1001, 1002, ..., 100K. K The transformed latent space y is quantized through staged RVQ quantization.

[0163] In block 530, for each binary data (i1, i2, ..., i...) obtained at the end of each RVQ quantization stage as described above, RVQ quantization is performed. k) are performed. Note that the present application does not change the order of the codewords, so the indices i1, i2,..., i k Can be decoded by the existing RVQ decoder (e.g. EnCodec) in 111.

[0164] The first embodiment of the present application has the advantage that the complexity of the search in the continuous dictionary and in the storage can be reduced by a factor of approximately D T / D (i.e. 80 / 128 in the example given here). However, the performance of the RVQ quantization remains the same as the performance of the state of the art.

[0165] In a variant of the quantization of the latent space using RVQ quantization and a dictionary other than the dictionary of EnCodec (e.g. SoundStream and other dictionaries with D = 256), the dictionary C' k will be defined according to a predefined dictionary C k .

[0166] In one variant, it is also possible to define the quantization dictionary C' k directly by a direct training on the signal sequence y without going through the preliminary step of defining an existing dictionary C k . Indeed, this variant is equivalent to the application of the second embodiment of the present application detailed below, where the RVQ quantization is performed on the transformed latent space y.

[0167] Figure 6 A first embodiment of a decoding device according to the present application and a first embodiment of a decoding method according to the present application are illustrated.

[0168] This figure illustrates in particular the module (600) for decoding the latent representation (or latent space) contained in the audio signal decoder (block 110) as described with reference to Figure 1 Thus, the block (block 111) for decoding the latent space of the Figure 1 is replaced by the module (block 600) for decoding the latent space described here.

[0169] The bitstream I received for the current frame is demultiplexed (block 630). In this first embodiment, the indices actually correspond to the RVQ quantization indices, however the original RVQ quantization dictionaries Ci, C2,..., C K are replaced by dictionaries C'i, C'2,..., C' T of smaller dimension D K , as illustrated in the encoder in Figure 5 . Typically, D T = 80 instead of D = 128 in the original RVQ quantization.

[0170] Thus, at the end of each inverse quantization stage 1 to K comprising blocks 601 to 60K and blocks 652 to 65K (corresponding to blocks 301 to 30K and blocks 352 to 35K of Figure 3 the encoder). .

[0171] Next, in block 613, a linear transform, e.g. an inverse KLT / PCA transform, is applied to the decoded transformed latent representation.

[0172] The rotation matrix U stored in 533 and obtained by training allows to apply the inverse transform according to the following equation:

[0173]

[0174] denotes the decoded latent vector in the middle, and ext(.) denotes the operation of extending the dimension D T to dimension D by adding (D-D T ) zeros at the end of the vector.

[0175] In block 610, the information stored in 510 is used to obtain the correction vector in the same way as for the encoder in block 510. Figure 5

[0176] Next, the vectors and are combined in 612 to form the decoded vector of the latent space.

[0177] The latter is then input to the neural network based synthesis module (Syn.) 112 as described with reference to Figure 1 to obtain the decoded audio signal .

[0178] In one variant, it is also possible to directly define the quantization dictionary C' k by a direct training on the signal sequence y, without going through the preliminary step of defining the existing dictionary C k . Indeed, this variant is equivalent to apply the second embodiment of the application detailed below, where the transformed latent space y is RVQ quantized.

[0179] Figure 7 A second embodiment of a device for encoding according to the application and a second embodiment of a method for encoding an audio signal are illustrated. This figure in particular shows the blocks comprising the encoder as described with reference to Figure 1 ​The module (700) in the described audio signal encoder (block 100) is used to encode the latent representation (or latent space).

[0180] therefore, Figure 1 The block (block 102) used for quantizing / encoding the latent space is replaced by the module (block 700) used for encoding the latent space as described herein.

[0181] Therefore, block 700 receives data from the neural network-based analysis module ( Figure 1 The potential representation z delivered by block 101 is taken as input.

[0182] The input dimension of this encoding module (block 700) is here represented as [D,T], where D is the dimension of the latent space in each frame and T is the number of frames.

[0183] Next, we will consider Figure 1 An example of an audio encoder (corresponding to EnCodec, where D = 128) is given, but in variants this dimension can have different values. Furthermore, to simplify the notation, consider the case where a given set of frames is encoded, where T = 1 frames. The case T > 1 relates to an extension of the embodiments of the invention, in which multiple frames are jointly encoded. Hereinafter, the input to block 700 will be considered to have dimension [D, 1].

[0184] As in the first embodiment, the encoding method according to the invention is to transform the latent space in a manner that makes the components as decorrelated as possible.

[0185] In the first step, the correction vector delivered by block 701 is subtracted from the potential vector z in block 702. To obtain the modified potential vector u.

[0186] vector It can be any of the following:

[0187] a) As in the first embodiment (block 510), the predetermined average of the latent variables in the latent space (therefore, a vector of dimension [D,1]). Preferably, the encoder input x is zero (silence) or idle noise, and the value of z is collected after a certain number of frames; in the EnCodec example including an LSTM layer (LSTM stands for Long Short-Term Memory), it is important to wait for the encoder-decoder to stabilize its analysis due to the recursive nature of the LTSM layer. In variants, the centroids of one or more RVQ dictionaries can be used.

[0188] b) In terms of K = 1 encoding, in the first RVQ stage (e.g. Figure 2 The decoded latent vector at the end of block 201 shown in the figure ,or

[0189] c) More generally, for example , at the end of the RVQ stages (e.g. Figure 2 decoded latent vectors at the end of the blocks 201 and 202 illustrated in

[0190] Thus, in the embodiment corresponding to the above-mentioned embodiment a), the module 701 reads from the memory (block 710) the defined mean vector. In a variant, the block 710 determines the centroid of the dictionary C1 in the following way:

[0191]

[0192] This centroid corresponds to the centroid of the first quantization dictionary (C1) of the RVQ blocks 201 and 301 as illustrated in Figure 2 and Figure 3 This quantization dictionary C1 is stored in the memory in the block 710, or more advantageously, the block 710 can directly store the predetermined values of As in the first embodiment, it is possible to generalize the definition of the vector to the case where several RVQ dictionaries are used to determine their centroid.

[0193] In the embodiments corresponding to the above-mentioned embodiments b) and c), the block 701 corresponds respectively to one or more vector quantization stages (blocks 201, 202, etc.) described for the EnCodec method and illustrated in Figure 2 In this case, the block 710 stores one or more corresponding RVQ dictionaries.

[0194] These quantization blocks are not illustrated here in Figure 7 but they are implemented in the block 701.

[0195] The advantage of the blocks 701 and 702 is to adapt the latent vector z before the application of the transform T of the block 704. To do this, one seeks to remove the component based on the mean value of the latent space. In case a), this component is approximately fixed and does not require the transmission of additional information such as an index (m). The bit budget used by the block 701 is then R M = 0 bits.

[0196] In cases b) and c), this component is adaptively removed and a quantization index m of 10 bits (case b) or 20 bits (example of case c) is delivered as output by the block 701 to the block 710 for multiplexing. The bit budget used by the block 701 is then R M = 10 or R M = 20, which corresponds to the bit budget used in the quantization stage(s) Q1 or Q1 / Q2.

[0197] Block 703 optionally scales each of the individual components of the signal u, which represents the result of subtraction performed in 702. The resulting modified latent vector. Preferably, each component of u is normalized by its standard deviation; in variants, other normalizations are also possible. Standard deviation ( (or, as a variant, another normalized value) is pre-computed and stored in memory in 711. Thus, the modified and scaled (i.e., preprocessed) latent vector z' is obtained. In the variant, values ​​of u other than the standard deviation of each component may be used.

[0198] Next, block 704 performs a linear transformation of, for example, PCA / KLT, and then transforms the preprocessed latent vector z' by applying a rotation matrix U of size D × D (defined below) to obtain the transformed latent vector y, such that:

[0199] y = trunc(U T .z')

[0200] The trunc(.) operator specifies a truncation operation, which retains only the first D elements. T Each component. In a preferred embodiment, for example, D T = 80 (compared to D = 128).

[0201] Typically, block 704 does not require a bit budget.

[0202] Matrix U is stored in block 712.

[0203] In the embodiments described herein, the rotation matrix U can be determined as in the first embodiment.

[0204] In one embodiment, the encoding in block 705 pairs dimension D T (For example, fixed as D) T The transformed latent vector y is quantized (= 80).

[0205] In one example of the embodiment, gain-shape algebraic vector quantization (AVQ) based on the method described in the following article may be used:

[0206] S. Ragot, B. Bessette, and R. Lefebvre, “Low-complexity Multi-rate Lattice Vector Quantization with Application to Wideband TCX Speech Coding at 32 kbit / s,” Proceedings of the ICASSP Conference, Montreal, Canada, May 2004.

[0207] In this case, the transformed signal y (or the transformed latent vector) is partitioned into N dimensions of 8. v A continuous subvector. Truncate to dimension D T In the example with = 80, there are N v = 10 sub-vectors to be encoded.

[0208] As described in the article by Ragot et al. above, the signal y is scaled by the reciprocal (1 / g) of the global gain (denoted as g), which can be obtained via a binary search method while adhering to the budget constraint B ≤ (RR). M While maximizing the number of bits B used, the value can be predetermined or estimated in advance.

[0209] Therefore, the transformed latent signal, scaled by a factor of 1 / g, is (1 / g). y is encoded. Thus, a series of quantization indices with a variable number of bits (multiples of 4) and unary codes representing the quantizer indices are obtained. An example of a complete implementation of this AVQ quantization is given in the standard 3GPP AMR-WB+ (see 3GPP TS 26.273 for the fixed-point version, or TS 26.304 for the floating-point version); therefore, the manner in which this embodiment of implementing block 705 is performed will not be described further here.

[0210] The binary data generated by quantization is fully multiplexed in block 705 to form bit stream b.

[0211] Block 720 reuses the various binary information items (m, b) defined above.

[0212] In the variant, entropy encoding of indices m and b can also be implemented.

[0213] In some variations, other methods for quantizing the transformed latent space y can be implemented in block 705. For example, a pyramid vector quantization (PVQ) might be used for each subvector, where the binary allocation is fixed or adaptive; examples of PVQ implementations are given, for example, in the Opus or EVS codecs. The PVQ implementation is similar to what has been described above with respect to AVQ quantization. The signal y is typically divided into subvectors and normalized by a global gain. Each subvector can be quantized by a PVQ quantizer using a fixed (predetermined) or variable (as in the case of AVQ) binary allocation.

[0214] In another embodiment, the signal y is determined as described above, but by RVQ using a truncated dimension of D based on the principle of encoding y in block 500. T The dictionary C' k Quantization is then performed. In this case, once blocks 701, 702, 703, 710, 711, and 712 are determined, the dictionary can be determined through training based on the principles of RVQ quantization. In this case, bitstream b consists of sub-indices associated with each RVQ dictionary.

[0215] In some variants, uniform scalar quantization followed by entropy coding may also be used as block 705.

[0216] An example of the implementation is given, for instance, in the following article: “End-to-end Optimized Image Compression”, Johannes Ballé, Valero Laparra, and Eero P. Simoncelli, ICLR Conference Proceedings, 2017. Instead of quantizing the latent space of the autoencoder (in this case, the image signal), the transformed latent space y is encoded here. Uniform scalar quantization with a predetermined increment is applied to y, and then the integer quantization index is encoded by arithmetic coding.

[0217] Here, one advantage of the present invention is that the arithmetic encoding can utilize the decomposed eigenvalues ​​associated with the determination of the transformation (block 704). Only keep the first D. T There are several values, because y is defined here after truncation. Therefore, by assuming y (where the variance of each component is respectively given by...) Given a distribution (e.g., Gaussian or Laplace distribution), a probabilistic model for entropy coding can be easily derived.

[0218] Figure 8 A second embodiment of the decoding device according to the present invention and a second embodiment of the decoding method according to the present invention are shown.

[0219] The figure specifically illustrates what is included, for example, in the reference. Figure 1 The described audio signal decoder (block 110) includes a module (800) for decoding the latent representation (or latent space). Therefore, Figure 1 The block (block 111) used for decoding the latent space is replaced by the module (block 800) used for decoding the latent space as described herein.

[0220] Demultiplex the bit stream I received for the current frame (block 801).

[0221] In block 802, the correction vector is obtained in the same manner as the encoder in block 701. The correction vector delivered by block 802 can be:

[0222] a) The predetermined average of the latent variables in the latent space;

[0223] b) Regarding decoding with K = 1 Figure 3 The decoded latent vector generated from the first inverse RVQ vector quantization stage in decoding block 301, or

[0224] c) More generally, from a predetermined number of inverse RVQ vector quantization stages (e.g., by...) greater than 1 (e.g., K = 2). Figure 3 The decoded latent vectors generated by decoding blocks 301 and 302.

[0225] In cases b) and c), block 802 then corresponds to the method described for EnCodec and in Figure 3 One or more decoding stages D1, D2, ..., (block 301, block 302, etc.) are shown in the diagram. These inverse quantization blocks are shown here... Figure 8 Not shown in the image.

[0226] These decoding stages use R, which is transmitted in the bitstream and decoded in 802. M The bit budget on the bit, indicated by the quantization index m.

[0227] Figure 8 Block 803 pairs of transformed latent vectors from bitstream b Perform decoding. This decoding step will not be described in more detail—it corresponds to the decoding implemented in block 705, with examples of variations as follows: AVQ, PVQ, RVQ, or scalar quantization with entropy coding.

[0228] To replace the truncated elements and make the obtained frame of transformed latent space complete, zeros can be added in the transformation step T of block 805. In one variant of the embodiment, it would be possible to provide for injecting noise via block 804 (e.g. noise surrogates known in the art).

[0229] Next, in block 805, a linear transformation is applied to the decoded transformed latent vector, e.g. the inverse KLT / PCA transformation.

[0230] The rotation matrix U stored in 712 and obtained by training allows to apply the inverse transformation according to the following equation:

[0231]

[0232] is the intermediate decoded latent vector, and ext(.) denotes the operation of extending the dimension D T ) to dimension D by adding (D-D T ) zeros at the end of the vector. This vector is then scaled component-wise in the same way as in block 703 of Figure 7 to obtain the vector .

[0233] Next, the vectors and are combined in 807 to form the decoded vector of the latent space .

[0234] The latter is then input to the neural network based synthesis module (Syn.) 112 as described with reference to Figure 1 to obtain the decoded audio signal .

[0235] Figure 9 An example of an implementation of the elements of the neural network used in the present application, and in particular of the analysis part 101 and of the synthesis part 112, is shown. The case of EnCodec is illustrated here by way of example. The architecture used is very similar and derives from the architecture of the SoundStream codec - the main difference between EnCodec and the SoundStream architecture is that the "ResNetBlock" blocks are more complex in SoundStream and that SoundStream additionally contains a special layer called FiLM. In contrast, EnCodec contains LSTM layers which are not present in SoundStream.

[0236] The analysis part of block 101 is composed of a plurality of successive layers:

[0237] - 1D convolution in block 901 (with filter or "kernel" of length 7), with 32 channels

[0238] - 4 convolutional blocks (902-904, 905-907, 908-910, 911-913) including residual unit blocks (two convolutions and one skip connection) and convolutions with kernels of length twice the stride and with strides of 2, 4, 5 or 8

[0239] - LSTM layer split into 2 sub-layers in blocks 914-915

[0240] - final 1D convolutional layer in block 916

[0241] The activation function is the ELU activation function (ELU stands for Exponential Linear Unit), defined as: f(x) = x if x > 0; and a(exp(x) - 1) if x < 0, a being a positive value.

[0242] Figure 10a The dimensions of the input and output tensors of each block are detailed in the analysis section. The input of the analysis neural network is a column vector of dimension L = 320 (samples); this dimension is written here in the form [1, 1, 320]. Block 900 uses 32 channels, which gives an output of dimension [1, 32, 320]. Successive blocks then modify the dimensions of the tensor, the number of channels doubling each time (from 32 to 64, then to 128, 256 and 512), and the last dimension being reduced by a factor given by the stride (2, 4, 5 and 8 respectively). Blocks 914 and 915 do not change the dimensions of the tensor. The last block 916 reduces the dimensions from [1, 512, 1] to only [1, 128, 1], i.e. a latent vector of dimension D = 128.

[0243] The synthesis part of block 112 is an "inverse" architecture with the same number of blocks:

[0244] - initial 1D convolutional layer in block 921

[0245] - LSTM layer split into 2 sub-layers in blocks 922-923

[0246] - 4 transposed convolutional blocks (924-926, 927-929, 930-932, 933-935) including transposed convolutions, residual unit blocks and ELU activations

[0247] - 1D convolution in block 936

[0248] Figure 10b ​The dimensions of the input and output tensors of each block are detailed for the synthesis part. It starts with a latent vector of dimension D = 128 represented in tensor form [1, 128, 1] and ends with an audio vector of dimension L = 320 represented in tensor form [1, 1, 320].

[0249] Figure 11 An encoding device DCOD and a decoding device DDEC are shown, which in the sense of the invention are paired with each other (in the sense of "invertible") and connected to each other via a communication network RES.

[0250] The encoding device DCOD comprises a processing circuit, which typically comprises:

[0251] - a memory MEM1 for storing in the sense of the invention instruction data of a computer program (these instructions can be distributed between the encoder DCOD and the decoder DDEC) and data calculated during the training phase;

[0252] - an interface INT1 for receiving an initial audio signal x;

[0253] - a processor PROC1 for receiving this signal and processing this signal to encode this signal by executing the computer program instructions stored in the memory MEM1, this processor being in particular able to control an analysis module based on a neural network and a module for quantizing the latent representation as described with reference to Figure 5 and Figure 7 ; and

[0254] - a communication interface COM1 for transmitting the encoded signals via the network.

[0255] The decoding device DDEC comprises its own processing circuit, which typically comprises:

[0256] - a memory MEM2 for storing in the sense of the invention instruction data of a computer program (these instructions can be distributed between the encoder DCOD and the decoder DDEC, as indicated above) and data calculated during the training phase;

[0257] - an interface COM2 for receiving the encoded signals from the network RES to decompress these encoded signals in the sense of the invention;

[0258] - a processor PROC2 for processing these signals to decode these signals by executing the computer program instructions stored in the memory MEM2, this processor being in particular able to control a synthesis module based on a neural network and a module for quantizing the latent representation as described with reference to Figure 6 and Figure 8The module described for decoding the latent space; and

[0259] - an output interface INT2 for delivering the decoded audio signal .

[0260] Of course, this Figure 11 An example of a structural embodiment of a codec (encoder / decoder) in the sense of the present application is shown. As already evaluated above, Figs. 4 to 10 describe in detail the functional embodiment of these codecs.

Claims

1. A method for encoding an audio signal, the method comprising analyzing the audio signal by means of an analytical neural network (E401) to obtain a potential representation of the audio signal, wherein, The potential representation is encoded via the following steps: - Apply (E404) linear transformation (T) to the latent vector (z) representing the latent representation of the audio signal or to the preprocessed latent vector (z') to obtain the transformed latent vector (y); - Quantize the obtained transformed latent vector (y) (E405).

2. The method as described in claim 1, wherein, The preprocessed latent vector (z') is obtained by subtracting the correction vector from the latent vector (z). The correction vector obtained represents the average value of the latent vector or the latent vector generated by at least one first direct quantization.

3. The method as described in claim 2, wherein, The preprocessed latent vector is further obtained by scaling the components of the preprocessed latent vector.

4. The method according to any one of claims 1 to 3, wherein, The transformation further includes reducing the components to be quantized by removing some components of the latent vector.

5. The method of claim 2, wherein, The quantization index corresponding to the bit budget used in the at least one first direct quantization is encoded.

6. The method according to any one of claims 1 to 5, wherein, The quantization of the transformed latent vector is performed via gain-shape vector quantization.

7. The method according to any one of claims 1 to 5, wherein, The quantization of the transformed latent vector is performed using the transformed quantization dictionary via at least one step of vector quantization.

8. A method for decoding an audio signal, the method comprising synthesizing (E415) the decoded latent vectors using a synthetic neural network to obtain a decoded audio signal, wherein, The latent vector is decoded via the following steps: - Receive a bit stream containing at least some of the quantization indices of the transformed potential vector; - Based on the received quantization index, the transformed latent vector ( Decode (E411); - For the decoded transformed latent vector ( Applying the (E412) linear transform to obtain the decoded latent vector ( ).

9. The method of claim 8, wherein, The decoded latent vector is an intermediate latent vector, which is then supplemented with a correction vector. That is, the decoded latent vector can be obtained, and the correction vector represents the average value of the latent vector or the decoded latent vector generated by at least one first direct inverse quantization.

10. The method of any one of claims 8 and 9, further comprising the step of adding additional components to the decoded latent vector to obtain a predefined vector size.

11. An encoding device comprising processing circuitry for implementing the encoding steps as described in any one of claims 1 to 7.

12. A decoding device, comprising processing circuitry for implementing the decoding steps as described in any one of claims 8 to 10.

13. A processor-readable storage medium storing a computer program comprising instructions for performing an encoding method as described in any one of claims 1 to 7 and / or a decoding method as described in any one of claims 8 to 10.

Images