Method and device for audio encoding and decoding

Abstract

Method and device for audio encoding and decoding This device includes an audio encoder (4) trained by machine learning to provide as output, for an input audio signal (SE), a compressed signal formed of symbols represented on a number N of successive quantization layers and an audio decoder (6) configured to reconstruct a reconstructed audio signal (SR) from a compressed signal formed of symbols, comprising a bitrate reduction module (14), implementing a bitrate reduction model previously trained by machine learning on a database of compressed audio signals, to predict from the first N1 quantization layers, a number N2 of subsequent quantization layers, called refinement layers, the bitrate reduction model being a recurrent neural network and a module (16) for training an enriched compressed signal,then supplied to a decoding module (18) to obtain the reconstructed audio signal. Figure for the abbreviation: Figure 1,

Description

Title of the invention: Method and device for audio encoding and decoding

[0001] The present invention relates to a method and device for audio encoding and decoding, as well as an associated computer program.

[0002] The invention is in the field of audio coding and decoding, for the realization of audio coding and decoding devices, also called audio coded devices.

[0003] More particularly, the invention lies in the field of low or very high bitrate audio codecs, which can be embedded on platforms with limited computing and energy resources.

[0004] Many practical applications require audio communication, for example of commands or information spoken by an operator to a remote operator, in real time and of sufficient quality for understanding.

[0005] As is known, it is necessary to compress audio signals, particularly speech signals, to enable their transmission over a data rate-constrained communication network. Audio encoding and decoding devices have been developed for this purpose. Such devices are used in many communication systems, for example, radio communication systems or videoconferencing systems.

[0006] One problem to be solved is to obtain the most faithful possible reproduction of the audio signal before compression, with the lowest possible transmission rate, or in other words, the best possible compression ratio.

[0007] To this end, numerous audio encoding and decoding systems have been developed. Recently, artificial intelligence-based systems have been proposed.

[0008] In addition to the constraints of throughput and quality of reproduction, it is also important to consider the limitation of the consumption of computing and energy resources of an audio encoding and decoding device, in order to allow it to be embedded on platforms with reduced computing and / or energy resources.

[0009] The aim of the invention is therefore to remedy this problem, by proposing a method and device for audio encoding and decoding which combines a high compression ratio, good reconstruction quality and low resource consumption.

[0010] To this end, the invention relates to an audio encoding and decoding method, implemented in an audio encoding and decoding device comprising an audio encoder implementing a machine learning-trained compression and quantization model to provide, as output, for an input audio signal, a compressed signal consisting of symbols represented on a number N of layers of successive quantizations, the first N quantization layers of said N quantization layers corresponding to a first compression rate, and an audio decoder configured to reconstruct an audio signal from a compressed signal formed of symbols. This process comprises the following steps implemented by a computing processor: - application of a bitrate reduction model, previously trained by machine learning on a database of compressed audio signals, to predict, from the first N1 quantization layers, an N2 number of subsequent quantization layers, called refinement layers, the bitrate reduction model being a recurrent neural network, said bitrate reduction model being implemented on a received compressed signal, - formation of a compressed signal enriched by adding said refinement layers, the compressed enriched signal being provided to an audio decoding module to obtain the reconstructed audio signal.

[0011] Advantageously, the use of a bitrate reduction model makes it possible to predict, during decoding, the refinement layers to obtain an enriched compressed signal and consequently a reconstructed audio signal of higher quality. In other words, starting from a low-bitrate compressed signal, thanks to the use of the bitrate reduction model, an enriched compressed signal corresponding to high-bitrate compression is obtained. Furthermore, advantageously, the use of a recurrent neural network allows for implementation with low consumption of computing and energy resources, which is particularly useful for the use of such a process in embedded communication systems.

[0012] The audio encoding and decoding method according to the invention may also have one or more of the following characteristics, taken independently or according to all technically feasible combinations.

[0013] The application of a rate reduction model involves applying a transformation of each symbol of the received compressed signal into a vector of predetermined dimension, said vectors forming a time sequence of vectors which are provided as input to said recurrent neural network.

[0014] The recurrent neural network is formed of a succession of recurrence layers, each recurrence layer comprising a plurality of recurrence blocks.

[0015] The number of recurrence layers is less than or equal to 10.

[0016] An internal dimension and an output dimension of said recurrent neural network are reduced by applying a division parameter.

[0017] The division parameter has a chosen value, preferably between 5 and 30.

[0018] According to another aspect, the invention relates to an audio encoding and decoding device comprising an audio encoder implementing a machine learning-trained compression and quantization model to provide, as output, for an input audio signal, a compressed signal formed of symbols represented on N successive quantization layers, the first NI quantization layers of said N quantization layers corresponding to a first compression rate, and an audio decoder configured to reconstruct an audio signal reconstructed from a compressed signal formed of symbols, comprising a processing unit configured to implement:

[0019] - a bitrate reduction module, implementing a bitrate reduction model previously trained by machine learning on a database of compressed audio signals, to predict from the first NI quantization layers, an N2 number of subsequent quantization layers, called refinement layers, the bitrate reduction model being a recurrent neural network, said bitrate reduction model being implemented on a received compressed signal,

[0020] -a module for forming a compressed signal enriched by adding said refinement layers, the enriched compressed signal being provided to a decoding module to obtain the reconstructed audio signal.

[0021] According to another aspect, the invention relates to an information recording medium, on which software instructions are stored for the execution of an audio encoding and decoding process as briefly described above, when these instructions are executed by a programmable electronic device.

[0022] According to a particular aspect, the rate reduction module is configured to apply a transformation of each symbol of the received compressed signal into a vector of predetermined dimension, said vectors forming a time sequence of vectors which are provided as input to said recurrent neural network.

[0023] According to a particular aspect, the recurrent neural network is formed of a succession of recurrence layers, each recurrence layer comprising a plurality of recurrence blocks, the number of recurrence layers being less than or equal to 10.

[0024] According to another aspect, the invention relates to a computer program comprising software instructions which, when implemented by a programmable electronic device, implement an audio encoding and decoding process as briefly described above.

[0025] The invention will become clearer upon reading the following description, given solely by way of non-limiting example, and made with reference to the drawings in which:

[0026] [Fig.1] [Fig.1] is a schematic representation of an audio encoding / decoding system according to one embodiment;

[0027] [Fig.2] [Fig.2] is a synoptic diagram of the main functional blocks of a module of flow reduction according to an embodiment;

[0028] [Fig.3] [Fig.3] is a flowchart of an audio encoding / decoding process according to one embodiment;

[0029] [Fig.4] [Fig.4] is a functional diagram of an encoding / decoding device audio according to a particular embodiment.

[0030] The invention applies in the context of encoding / decoding audio signals with low and very low bitrate compression, and finds in particular applications in audio communication devices embedded on any type of platform, for example portable communication devices.

[0031] Low bitrate audio compression refers to compression between 1 and 5 kbps (kilobits per second). Very low bitrate audio compression corresponds to a bitrate below 1 kbps.

[0032] The audio signals processed are typically speech signals, spoken by an operator.

[0033] With reference to [Fig.1], an audio encoding and decoding system 2 comprises an encoder and transmitter 4, hereafter referred to as encoder 4, and a receiver and decoder 6, hereafter referred to as decoder 6.

[0034] An audio signal SE is provided as input to a compression block 8 of the encoder 4, which implements a compression model trained by machine learning on a database of audio signals. For example, the compression model is a neural network.

[0035] In a known manner, a neural network comprises an ordered succession of layers of neurons, each of which takes its inputs from the outputs of the previous layer.

[0036] More precisely, each layer comprises neurons taking their inputs from the outputs of the neurons of the previous layer, or from the input variables for the first layer.

[0037] Alternatively, more complex neural network structures can be envisaged with a layer that can be linked to a layer further away than the immediately preceding layer.

[0038] Each neuron is also associated with an operation, that is to say a type of processing, to be carried out by said neuron within the corresponding processing layer.

[0039] Each layer is connected to the other layers by a plurality of synapses. A synaptic weight is associated with each synapse, and each synapse forms a link between two neurons. It is often a real number, which takes both positive and negative values. In some cases, the synaptic weight is a complex number.

[0040] Each neuron is designed to perform a weighted sum of the value(s) received from the neurons of the preceding layer, each value being multiplied by the respective synaptic weight of each synapse, or connection, between said neuron and the neurons of the preceding layer, and then to apply an activation function, typically a non-linear function, to said weighted sum, and to deliver at the output of said neuron, in particular to the neurons of the next layer connected to it, the value resulting from the application of the activation function. The activation function introduces non-linearity into the processing performed by each neuron. The sigmoid function, the hyperbolic tangent function, and the Heaviside function are examples of activation functions.

[0041] As an optional complement, each neuron is also capable of applying, in addition, a multiplicative factor, and an additive bias, to the output of the activation function, and the value delivered at the output of said neuron is then the product of the value of the multiplicative factor and the value from the activation function, plus the bias.

[0042] A convolutional neural network is also sometimes called a convolutional neural network or by the acronym CNN, which refers to the English term "Convolutional Neural Networks".

[0043] In a convolutional neural network, each neuron in the same layer exhibits exactly the same connection pattern as its neighboring neurons, but at different input positions. The connection pattern is called the convolutional kernel or, more commonly, the "kernel" in reference to the corresponding English term.

[0044] A fully connected layer of neurons is a layer in which the neurons of said layer are each connected to all the neurons of the preceding layer.

[0045] Such a type of layer is more often referred to by the English term "fully connected", and sometimes designated by the name "dense layer".

[0046] The values ​​of the weights, multipliers and biases where applicable are learned during a machine learning phase to perform the classification or prediction task.

[0047] A quantization block 10 is applied at the output of the compression block 8.

[0048] For example, the quantization block 10 is implemented by learning a A dictionary represents the space of compressed audio signals. The quantization block then associates an input vector with the index of the dictionary word closest to that vector. This quantization method is called "vector quantization".

[0049] At the output of the quantization block 10, for the input audio signal SE, a corresponding compressed and quantized signal Sc is obtained, comprising a series of symbols (or "tokens" in English) s-cl,...,s-ct, each symbol having a quantized value on a given number N of quantization levels or depth levels, represented on a depth axis.

[0050] By way of non-limiting example, in [Fig.1], N=8.

[0051] Denoting M as the number of possible values ​​per quantization level, the number of possible values ​​of the symbols is then equal to MN ■

[0052] According to the depth dimension, the symbols forming the compressed signal are represented in N successive quantization layers, each layer corresponding to a level of quantization, starting from a first level of quantization (or depth) 1, corresponding to the coarsest quantization, up to the level of quantization (or depth) N corresponding to the finest quantization.

[0053] Thus, the first NI quantization layers, for example Nl=3, correspond to a first compression rate, which is a low rate, while the N quantization layers correspond to a second compression rate which is higher than the first compression rate.

[0054] In other words, the first compression rate corresponds to a first compression ratio and the second compression rate corresponds to a second compression ratio, the first compression ratio being higher than the second compression ratio.

[0055] In order to facilitate transmission over a communication network 5, preferably the first compression ratio is applied.

[0056] A portion SCi of the compressed signal, corresponding to the first NI quantization layers, for example NI between N / 4 and 3*N / 4, is transmitted via the communication network 5 to the receiver and decoder 6. The Sci portion is a compressed signal corresponding to a first rate compression, low or very low rate, of the audio signal SE.

[0057] The decoder 6 includes a rate reduction module 14, which implements a rate reduction model 15, previously trained on a database of compressed audio signals, the compressed audio signals in this database being able to be different from the compressed audio signals corresponding to the audio signals in the database used for training the encoder, the rate reduction model being trained by machine learning to predict from the first NI quantization layers, the next N2 quantization layers, called refinement layers, respecting the condition N1+N2 less than or equal to N (i.e. N2 less than or equal to N-N1).

[0058] For example, N2=N-N1.

[0059] In the example illustrated in [Fig.1], N2=5.

[0060] Thus, advantageously, the flow reduction module 14 makes it possible to predict the refinement layers 12 which have not been transmitted for all transmitted symbols, the flow corresponding to these refinement layers then being saved.

[0061] The decoder 6 further includes a recomposition module 16, which recomposes an enriched compressed signal, with N1+N2 quantization layers, from the received compressed signal and the predicted refinement layers.

[0062] The enriched compressed signal is supplied as input to a decoding module 18 which provides a reconstructed SR audio signal.

[0063] Advantageously, the reconstructed SR audio signal is of better quality than a reconstructed audio signal by providing at the input of the decoder 6 only the SCi portion of the compressed signal corresponding to the NI first quantization layers only.

[0064] Advantageously, the rate reduction module 14 implements a recurrent neural network, for example an LSTM neural network (for "Long Short Term Memory").

[0065] Figure [Fig. 2] illustrates an embodiment of a flow reduction module 14 based on a recurrent neural network.

[0066] The rate reduction module 14 receives as input a portion of compressed signal corresponding to the first N1 quantization layers, and comprising T successive symbols, each symbol being coded on NI quantization layers.

[0067] Module 14 is configured to implement a transformation 30 of each s-cj symbol into a vector Xj of predetermined dimension Dim. Such a transformation is known as "embedding" in English.

[0068] The predetermined dimension Dim is typically equal to 64, 128 or 256, preferably equal to 64 in order to limit the size of the vectors and consequently, to limit the number of calculations.

[0069] A time sequence of vectors Xb.. XT represented in a discrete space of dimension Dim, also called latent space, is obtained.

[0070] The rate reduction module 14 is configured to apply a recurrent LSTM model in step 32. The vectors Xi....XT are then provided, in ascending time order, to the recurrent neural network.

[0071] Step 32 of applying an LSTM recurrence pattern implements recurrence blocks 34i to 34L on L recurrence levels or layers. Each recurrence block applies predefined operations.

[0072] For a given recurrence layer, each recurrence block 34j takes as input the results of corresponding calculations applied to temporally preceding elements. For a subsequent recurrence layer, the outputs of the preceding recurrence layer are used as input.

[0073] In the example, for a recurrence layer of index 1, a recurrence block takes as input elements c / {which are results of the previous temporal recurrence block (temporal index t-1) of the same recurrence layer and an output element of the previous recurrence layer of index 1-1.

[0074] For the first recurrence layer, the elements Xi...XT are provided as input.

[0075] Advantageously, the recurrent neural network model used is sized to have a limited number of parameters and a limited number of operations to be performed, so as to consume less memory and computing resources.

[0076] Preferably, in order to limit the number of computing resources used, the number L of recurrence layers is limited, for example L is less than or equal to 10.

[0077] Preferably, the internal dimension (dimension of intermediate results, vectors C and H) and the output dimension of the LSTM recurrent neural network is reduced to M / div, where "div" is a division parameter to be adjusted to meet computational complexity constraints.

[0078] For example, the division parameter “div” has a value chosen so as to limit the number of parameters implemented in the recurrent neural network to a predetermined maximum number.

[0079] For example, the division parameter “div” has a value between 5 and 30.

[0080] The throughput reduction module 14 is configured to then apply a projection step 36 (commonly called a dense layer) to transform the output latent space of the recurrent neural network into non-normalized probability predictions of the corresponding refinement layers. The number of predictions is equal to (N²M)T.

[0081] At the output, the flow reduction module provides the predicted N2 refinement layers (reference 12), from the probabilities obtained in step 36.

[0082] Advantageously, the flow reduction module 14 described performs a prediction of all the refinement layers simultaneously, which makes it possible to improve the time complexity.

[0083] The main steps of an audio encoding and decoding process in an embodiment are described with reference to [Fig.3].

[0084] On the encoder side, the process includes, on an input audio signal SE, the implementation 40 of a compression and quantization model 42.

[0085] Preferably the compression and quantization model is a machine learning model trained on a database of audio signals, preferably of good quality audio signals (intelligible, noise-free...

[0086] For example, in one embodiment the compression and quantization model is formed of a neural network and a vector quantization layer as described above.

[0087] The process then includes a transmission step 44 of a compressed signal formed from the first NI quantization layers, a crude representation of the input audio signal corresponding to a low or very low bitrate coding.

[0088] On the decoder side, the compressed signal is received at reception stage 46.

[0089] Next, the previously trained rate reduction model is implemented in step 48 on the received compressed signal to predict N2 untransmitted quantization layers. For example, a rate reduction module such as the one described above with reference to [Fig. 2] is implemented.

[0090] The process then includes a step 50 of forming an enriched compressed signal, with N1+N2 quantization layers.

[0091] This enriched compressed signal is provided as input to a decoder 52, which allows the reconstruction of an audio signal called the reconstructed audio signal, SR, from a signal compressed by the compression and quantization model 42.

[0092] In one embodiment, the audio encoding and decoding process is implemented by an audio encoding and decoding device, also called audio encoding.

[0093] Fig. 4 schematically illustrates an audio coded 60 configured to implement an audio encoding and decoding process, according to one embodiment.

[0094] The audio coded 60 is a programmable electronic device.

[0095] The device 60 includes at least one computing processor 62, at least one electronic memory unit 64, a communication interface 66 and optionally, a human-machine interface 68, adapted to communicate via a communication bus 65.

[0096] The electronic memory 64 is configured to store the compression and quantization model 42, previously trained for this task, as well as the rate reduction model 15, previously trained for this task.

[0097] The computing processor 62 is configured to run an encoder module 70, which applies the compression and quantization model to an input audio signal, to provide a compressed signal, applied during the encoding of an audio signal.

[0098] The computing processor 62 is also configured to execute the following modules when decoding an audio signal:

[0099] - a rate reduction module 72, configured to apply the rate reduction model previously trained on a database of audio signals compressed, to predict, from the first NI quantization layers, an N2 number of subsequent quantization layers,

[0100] - a module 74 for forming an enriched compressed signal,

[0101] - a decoding module 76, which implements a decoding adapted to reconstruct an audio signal from a signal compressed by the audio encoder 70.

[0102] In one embodiment, modules 70, 72, 74, 76 are implemented in the form of software instructions forming a computer program, which, when executed by a programmable electronic device, implements an automatic unlearning analysis method for at least one class by means of a data classification model according to the invention.

[0103] In an alternative not shown, modules 70, 72, 74, 76 are each implemented as programmable logic components, such as FPGAs (Field Programmable Gate Arrays), microprocessors, GPGPUs (General-Purpose Processing Units), or dedicated integrated circuits, such as ASICs (Application-Specific Integrated Circuits).

[0104] The computer program comprising software instructions is further capable of being stored on a non-transient, computer-readable information storage medium. This computer-readable medium is, for example, a medium capable of storing electronic instructions and being connected to a bus of a computer system. By way of example, this medium is an optical disc, a magneto-optical disc, a ROM, a RAM, any type of non-volatile memory (e.g., EPROM, EEPROM, FLASH, NVRAM), a magnetic card, or an optical card.

[0105] Advantageously, the described process makes it possible to achieve a gain in throughput by going from 2400 bits / second to 600 bits / second while maintaining sufficient audio quality and intelligibility.

[0106] Advantageously, the audio encoding and decoding process described can be implemented in a low power consumption encoded system, typically less than 50mW, through the implementation of neural network models with a limited number of parameters, typically less than 2 million.

Claims

Demands

1. An audio encoding and decoding method, implemented in an audio encoding and decoding device comprising an audio encoder (4) implementing a machine-learning-trained compression and quantization model to provide, as output, for an input audio signal, a compressed signal formed of symbols represented on a number N of successive quantization layers, the first NI quantization layers of said N quantization layers corresponding to a first compression rate, and an audio decoder (6) configured to reconstruct an audio signal reconstructed from a compressed signal formed of symbols, the method being characterized in that it comprises the following steps implemented by a computing processor: - application (48) of a bitrate reduction model (15), previously trained by machine learning on a database of compressed audio signals,to predict, from the first NI quantization layers, an N2 number of subsequent quantization layers, called refinement layers, the rate reduction model (15) being a recurrent neural network, said rate reduction model (15) being implemented on a received compressed signal (SCi), - formation (50) of an enriched compressed signal by adding said refinement layers, the enriched compressed signal being provided to an audio decoding module to obtain the reconstructed audio signal.

2. A method according to claim 1, wherein the application of a rate reduction model (15) comprises an application of a transformation (30) of each symbol of the received compressed signal into a vector of predetermined dimension, said vectors forming a time sequence of vectors which are supplied as input to said recurrent neural network.

3. A method according to claim 1 or 2, wherein said recurrent neural network is formed of a succession of recurrence layers, each recurrence layer comprising a plurality of recurrence blocks.

4. A method according to claim 3, wherein the number of recurrence layers is less than or equal to 10.

5. A method according to any one of claims 3 or 4, wherein an internal dimension and an output dimension of said recurrent neural network are reduced by applying a division parameter.

6. A method according to claim 4, wherein said division parameter has a chosen value, preferably between 5 and 30.

7. A computer program comprising software instructions which, when executed by an audio encoding and decoding device, implement an audio encoding and decoding method in accordance with claims 1 to 6.

8. Audio encoding and decoding device, comprising an audio encoder (4) implementing a machine-learning-trained compression and quantization model to provide, as output, for an input audio signal, a compressed signal formed of symbols represented on a number N of successive quantization layers, the first NI quantization layers of said N quantization layers corresponding to a first compression rate, and an audio decoder (6) configured to reconstruct an audio signal reconstructed from a compressed signal formed of symbols, characterized in that it comprises a computing processor (62) configured to implement: - a bitrate reduction module (72), implementing a bitrate reduction model (15) previously trained by machine learning on a database of compressed audio signals, to predict from the first N1 quantization layers,a number N2 of subsequent quantization layers, called refinement layers, the bitrate reduction model being a recurrent neural network, said bitrate reduction model (15) being implemented on a received compressed signal, -a module (74) for forming a compressed signal enriched by adding said refinement layers, the enriched compressed signal being provided to a decoding module (76) to obtain the reconstructed audio signal.

9. Device according to claim 8, wherein said rate reduction module (72) is configured to apply a transformation (30) to each symbol of the compressed signal received in 13 a vector of predetermined dimension, said vectors forming a temporal sequence of vectors which are provided as input to said recurrent neural network.

10. Device according to claim 8 or 9, wherein said recurrent neural network is formed of a succession of recurrence layers, each recurrence layer comprising a plurality of recurrence blocks, the number of recurrence layers being less than or equal to 10.

Images