Encoding and decoding of audio and / or video data

Abstract

The invention relates to a method for encoding audio and / or video data, the method being performed by an encoding device configured to perform at least one step of encoding the audio and / or video data using an encoding artificial neural network, the encoding method comprising the following steps:-encoding (C2-C4) the data; -generating (C5) a data signal comprising the encoded data; -encoding (C6) information representative of a decoding configuration, the decoding device must have to have the decoding configuration in order to decode the encoded data; -inserting (C7) the encoded information into the signal.

Description

Technical Field

[0001] The present invention relates generally to the field of audio and / or video data processing and in particular to encoding and decoding digital images and digital image sequences.

[0002] The encoding / decoding of digital images is particularly applicable to images from at least one video sequence, the images comprising:

[0003] - images from the same camera and temporally adjacent to each other (2D encoding / decoding),

[0004] - images from different cameras oriented at different viewing angles (3D encoding / decoding),

[0005] - corresponding texture components and depth components (3D encoding / decoding),

[0006] - wait.

[0007] The present invention is applicable in a similar manner to encoding / decoding of 2D or 3D images.

[0008] The present invention can be particularly, but not exclusively, applicable to video encoding and corresponding decoding implemented in current AVC (Advanced Video Coding) video encoders, HEVC (High Efficiency Video Coding) video encoders and VVC (Versatile Video Coding) video encoders and their extensions (MVC (Multi-View Video Coding), 3D-AVC, MV-HEVC, 3D-HEVC, etc.). Background Art

[0009] Artificial intelligence methods, specifically neural methods, are becoming increasingly popular for the compression of still images, video or audio data, and many studies have reported surprising results on their ability to efficiently represent compressed data signals.

[0010] For example, in the context of image processing, such neural methods are now capable of addressing image compression, not just as methods designed to replace or improve certain steps in traditional compression methods (such as prediction or filtering), but rather by completely replacing the encoder and decoder, specifically using "autoencoders." Such autoencoders are described, for example, in the following document: Théo Ladune and Pierrick Philippe, "AIC Artificial Intelligence Based Videocodec," February 17, 2022. Such autoencoders include an encoding neural network that takes as input the images of a video and outputs latent variables representing a signal representing the compressed images. These latent variables are then quantized and encoded using entropy coding (e.g., Huffman coding or CABAC (Context-Adaptive Binary Arithmetic Coding)) to produce a signal representing the compressed images.

[0011] The signal is then passed to a decoder, which entropy decodes the signal's data and then dequantizes it, corresponding to the entropy encoding and quantization performed in the autoencoder. At the end of this decoding process, decoded latent variables are generated. These decoded latent variables are then fed to the decoding neural network corresponding to the encoding neural network, which provides the decoded image of the video at its output.

[0012] It is also known that in conventional video encoders (e.g., VVC encoders), there are various tasks involved in decoding an image or image sequence, each of which is associated with various constraints placed on the encoder, specifically the coding tools used, computing power, data storage, image resolution, and so on. To this end, the encoder transmits information representing these constraints to the decoder in the form of syntax elements. A VVC decoder is configured to know how to interpret such syntax elements and is therefore able to decode the signal received from the encoder. A decoder that does not conform to VVC will not know how to interpret such syntax elements and is therefore unable to encode the signal received from the decoder.

[0013] Since coding neural networks operate in a completely different way from conventional video encoders, they need to meet different constraints, and conventional syntax elements, specifically those in the VVC standard, are not suitable for these coding neural networks. For example, some VCC syntax elements are defined within a restricted range of values, while coding neural networks require broader coding instructions. Summary of the Invention

[0014] One of the purposes of the present invention is to solve the above-mentioned shortcomings of the prior art by proposing the following:

[0015] - an audio and / or video encoder based on artificial intelligence methods, the audio and / or video encoder being configured to transmit in a compressed audio and / or video signal one or more indications of decoding features that an audio and / or video decoder must support in order to decode the compressed video signal,

[0016] - an audio and / or video decoder configured to receive the compressed audio and / or video signal and to read these one or more indications of decoding characteristics in order to identify in a very simple manner whether it is capable of decoding the compressed audio and / or video signal.

[0017] The present invention thus advantageously allows an audio or video decoder (whether it is a standardized decoder (of the type AVC, HEVC, VVC, AAC, MPEG-H 3D Audio, etc.) or one based on an artificial intelligence implementation of decoding) to be compatible in reading mode with information read from a coded audio and / or video data signal, even if the audio and / or video data has been encoded using an encoder based on artificial intelligence methods.

[0018] To this end, one subject of the invention relates to a method for encoding audio and / or video data, the method being implemented by a coding device configured to implement at least one step of encoding the audio and / or video data using a coding artificial neural network, the coding method comprising the following steps:

[0019] - encoding the audio and / or video data,

[0020] - generating a data signal comprising the encoded audio and / or video data,

[0021] - encoding information representing a decoding configuration that a decoding device must have in order to decode the encoded data,

[0022] - inserting the encoded information into the data signal.

[0023] The present invention advantageously allows an audio and / or video encoder (in which at least one encoding step is implemented using an encoding artificial neural network and therefore requires one or more configuration elements dedicated to the implementation of this particular encoding step) to encode information representative of these one or more corresponding encoding configuration elements and therefore decoding configuration elements, in order to transmit this information to an audio and / or video data decoder in order to inform the decoder of the decoding capabilities it must have in order to be able to decode the audio and / or video data.

[0024] Another subject of the invention is a method for decoding coded audio and / or video data, the method being implemented by a decoding device and comprising the following steps:

[0025] - receiving coded audio and / or video data signals,

[0026] - decoding the information representing the decoding configuration according to the signal, wherein at least one step of decoding the audio and / or video data is performed using a decoding artificial neural network,

[0027] - checking whether the decoding device has the decoding configuration corresponding to the decoding information,

[0028] - Decode the signal or not, depending on the result of the check.

[0029] The invention advantageously allows an audio and / or video decoder to identify in a received coded audio and / or video data signal one or more items of information relating to a decoding configuration that the audio and / or video decoder must have in order to decode the signal.

[0030] According to a specific embodiment of the above encoding method or decoding method, the decoding configuration belongs to:

[0031] - a first category corresponding to at least one specific physical feature of hardware or software supported by the decoding device in order to be able to decode the signal, and / or

[0032] - a second category corresponding to a specific characteristic of said signal, and / or

[0033] - a third category corresponding to at least one specific processing function applied by the decoding device in order to be able to decode the signal.

[0034] If a variety of encoding / decoding configurations are used, this embodiment advantageously allows these configurations to be grouped together in categories, thereby reducing the amount of information to be encoded. According to a specific embodiment of the encoding method or decoding method described above, the information representing the decoding configuration, respectively, is associated with at least one of the first category, the second category, or the third category.

[0035] Such an embodiment advantageously allows encoding / decoding of information representing various types of decoding configurations in a structured manner. Furthermore, when multiple pieces of information representing decoding configurations are associated with various decoding parameters or features of the same category, such an embodiment allows for more compact signaling of the information, since a single syntax element or indicator is signaled for the entire category of decoding parameters or features, rather than indicating each decoding parameter or feature individually in the signal.

[0036] According to a specific embodiment of the above encoding method or decoding method, the decoding configuration belongs to a set including the following items:

[0037] - Maximum size of data storage memory;

[0038] - Minimum number of operations per second;

[0039] - Minimum latent variable rate;

[0040] - certain types of electronic circuits;

[0041] - the level of accuracy of the mathematical representation of at least one operating parameter of the decoded artificial neural network;

[0042] - activation or inactivation of at least one reference decoding step;

[0043] - at least one specific mathematical operator, or a list of specific mathematical operators;

[0044] - specific mathematical functions;

[0045] - Multiple sources of entropy decoding statistics.

[0046] According to a specific embodiment of the above-mentioned encoding method or decoding method, the information representing the decoding configuration is respectively contained in a set of predefined video parameters of the encoding method or the decoding method, or when the video data represents an image sequence, the information is contained in a set of parameters associated with the sequence.

[0047] This embodiment advantageously allows the information representing the configuration element to be encoded using the coding syntax of an existing or standardized encoder. In the case of an AVC encoder, an HEVC encoder, or a VVC encoder, for example, the predefined video parameter set of the encoding method is a VPS (Video Parameter Set), and the parameter set associated with the sequence is an SPS (Sequence Parameter Set). In another example, the information representing the decoding configuration is associated with a sub-image, specifically a tile or slice, such as defined in the HEVC standard.

[0048] According to a specific embodiment of the above encoding method or decoding method, the information indicating the decoding configuration includes:

[0049] - a first value associated with a first configuration element of said decoding configuration, and

[0050] - a second value associated with the first configuration element and the second configuration element of the decoding configuration.

[0051] When the configuration comprises a plurality of configuration elements corresponding to various decoding capabilities to be supported by the decoding device, such an embodiment advantageously makes it possible to indicate these configuration elements by being embedded in a signal transmitted to the decoder.

[0052] The features of the various embodiments or implementations described above may be added to the encoding method or decoding method defined above, alone or in combination with one another.

[0053] Another subject of the invention is an audio and / or video data signal comprising:

[0054] - audio and / or video data that have been encoded by a coding device configured to implement at least one step of encoding the audio and / or video data using a coding artificial neural network,

[0055] - Encoded information indicating a decoding configuration that a decoding device must have in order to decode the encoded audio and / or video data.

[0056] Another subject of the invention is a device for encoding audio and / or video data, the device being configured to implement at least one step of encoding the audio and / or video data using an encoding artificial neural network, the encoding device being configured to implement the following steps:

[0057] - encoding the audio and / or video data,

[0058] - generating a data signal comprising the encoded audio and / or video data,

[0059] - encoding information representing a decoding configuration that a decoding device must have in order to decode the encoded data,

[0060] - inserting said coded information into said data signal.

[0061] Such an encoding device is in particular capable of implementing the encoding method described above.

[0062] Another subject of the invention is a device for decoding coded audio and / or video data, configured to implement the following steps:

[0063] - receiving coded audio and / or video data signals,

[0064] - decoding the information representative of the decoding configuration according to the signal, wherein at least one step of decoding the audio and / or video data is performed using a decoding artificial neural network,

[0065] - checking whether the decoding device has the decoding configuration corresponding to the decoding information,

[0066] - decoding or not decoding the signal according to the result of this check.

[0067] Such a decoding device is in particular capable of implementing the above-described decoding method.

[0068] The present invention also relates to a computer program comprising instructions for implementing the encoding method or decoding method according to the present invention according to any one of the above-described specific embodiments when the computer program is executed by a processor.

[0069] Such instructions may be permanently stored in a non-transitory memory medium of an encoding device implementing the above encoding method or a decoding device implementing the above decoding method.

[0070] The program may use any programming language and may be in the form of source code, object code, or a code intermediate between source code and object code, such as in partially compiled form or in any other desirable form.

[0071] The present invention is also directed to a computer-readable recording medium or information medium including instructions of the computer program as mentioned above.

[0072] The recording medium may be any entity or device capable of storing a program. For example, the medium may include a storage device such as a ROM (e.g., a CD-ROM, a DVD-ROM, a synthetic DNA (deoxyribonucleic acid)), a microelectronic circuit ROM, or a magnetic recording device (e.g., a USB key or a hard disk).

[0073] Furthermore, the recording medium may be a transmissible medium (such as an electric signal or an optical signal) that can be conveyed via an electric or optical cable, by radio, or by other means. Specifically, the program according to the present invention may be downloaded from the Internet.

[0074] Alternatively, the storage medium may be an integrated circuit in which the program is incorporated, the circuit being adapted to execute or be used to execute the encoding method or decoding method according to the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Other characteristics and advantages will become apparent after reading the detailed description of the invention given by way of illustrative and non-limiting examples, together with the accompanying drawings, in which:

[0076] - [ Figure 1 ] shows the main steps of a method for encoding audio and / or video data in a specific embodiment of the present invention,

[0077] - [ Figure 2A ] shows a specific embodiment of the present invention according to [ Figure 1 ] encoding method, the transmission mode of the encoded data,

[0078] - [ Figure 2B ] shows another specific embodiment of the present invention according to [ Figure 1 ] encoding method, the transmission mode of the encoded data,

[0079] - [ Figure 3 ] shows the implementation in a specific embodiment of the present invention Figure 1 an encoding device for encoding a method,

[0080] - [ Figure 4 ] shows the main steps of a method for decoding audio and / or video data in a specific embodiment of the present invention,

[0081] - [ Figure 5 ] shows the implementation in a specific embodiment of the present invention Figure 4 A decoding device for a decoding method. DETAILED DESCRIPTION

[0082] Encoding of audio and / or video data

[0083] The following describes a method for encoding audio and / or video data representing a 2D or 3D image or image sequence. This encoding method can be implemented in any type of video encoder or decoder, such as those compliant with the JPEG, AVC, HEVC, VVC standards and their extensions (MVC, 3D-AVC, MV-HEVC, 3D-HEVC, etc.), such as neural network-based video encoders. This encoding method can also be implemented in any type of audio encoder or decoder, such as those compliant with the MP3, AAC (Advanced Audio Coding), MPEG-H 3D Audio standards, such as neural network-based audio encoders.

[0084] refer to Figure 1 , the encoding method according to the present invention comprises the following steps:

[0085] In C1, current audio and / or video data is selected.

[0086] This audio data is the current sample set Bc In the form of, the current sample set can be:

[0087] - One-dimensional time audio signal;

[0088] - a part of such signal;

[0089] - Multidimensional time audio signals (stereo or with dimensions greater than two).

[0090] This video data is the current pixel set B c The current pixel set can be in the form of:

[0091] - original current image;

[0092] - a part or region of the original current image;

[0093] - blocks of the current picture resulting from partitioning of that picture according to operations performed in a standardized AVC, HEVC or VVC encoder.

[0094] In C2, the current prediction data set BP is calculated by intra-frame, inter-frame, IBC (intra-frame block copy), SKIP and other prediction methods known to those skilled in the art. c (The data is, for example, pixels).

[0095] In the context of audio coding, the current prediction dataset BP c The data in is a sample.

[0096] In C3, the current pixel set B is calculated c and the current predicted pixel set BP obtained in C2 c The difference between the signals BE c .

[0097] In C4, the signal BE c In the case of a signal for which coding is optimized with respect to conventional coding performance criteria (e.g. minimizing distortion / rate cost, or selecting the best efficiency / complexity trade-off, as criteria well known to those skilled in the art), the signal BE c Quantize and encode.

[0098] At the end of this operation, a quantized and entropy-coded difference signal BE is obtained c cod This entropy coding is performed, for example, by Huffman coding or CABAC coding. In a preferred embodiment, the entropy coding is CABAC coding.

[0099] In C5, a signal or stream F is generated to contain the quantized and coded difference signal BE c codIn a manner known per se, the signal F can be transmitted to a decoding device or decoder, which will be described later in this description.

[0100] According to the present invention, at least one of operations C1 to C5 is implemented using an artificial intelligence-based computing device (labeled DCIA_C) configured to automatically perform the at least one encoding operation to make it more efficient and adaptive. Such computing devices include, for example, one or more neural networks, support vector machines, inference engines, expert systems, fuzzy logic systems, and the like.

[0101] In a preferred embodiment, the computing device is an encoded artificial neural network, such as a convolutional neural network (CNN), a multilayer perceptron, an LSTM (long short-term memory), etc. Such a neural network is defined by a structure comprising, for example, a plurality of artificial neuron layers and / or a set of weights respectively associated with the artificial neurons of the network.

[0102] More specifically, in a preferred embodiment, the neural network used is a convolutional neural network. In a specific embodiment, the neural network calculates the difference signal BE in C3 c Or the current pixel set B c and the predicted pixel set BP generated in C2 c The audio samples are encoded together, thereby performing operations C3 and C4. This type of neural network is, for example, of the type described in Ladune, "Optical Flow and Mode Selection for Learning-based Video Coding," IEEE MMSP 2020. In another specific embodiment, prediction operation C2 is also performed using a convolutional neural network, rather than a traditional prediction device, such as a VVC or CELP (Code Excited Linear Prediction) type prediction device in the case of audio samples. This type of neural network is specifically described in Théo Ladune and Pierrick Philippe, "AIC Artificial Intelligence Based Video Codec," February 17, 2022.

[0103] The use of one or more computing devices based on artificial intelligence to implement a method for encoding audio and / or video data requires that the encoding device implementing the encoding method has a specific hardware or software configuration. Such a configuration is also required in the decoding device so that the decoding device can decode the encoded audio and / or video data signal received from the encoding device including the one or more computing devices in real time. The decoding configuration belongs to one or more categories, including, for example:

[0104] - a first category corresponding to at least one specific physical characteristic of the hardware or software supported by the decoding device in order to be able to decode the signal F, and / or

[0105] - a second category (when at least one encoding step is carried out by a computing device DCIA_C), this second category corresponding to one or more specific characteristics of the data signal F,

[0106] - a third category corresponding to at least one specific processing function applied by the decoding device in order to be able to decode the signal F.

[0107] By way of non-exhaustive example, decoding features of the first category include:

[0108] - the specific type of electronic circuitry used by the computing device, such as a GPU (graphics processing unit), a TPU (tensor processing unit), a DSP (digital signal processor), or any other suitable type of electronic circuitry;

[0109] - the size of the data memory (e.g., buffer memory) that the decoding device must have in order to store all the data required for neural decoding of the signal F;

[0110] - The minimum number of operations per second that the decoding device must be able to perform, for example TOPS (tera operations per second);

[0111] - the level of accuracy of the representation of the parameters of the computing device DCIA_C with which the decoding device must comply in order to be able to decode the signal F, such parameters including, for example, the weights of the decoding artificial neural network and / or the parameters of the activation function applied to the outputs of the artificial neurons used by the decoding device;

[0112] - a plurality of latent variable entropy decoding sources that the decoding device must support in order to be able to decode the signal F,

[0113] - wait.

[0114] By way of non-exhaustive example, decoding features of the second category include:

[0115] - when using one or more neural networks, the minimum number of latent variables to be processed per unit time by the decoding device so that the decoding device maintains its real-time decoding capability;

[0116] - the maximum number of latent variables per unit time, allowing a decoding device to check whether it is capable of decoding the signal;

[0117] - the minimum number of bits (entropy rate) required to represent a syntax element per unit time to reconstruct the latent variable, so that a decoding device maintains its real-time decoding capabilities;

[0118] - represents the maximum number of bits per unit time of syntax elements intended to reconstruct the latent variable (entropy rate), so that a decoding device can check whether it is capable of decoding the signal;

[0119] - wait.

[0120] The decoded features are transmitted to the decoding device by way of indicators that explicitly give the values ​​of these features, or by more comprehensive indicators that indicate the values ​​of multiple features at once using a predetermined association table (eg, Table T10 or Table T11 described below).

[0121] The benefit of transmitting latent variables at a rate (expressed as the number of latent variables per unit time or the rate at which these latent variables are encoded per unit time) is that it allows the complexity of the data to be processed by the neural network to be adjusted based on the neural network processing capabilities of the decoding device. In practice, an encoded signal representing, for example, an image or video may contain both encoded data whose decoding implements conventional processing operations typically performed by a CPU (central processing unit) processor, and data whose decoding implements neural processing operations typically performed by a processor (GPU or TPU processor) that allows for the parallel execution of numerous small computations of the same nature. The characteristic of specifying the latent variable rate facilitates adjusting the sub-portion of the signal that is related to the neural data processing and GPU or TPU capabilities of the decoding device.

[0122] By way of non-exhaustive example, decoding features of the third category include:

[0123] - a set of predefined and standardized logical and / or mathematical operators, and / or a list of logical and / or mathematical operators, supported by a decoding device in order to be able to decode the signal;

[0124] - a list of activation functions applied by the decoding device at the output of the neural network and / or neural networks in order to be able to decode the signal;

[0125] - the ability to reproduce the decoding result in a manner faithful to the reference (inter-platform reproducibility), which a decoding device must have in order to be able to decode the signal;

[0126] - wait.

[0127] All of these decoding features are considered in the context of encoding / decoding audio data and encoding / decoding video data.

[0128] According to the invention, the encoding method comprises a step C6 of encoding one or more information ICD representing a decoding configuration which the decoding device must have in order to decode the encoded data DAT, and which belongs to the first and / or second and / or third categories mentioned above.

[0129] At the end of coding C6, one or more information ICDs are obtained cod .

[0130] Then, in C7, one or more information ICD cod The data signal F or a signal F' associated with the signal F is written.

[0131] refer to Figure 2A , in C7, one or more information ICD cod Written in a manner similar to the VPS or SPS syntax (e.g. as implemented in the VVC standard) in the signal F, a data packet is provided, which can be identified and decoded independently of the coded audio and / or video data DAT, said packet comprising other information required for decoding the coded data DAT, such as predefined audio and / or video parameters of the coding method or, when the coded data DAT represents a sequence of images, parameters associated with this sequence of images. According to another example, one or more items of information ICD cod are parameters associated with a sub-image, specifically a tile or a slice as defined in the HEVC standard.

[0132] refer to Figure 2B , in C7, one or more information ICD cod The optional packet F' is written in a manner such as writing information into a SEI (Supplemental Enhancement Information) message according to the VVC standard, which optional packet does not need to be decoded in order to decode the data DAT.

[0133] In C8, the coded data DAT and one or more information ICD are included cod A signal F comprising coded data DAT and a signal F comprising one or more coded information items ICD cod The message F' is stored or transmitted to a decoding device, which will be described later in this specification.

[0134] In a preferred embodiment, the information ICD codEach decoding configuration or feature represented is signaled separately.

[0135] The coding information ICD is shown in the corresponding syntax table below cod Various examples of .

[0136] Regarding specific hardware to be supported by the decoding device, such as a specific type of electronic circuit or processor, the indicator proc_idc as shown in the syntax table T1 below takes eight values ​​ranging from 0 to 7 to indicate to the decoding device the type of processor that supports decoding signal F.

[0137]

[0138] Regarding the data memory size that the decoding device must have, the indicator buffer_size_idc specifies the size of this memory in bytes, or alternatively can take a predetermined number of values ​​associated with a predefined size limit, as shown in the following syntax table T2. For example, the indicator buffer_size_idc takes eight values ​​ranging from 0 to 7.

[0139]

[0140] Regarding the minimum number of operations per second to be supported by the decoding device, the indicator ops_idc specifies the number of operations per second, or alternatively may take a predetermined number of values ​​associated with a predefined limit, such as the values ​​in the following syntax table T3, in which the indicator ops_idc takes four values, for example, 0 to 3:

[0141]

[0142] Regarding the minimum number of latent variables to be processed per unit time by the decoding device, the indicator latent_rate_idc specifies this number as the number of variables per second (that is to say in terms of rate), or alternatively may take a predetermined number of values ​​associated with a predefined rate limit, such as the values ​​in the following syntax table T4, in which the indicator latent_rate_idc takes seven values, e.g. 0 to 6:

[0143]

[0144] Regarding the level of precision of the representation of the parameters of the computing device DCIA_C, and in the case where this device is a neural network, the indicator precision_idc specifies the required precision of the weights of this network using the following predetermined association table T5, in which the indicator precision_idc takes, for example, eight values ​​from 0 to 7:

[0145]

[0146] Regarding the capability that a decoding device must have to reproduce the decoding result in a manner faithful to the reference (inter-platform reproducibility), in Table T6 below, the indicator repro_flag is set to:

[0147] - a first value (e.g. 0) to indicate that the decoding device does not have to reproduce the reference decoding exactly,

[0148] - or a second value (e.g., 1) to indicate that the decoding device must fully reproduce the reference decoding.

[0149]

[0150] Regarding the number of latent variable entropy decoding sources that a decoding device must support, the indicator sources_idc specifies the number of sources, or alternatively can take a predetermined number of values ​​associated with a predefined limit on the number of sources, such as the values ​​in the following Table T7, in which the indicator sources_idc takes four values, for example, from 0 to 3:

[0151]

[0152] With respect to a set or list of predefined and standardized logical and / or mathematical operators to be supported by a decoding device, the indicator operator_idc specifies the logical and / or mathematical operators supported on the decoder side. According to a preferred embodiment shown in the following syntax table T8, the indicator operator_idc includes five values ​​from 0 to 4, with the values ​​0 to 3 being constructed in a nested manner such that:

[0153] - The value 0 is associated with a set of basic mathematical operators "+, -, x, / ",

[0154] - The value 1 is associated with a set of basic mathematical operators "+, -, x, / " and a set of mathematical operators "x y , exp(), sqrt()”,

[0155] - The value 2 is associated with a set of basic mathematical operators "+, -, x, / ", a set of mathematical operators "x y , exp(), sqrt()" and the operator "N!", where N is a natural number.

[0156] - The value 3 is associated with a set of basic mathematical operators "+, -, x, / ", a set of mathematical operators "x y , exp(), sqrt()", operator "N!" and a set of mathematical operators "sin(), cos(), tan()".

[0157]

[0158] Of course, this way of signaling operators is not exhaustive. In other embodiments, each operator or operator list may be signaled individually, which may result in higher signaling costs.

[0159] Regarding the list of activation functions applied by the decoding device, the indicator activations_idc specifies the activation functions supported by the decoding device. According to a preferred embodiment shown in the following syntax table T9, the indicator activations_idc includes three values ​​from 0 to 2, and the values ​​0 and 1 are constructed in a nested manner, such that:

[0160] - The value 0 is associated with the following list of activation functions:

[0161] F(x) = x,

[0162] If x < 0, then G(x) = 0, otherwise 1

[0163] If x < 0, then H(x) = 0, otherwise x

[0164] - The value 1 is associated with this activation function list, and is also associated with the following activation function lists:

[0165] I(x) = 1 / (1+exp(-x))

[0166] J(x) = tan -1 (x)

[0167]

[0168] In one specific embodiment, a single indicator, level_idc, simultaneously specifies multiple decoding features to be supported by a decoding device. To this end, a mapping table (hereinafter labeled T10) is generated, which is predefined on both the encoding and decoding devices. Table T10 maps at least one specific value of the indicator level_idc to a specific latent variable rate, a specific data memory size, and so on. In the example shown, the indicator level_idc has six values, from 0 to 5.

[0169]

[0170] In a specific embodiment, a series of indicators cat1_level_idc, cat2_level_idc, cat3_level_idc, instead of a single indicator level_idc, specifies the one or more decoding features according to whether the one or more decoding features belong to the first category, the second category, or the third category.

[0171] To this end, a correspondence table predefined on both the encoding device side and the decoding device side is generated for each indicator cat1_level_idc, cat2_level_idc, cat3_level_idc. Such a table is shown below and is marked with a reference T11.

[0172] Regarding the first category, the following table T11 maps at least one specific value of the indicator cat1_level_idc to a specific type of processor, a specific data memory size, etc. In the example shown, the indicator cat1_level_idc has six values ​​from 0 to 5.

[0173]

[0174] Regarding the second category, the following table T12 maps at least one specific value of the indicator cat2_level_idc to a specific latent variable rate. In the example shown, the indicator cat2_level_idc has five values ​​0 to 4.

[0175]

[0176] Regarding the third category, the following table T13 maps at least one specific value of the indicator cat3_level_idc to the ability or inability to reproduce the decoding result, a specific mathematical operator list, etc. In the example shown, the indicator cat3_level_idc has four values ​​of 0 to 3.

[0177]

[0178] Now refer to Figure 3 A description is given of a coder COD shown in schematic form which, in a specific embodiment of the invention, is designed to implement Figure 1 The encoding method shown in .

[0179] According to this particular embodiment, the actions performed by the coding method are implemented by computer program instructions. To this end, the coding device COD has the conventional architecture of a computer and specifically comprises a memory MEM_C, a processing unit UT_C, which is equipped with, for example, a processor PROC_C, and is driven by a computer program PG_C stored in the memory MEM_C. The computer program PG_C includes instructions for implementing the actions of the coding method, such as those described above, when the program is executed by the processor PROC_C.

[0180] At initialization, the code instructions of the computer program PG_C are loaded into a RAM memory (not shown) for example before being executed by the processor PROC_C. The processor PROC_C of the processing unit UT_C implements the actions of the encoding method described above in particular according to the instructions of the computer program PG_C.

[0181] The encoder COD receives the current pixel set or sample set B at the input terminal E_C c , and delivers at an output S_C a transport stream F which is transmitted to a decoder using a suitable communication interface (not shown).

[0182] The coder COD comprises a prediction device PRED configured to implement the above-mentioned prediction step C2. As already explained in the description above, this prediction device may be:

[0183] - conventional, and configured according to standards such as HEVC, VVC, CELP, etc.;

[0184] - Neural networks, e.g. of the type described in the above documents, Théo Ladune, Pierrick Philippe, "AIC ARTIFICIAL INTELLIGENCE BASED VIDEO CODEC [AIC Artificial Intelligence Based Video Codec File]", February 17, 2022,

[0185] - wait.

[0186] The encoder COD also includes an artificial intelligence-based computing device DCIA_C, which is, for example, of the type described in the above-mentioned document: Ladune, "Optical Flow and Mode Selection for Learning-based Video Coding", IEEE MMSP2020.

[0187] The encoder COD also comprises an information coding device CICD configured to implement the above-mentioned step C6 of encoding one or more items of information ICD representative of a decoding configuration that a decoding device must have in order to decode the signal F.

[0188] The encoder COD also comprises a device IICD configured to implement the information ICD to be obtained by the device CICD cod The above-mentioned step C7 of writing the data signal F or writing the message F' associated with the signal F.

[0189] The encoder COD further includes a storage memory MS_C configured to store syntax tables T1 to T12. Alternatively, the storage memory MS_C is not included in the encoder COD but is accessible using any suitable means (e.g., via a communications network). In one embodiment, the data signal F and optional message F' may also be stored in the storage memory MS_C or in an additional storage memory (not shown).

[0190] Decoding of encoded audio and / or video data

[0191] The following describes a method for decoding coded audio and / or video data signals associated with 2D or 3D images or image sequences. This decoding method can be implemented in any type of video decoder, such as those compliant with the JPEG, AVC, HEVC, VVC standards and their extensions (MVC, 3D-AVC, MV-HEVC, 3D-HEVC, etc.), such as neural network-based video decoders. This decoding method can also be implemented in any type of audio decoder, such as those compliant with the MP3, AAC, MPEG-H 3D Audio standards, such as neural network-based audio decoders.

[0192] refer to Figure 4 , the decoding method according to the present invention includes the following steps.

[0193] In D1, the data signal F is generated by Figure 5 The decoding device DEC shown in FIG. 1 receives the data signal containing the coded audio and / or video data DAT and the coding information ICD indicating a specific decoding configuration. cod , the decoding device DEC must have this specific decoding configuration in order to decode the signal F.

[0194] Alternatively, in D1 the decoding device DEC receives:

[0195] - a data signal F containing coded audio and / or video data DAT,

[0196] - Contains coded information ICD indicating a specific decoding configuration cod The decoding device DEC must have this specific decoding configuration in order to decode the signal F.

[0197] In D2, the coded audio and / or video data DAT and the coding information ICD are extracted from the received data signal F. cod .

[0198] Alternatively, in D2, the coded audio and / or video data DAT are extracted from the received data signal F and the coded information ICD is extracted from the received message F' cod .

[0199] According to the present invention, in D3, one or more pieces of coded information ICD are cod At the end of this operation, the information ICD is reconstructed, thereby allowing the decoder to identify one or more decoding characteristics required for it to be able to decode the coded audio and / or video data DAT. To this end, in a particular embodiment of the invention, the value of one or more indicators (for example, the indicators proc_idc, buffer_size_idc, ops_idc, latent_rate_idc, precision_idc, repro_flag, sources_idc, operators_idc, activations_idc, level_idc, cat1_level_idc, cat2_level_idc, cat3_level_idc) is read and then mapped to its associated decoding characteristic or other associated decoding characteristics, in particular in the case of the indicators cat1_level_idc and cat3_level_idc. This mapping is implemented using the aforementioned correspondence tables T1 to T13, which are accessible to the decoding device DEC.

[0200] In D4, the decoding device DEC compares the one or more decoding features identified in D3 with its specific one or more decoding features. This comparison is possible because the decoding device DEC has access to the technical characteristics of the platform (whether software, hardware or a hybrid) on which it runs and the performance it can achieve.

[0201] If, at the end of the comparison D4 , the decoding device DEC does not possess one or more decoding features identified in D3 , the decoding of the coded audio and / or video data DAT is not performed. The decoding is therefore aborted (ABD).

[0202] If, at the end of the comparison D4, the decoding device DEC possesses one or more decoding features identified in D3, decoding of the coded audio and / or video data DAT is performed using an artificial intelligence-based computing device DCIA_D, which performs a decoding corresponding to the encoding performed by the computing device DCIA_C.

[0203] To this end, in D5, the coded audio and / or video data DAT are dequantized and entropy decoded. Such entropy decoding is, for example, Huffman decoding or CABAC decoding. In a preferred embodiment, the entropy decoding is CABAC decoding. At the end of this operation, a decoded difference signal BE is obtained. c dec .

[0204] In D6, the prediction is implemented to generate the current prediction data set BP c , these data are pixels, for example, but could also be samples of an audio signal.

[0205] Step D5 and step D6 may be performed in any order or simultaneously.

[0206] In D7, the decoded difference signal BE obtained in D5 is converted into c dec The predicted pixel set or sample set BP obtained in D6 c Combine to calculate the reconstructed current pixel set BD c .

[0207] In a manner known per se, the current pixel set BD is reconstructed c The reconstructed signal may be subjected to filtering by performing loop filtering (which is well known to those skilled in the art).

[0208] Of course, the difference signal BE calculated during the above encoding method c When the value is zero, that is, in the skip coding mode, the difference signal BE is not extracted. c step D2 and step D5 of dequantization and entropy decoding.

[0209] Now refer to Figure 5 A description is given of a decoder DEC shown in schematic form which, in a particular embodiment of the invention, is designed to implement Figure 4 The decoding method shown in .

[0210] According to this embodiment, the actions performed by the decoding method are implemented by computer program instructions. To this end, the decoding device DEC has the conventional architecture of a computer and specifically comprises a memory MEM_D, a processing unit UT_D, which is equipped with, for example, a processor PROC_D, and is driven by a computer program PG_D stored in the memory MEM_D. The computer program PG_D includes instructions for implementing the actions of the decoding method, such as those described above, when the program is executed by the processor PROC_D.

[0211] The decoder DEC receives the Figure 3The encoder COD transmits a data signal F (possibly a message F') and delivers the currently decoded set of pixels or samples BD at the output S_D. c .

[0212] The decoder DEC further comprises an information decoding device DICD configured to implement the above-mentioned processing of one or more pieces of coded information ICD representing a decoding configuration. cod To perform the decoding step D3 , the decoder DEC must have the decoding configuration in order to decode the signal F.

[0213] The decoder DEC also comprises a device COMP configured to compare in D4 information ICD representative of one or more reconstructions of the decoding configuration that the decoder DEC must have in order to decode the signal F with specific decoding characteristics of the decoder DEC.

[0214] With Figure 3 In a manner corresponding to the encoder COD, the decoder DEC further includes a storage memory MS_D configured to store the aforementioned syntax tables T1 to T13. Alternatively, the storage memory MS_D is not included in the decoder DEC but is accessible using any suitable means (e.g., via a communication network). In one embodiment, the data signal F and optional message F' may also be stored in the storage memory MS_D or in an additional storage memory (not shown).

[0215] The decoder DEC comprises a prediction device PRED_D configured to implement the above-mentioned prediction step D6. As already explained above in the description, this prediction device may be:

[0216] - conventional, and configured according to standards such as HEVC, VVC, CELP, etc.;

[0217] - Neural networks, e.g. of the type described in the above documents, Théo Ladune, Pierrick Philippe, "AIC ARTIFICIAL INTELLIGENCE BASED VIDEO CODEC [AIC Artificial Intelligence Based Video Codec File]", February 17, 2022,

[0218] - wait.

[0219] The decoder DEC may also include an artificial intelligence-based computing device (denoted as DCIA_D) configured to automatically perform at least one decoding operation in order to make it more efficient and adaptive. Such computing device may include, for example, a decoding artificial neural network or multiple decoding artificial neural networks, a support vector machine, an inference engine, an expert system, a fuzzy logic system, etc.

[0220] In a preferred embodiment, the computing device DCIA_D is a neural network, such as a convolutional neural network or CNN, a multilayer perceptron, an LSTM, etc. More specifically, in a preferred embodiment, the neural network used is a convolutional neural network. Such a neural network is defined by a structure comprising, for example, a plurality of artificial neuron layers and / or a set of weights associated with each of the artificial neurons of the network.

[0221] In one embodiment, the neural network DCIA_D converts in D7 the decoded difference signal BE obtained in D5 c dec The predicted pixel set or sample set BP obtained in D6 c This type of neural network is, for example, the type described in Ladune, "Optical Flow and Mode Selection for Learning-based Video Coding," IEEE MMSP 2020. In another specific embodiment, prediction operation D6 is also implemented using a convolutional neural network, rather than a traditional prediction device (e.g., of the VVC type). This type of neural network is specifically described in Théo Ladune and Pierrick Philippe, "AIC Artificial Intelligence Based Video Codec," February 17, 2022.

[0222] It goes without saying that the embodiments described above have been given by way of complete non-limiting indication only and that a person skilled in the art may easily make numerous modifications without otherwise departing from the scope of the present invention.

Claims

1. A method for encoding audio and / or video data, the method being implemented by a coding device configured to implement at least one step of encoding the audio and / or video data using a coding artificial neural network (DCIA_C), the coding method comprising the following steps: - encode the audio and / or video data (C2-C4), - generating (C5) a data signal comprising the coded audio and / or video data, - encoding (C6) information representing a decoding configuration related to the use of the network, which a decoding device must have in order to decode the encoded data, - Inserting (C7) the coded information into the data signal.

2. A method for decoding coded audio and / or video data, the method being implemented by a decoding device, the method comprising the following steps: - receiving (D1) coded audio and / or video data signals, - decoding (D3) information representative of a decoding configuration as a function of said signal, wherein at least one step of decoding the audio and / or video data is implemented using a decoding artificial neural network (DCIA_D), - checking (D4) whether the decoding device has the decoding configuration corresponding to the decoding information, - Depending on the result of this check the signal is decoded (D5-D7) or not decoded.

3. The encoding method according to claim 1 or the decoding method according to claim 2, wherein: The decoding configuration belongs to: - a first category corresponding to at least one specific physical feature of hardware or software supported by the decoding device in order to be able to decode the signal, and / or - a second category corresponding to a specific characteristic of said signal, and / or - a third category corresponding to at least one specific processing function applied by the decoding device in order to be able to decode the signal.

4. The encoding method or decoding method according to claim 3, wherein: The information, respectively encoded or decoded, representing a decoding configuration is associated with at least one of the first category, the second category or the third category.

5. The encoding method according to any one of claims 1 and claims 3 to 4 or the decoding method according to any one of claims 2 to 4, wherein: The decode configuration belongs to a collection that includes the following: - Maximum size of data storage memory; - Minimum number of operations per second; - minimum latent variable rate; - a predetermined number of values ​​each associated with a predefined latent variable rate limit; - certain types of electronic circuits; - the minimum number of bits representing a syntactic element intended to reconstruct a latent variable per unit time; - the maximum number of bits representing the syntactic elements intended to reconstruct the latent variable per unit time; - the level of accuracy of the mathematical representation of at least one operating parameter of the decoded artificial neural network; - activation or inactivation of at least one decoding identical to a reference decoding; - at least one specific mathematical operator, or a list of specific mathematical operators; - specific mathematical functions; - Multiple sources of entropy decoding statistics.

6. The encoding method according to any one of claims 1 and claims 3 to 5 or the decoding method according to any one of claims 2 to 5, wherein: The information representative of the decoding configuration is contained in a set of predefined video parameters of the encoding method or the decoding method, respectively, or, when the video data represent a sequence of images, in a set of parameters associated with said sequence.

7. The encoding method according to any one of claims 1 and claims 3 to 6 or the decoding method according to any one of claims 2 to 6, wherein: The information indicating the decoding configuration includes: - a first value, the first value being associated with a first configuration element of the decoding configuration, and - a second value associated with the first configuration element and the second configuration element of the decoding configuration.

8. A device for encoding audio and / or video data, the device being configured to implement at least one step of encoding the audio and / or video data using an encoding artificial neural network, the encoding device being configured to implement the following steps: - encoding the audio and / or video data, - generating a data signal comprising the encoded audio and / or video data, - encoding information representing a decoding configuration related to the use of the network, which a decoding device must have in order to decode the encoded data, - inserting said coded information into said data signal.

9. A device for decoding encoded audio and / or video data, the device being configured to implement the following steps: - receiving coded audio and / or video data signals, - decoding information representing a decoding configuration according to the signal, wherein: At least one step of decoding the audio and / or video data is implemented using a decoding artificial neural network, - checking whether the decoding device has the decoding configuration corresponding to the decoding information, - Decoding or not decoding the signal according to the result of this check.

10. A computer program comprising program code instructions for implementing the encoding method according to claim 1 and any one of claims 3 to 7 or the decoding method according to any one of claims 2 to 7 when the computer program is executed on a computer.

11. A computer-readable information medium comprising instructions of the computer program as claimed in claim 10.

Images