Video Decoding Using Neural Networks

Abstract

The data stream includes an indicator (IND) and data (Fnn) representing video or audio content. A method for decoding this data stream comprises the steps of: decoding the indicator (IND) to identify (E60) whether an artificial neural network used to decode the representation data (Fnn) is encoded in the data stream or forms part of a set of predetermined artificial neural networks; and decoding (E70) the representation data (Fnn) by the artificial neural network. If decoding the indicator (IND) identifies that the artificial neural network forms part of the aforementioned predetermined set, the method involves the step of decoding (E62) an identifier (Inn) of the neural network. Related decoding devices and related data streams are also described.

Description

[Technical Field]

[0001] The present invention relates to the technical field of decoding audiovisual content.

[0002] In particular, the present invention relates to a method for decoding a data stream and to an associated apparatus and data stream. [Background technology]

[0003] It has been proposed to compress data representing video content using an artificial neural network, and the decoding of the compressed data can then be performed by another artificial neural network, as described, for example, in the article "DVC: An End-to-end Deep Video Compression Framework" by Guo Lu et al., in 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 2019, pp. 10998-11007. Summary of the Invention [Means for solving the problem]

[0004] In this regard, the invention proposes a method for decoding a data stream comprising an indicator and data representative of an audio or video content, said method comprising the steps of: - decoding an indicator to identify whether the artificial neural network used to decode the representation data is encoded in the data stream or belongs to a predetermined set of artificial neural networks; - decoding said representation data by an artificial neural network; Includes:

[0005] Such an indicator therefore tells the decoder how it can access the artificial neural network used to decode the data representing the content.

[0006] The decoding method may comprise the step of decoding an identifier of the neural network if decoding the indicator identifies the artificial neural network as belonging to said predetermined set.

[0007] The invention also proposes, in a manner originally known per se, a method for decoding a data stream comprising data representative of an audio or video content and an identifier designating one of a set of predetermined artificial neural networks, said method comprising: - decrypting the identifier; - decoding said representation data by an artificial neural network specified by the decoded identifier; Includes:

[0008] Decoding the identifier therefore indicates which of a set of predetermined artificial neural networks, for example a set of predetermined artificial neural networks accessible to the electronic decoding device, must be used, as will be explained below.

[0009] According to one possibility, the decoding method may further comprise a step of reading, in the storage unit, the parameters of the artificial neural network identified by the decoded identifier.

[0010] The storage unit may be adapted to store a first set of parameters representing a first artificial neural network forming a random access decoder and / or a second set of parameters representing a second artificial neural network forming a low latency decoder.

[0011] Furthermore, the decoding method may comprise the step of generating an error message if there is no data (here in the storage unit) relating to the artificial neural network identified by the decoded identifier.

[0012] According to a second possibility, the method may comprise a step of receiving, from a remote server, parameters of the artificial neural network identified by the decoded identifier.

[0013] Furthermore, the decoding method may include, if decoding the indicator identifies that an artificial neural network is encoded in the data stream, decoding data contained in the data stream that encodes the artificial neural network to obtain parameters of the artificial neural network.

[0014] According to a possible embodiment, the decoding method may include a (preliminary) step of transmitting a list of artificial neural networks to a device for controlling the transmission of the data stream. In a particular embodiment, this list of artificial neural networks may correspond to the aforementioned predetermined set. In other words, in this case, the predetermined set of artificial neural networks may include (or be formed by) the artificial neural networks of this list.

[0015] The content may in fact comprise a first part of a video sequence, which video sequence may therefore comprise said first part and a second part.

[0016] In this case, the step of decoding the representation data generates the first portion, and the method may further include decoding other data with another artificial neural network to generate a second portion.

[0017] According to a possible embodiment, the other artificial neural networks have a structure identical to that of the said artificial neural network, which simplifies the updating of the artificial neural networks in the electronic decoding device.

[0018] The first part and the above-mentioned second part form, for example, two image groups respectively for the content representation format used.

[0019] The invention also proposes a decoding device, which comprises: - a unit for receiving a data stream containing an indicator and data representative of audio or video content; - a decoding assembly designed to determine whether the artificial neural network used to decode the representation data belongs to a set of predetermined artificial neural networks or is encoded in the data stream by decoding the indicator, and to decode the representation data by the artificial neural network; Includes:

[0020] The invention further proposes a decoding device, which comprises: - a unit for receiving a data stream containing data representing audio or video content and an identifier specifying an artificial neural network from a set of predetermined artificial neural networks; - a decoding assembly designed to decode the identifier and to decode said representation data by an artificial neural network designated by said decoded identifier; Includes:

[0021] In the embodiments described below, such a decoding assembly includes a processor designed or programmed to decode the indicators and / or identifiers and / or parallel processing units designed to perform multiple operations of the same type in parallel at a given time and to implement the above-mentioned artificial neural networks for decoding the above-mentioned representation data.

[0022] The invention also proposes a data stream, which comprises data representing an audio or video content and an indicator indicating whether the artificial neural network used to decode said representation data is encoded in the data stream or belongs to a set of predetermined artificial neural networks.

[0023] The invention finally proposes a data stream comprising data representing an audio or video content and an identifier specifying one artificial neural network from a set of predetermined artificial neural networks to be used to decode said representation data.

[0024] It should be understood that the various features, alternatives and embodiments of the invention can be associated with one another in various combinations, provided they are not mutually inconsistent or mutually exclusive.

[0025] Furthermore, various other features of the present invention will become apparent from the following description taken in conjunction with the following drawings, which illustrate non-limiting embodiments of the invention. [Brief explanation of the drawings]

[0026] [Figure 1] 1 shows an electronic encoding device used within the framework of the present invention; [Figure 2] 2 is a flow chart illustrating steps of an encoding method performed within the electronic encoding device of FIG. 1; [Figure 3] 3 is a first example of a data stream obtained by the method of FIG. 2. [Figure 4] 3 is a second example of a data stream obtained by the method of FIG. 2. [Figure 5] 3 is a third example of a data stream obtained by the method of FIG. 2. [Figure 6] 4 is a fourth example of a data stream obtained by the method of FIG. 2. [Figure 7]1 illustrates an electronic encoding device according to an embodiment of the present invention; [Figure 8] 8 is a flow chart showing the steps of a decoding method carried out in the electronic decoding device of FIG. 7; DETAILED DESCRIPTION OF THE INVENTION

[0027] FIG. 1 shows an electronic encoding device 2 that uses at least one artificial neural network 8 .

[0028] The electronic encoding device 2 includes a processor 4 (eg a microprocessor) and a parallel processing unit 6, such as a graphics processing unit, or GPU, or a tensor processing unit, or TPU.

[0029] As shown diagrammatically in FIG. 1, a processor 4 receives data P, B, here format data P and content data B, representing audio or video content to be compressed.

[0030] The format data P indicates the characteristics of the representation format of the audio or video content, for example in the case of video content the image size (in pixels), the frame rate, the binary depth of the luminance information and the binary depth of the chrominance information.

[0031] The content data B forms a (here uncompressed) representation of audio or video content. For example in the case of video content the content data comprises, for each pixel of each image of an image sequence, data representing the luminance value of that pixel and / or data representing the chrominance value of the pixel.

[0032] The parallel processing unit 6 is designed to implement the artificial neural network 8 after being configured by the processor 4. For that purpose, the parallel processing unit is designed to perform several operations of the same type in parallel at a given time.

[0033] As will be explained below, the artificial neural network 8 is used within the framework of processing content data B in order to obtain compressed data C.

[0034] In the embodiment described herein, when content data B is applied at the input of an artificial neural network 8, the artificial neural network 8 produces compressed data C at the output.

[0035] The content data B applied at the input of the artificial neural network 8 (i.e. applied at the input layer of the artificial neural network 8) may also represent a block of an image, or a block of an image component (e.g. a block of luminance or chrominance components or a block of color components of this image), or an image of a video sequence, or a component of an image of a video sequence (e.g. a luminance or chrominance component or color component), or a series of images of a video sequence.

[0036] For example, in this case, at least some of the neurons of the input layer may be adapted to receive pixel values ​​of the image components respectively, said values ​​being represented by one of the content data B.

[0037] Alternatively, processing of the content data B may involve the use of some artificial neural networks, as described, for example, in the aforementioned article, "DVC: An End-to-end Deep Video Compression Framework" by Guo Lu et al., 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.

[0038] An example of an encoding method performed by the electronic encoding device 2 will now be described with reference to FIG.

[0039] A memory linked to the processor 4 stores, for example, computer program instructions which, when executed by the processor 4, are designed to perform at least some of the steps of the method of Figure 2. In other words, the processor 4 is programmed to perform at least some of the steps of Figure 2.

[0040] The method of FIG. 2 now begins with an optional step E2 of receiving a list of artificial neural networks accessible by the electronic decoding device.

[0041] The list is for example received by the processor 4 of the electronic encoding device 2 directly from an electronic decoding device (for example corresponding to the electronic decoding device 10 of FIG. 7 described below).

[0042] As will be explained below, an artificial neural network accessible by an electronic decoding device is one in which the electronic decoding device stores parameters defining the artificial neural network to which it pertains, or can access these parameters by connection to a remote electronic facility such as a server.

[0043] Alternatively, the list can be received directly by the processor 4 of the electronic device 2 from a remote server, such as the servers mentioned above.

[0044] The method of Figure 2 continues with step E4 of selecting an encoding process-decoding process pair, as described above for the encoding process, each of the encoding and decoding processes using at least one artificial neural network.

[0045] In the examples described herein, the encoding process is performed by an encoding artificial neural network, and the decoding process is performed by a decoding artificial neural network.

[0046] The set formed by an encoding artificial neural network and a decoding artificial neural network, the output of which is applied to the input of the decoding artificial neural network, forms, for example, an autoencoder.

[0047] The encoding process-decoding process pair is selected, for example, from a plurality of preset encoding process-decoding process pairs, ie, here, a plurality of encoding artificial neural network-decoding artificial neural network pairs.

[0048] If a list of artificial neural networks accessible by the electronic decoding device has been received in advance (as described above in step E2), the encoding process-decoding process pair is selected, for example, from among the encoding process-decoding process pairs that use an artificial neural network that is in the list for which the decoding process has received it.

[0049] The encoding process-decoding process pair may also be selected depending on the intended use (e.g., indicated by a user using a user interface, not shown, of electronic encoding device 2). For example, if the intended use is video conferencing, the selected encoding process-decoding process pair may include a low-latency decoding process. For other uses, the selected encoding process-decoding process pair may include a random-access decoding process.

[0050] In a low latency process for decoding a video sequence, the images of the video sequence may, for example, be represented by coded data that is immediately transmitted and decoded, and the data may then be transmitted in the display order of the video images, thereby ensuring a one-frame latency between encoding and decoding.

[0051] In a random access process for decoding a video sequence, the coded data for each of several images is transmitted in an order other than the display order of these images, thereby increasing compression. Images coded without reference to other images (so-called intraframes) can then be coded in an orderly manner, allowing decoding of the video sequence to start from several points within the coded stream.

[0052] For that purpose, the reader is referred to the article "Overview of the High Efficiency Video Coding (HEVC) Standard" by GJ Sullivan, J.-R. Ohm, W.-J. Han and T. Wiegand in IEEE Transactions on Circuits and Systems for Video Technology, vol. 22, no. 12, pp. 1649-1668, December 2012.

[0053] The encoding process-decoding process pair may also be selected to obtain the best compression-distortion balance.

[0054] To that end, it is possible to apply several encoding process-decoding process pairs to the content data B and select the set that reaches the best compression-distortion balance.

[0055] Alternatively, it is possible to identify the type of content (eg, by analyzing the content data B) and select an encoding process-decoding process pair according to the identified type.

[0056] The encoding process-decoding process pair may also be selected depending on the computational power available at the electronic decoding device, information representing this computational power may have been previously transmitted from the electronic decoding device to the electronic encoding device (e.g. received by the electronic encoding device in step E2 above).

[0057] It may also be possible to combine various criteria for selecting an encoding process-decoding process pair with one another.

[0058] Once an encoding process-decoding process pair has been selected, the processor 4 proceeds to step E6 in which it configures the parallel processing units 6 in such a way that they are able to carry out the selected encoding method.

[0059] This step E6 comprises the instantiation in the parallel processing unit 6 of an encoding artificial neural network 8 to be used in particular by the encoding process selected.

[0060] This instantiation involves, among other things, the following steps: - reserving, in the parallel processing unit 6, the memory space necessary to implement the encoding artificial neural network; and / or - programming the parallel processing units 6 with the weights W and activation functions defining the encoding artificial neural network 8, and / or - loading at least a portion of the content data B into the local memory of the parallel processing unit 6; may include:

[0061] The method of Figure 2 therefore comprises a step E8 of carrying out an encoding process, i.e. applying the content data B to the input of the encoding artificial neural network 8 (or in other words operating the encoding artificial neural network 8 by taking the content data B as input).

[0062] Step E8 therefore makes it possible to generate compressed data C (here at the output of the encoding artificial neural network 8).

[0063] The following steps relate in particular to the encoding (ie, preparation) of a data stream containing compressed data C and destined for an electronic decoding device (eg, electronic decoding device 10 described with reference to FIG. 7).

[0064] The method therefore comprises a step E10 of encoding a first header part Fc containing in particular data characteristics of the representation format of the audio or video content (here for example data associated with the format of the video sequence being encoded).

[0065] These data forming the first header part Fc indicate, for example, the image size (in pixels), the frame rate, the binary depth of the luminance information and the binary depth of the chrominance information, and are constructed, for example, based on the above-mentioned format data P (after possible reformatting).

[0066] The method of Figure 2 therefore continues with step E12 of identifying the availability of a decoding artificial neural network (to be used by the decoding process selected in step E4) to an electronic decoding device (e.g., electronic decoding device 10 described below in relation to Figure 7) that is able to decode the data stream.

[0067] This determination may be made on the basis of the list received in step E2, in which case processor 4 determines whether the decoding artificial neural network used by the decoding process selected in step E4 belongs to the list received in step E2. (Of course, in embodiments in which the encoding process-decoding process pairs are systematically selected to correspond to the decoding artificial neural networks available to the electronic decoding device, step E12 may be omitted and the method therefore continues with step E14.)

[0068] According to a possible embodiment, if there is no information about the availability of a decoding artificial neural network to the electronic decoding device, the method continues to step E16 (as already explained, so that data describing the decoding artificial neural network are transmitted to the electronic decoding device).

[0069] If the processor 4 determines in step E12 that a decoding artificial neural network is available to the electronic decoding device (arrow P), the method continues with step E14 described below.

[0070] If the processor 4 determines in step E12 that a decoding artificial neural network is not available to the electronic decoding device (arrow N), the method continues with the following step E16.

[0071] Alternatively, the choice between step E14 and step E16 as the step after step E12 can also depend on other criteria, for example on dedicated indicators stored in the electronic encoding device 2 (and possibly adjustable by the user via the user interface of the electronic encoding device 2) or on a selection made by the user (for example obtained via the user interface of the electronic encoding device 2).

[0072] The processor 4 proceeds to step E14 in which it encodes a second header part containing the indicator IND and a third header part containing here the identifier Inn of the decoding artificial neural network.

[0073] In step E14, an indicator IND is coded in the data stream indicating that the decoding artificial neural network belongs to a predetermined set of artificial neural networks, here the set of artificial neural networks available (or accessible) to the electronic decoding device (i.e. the set of artificial neural networks in the list received in step E2).

[0074] The identifier Inn of the decoding artificial neural network is an identifier that, by transformation, defines this decoding artificial neural network (shared in particular by the electronic encoding device and the electronic decoding device), for example within the above-mentioned predetermined set.

[0075] The processor 4 then proceeds to step E16, at which it encodes the second header part containing the indicator IND' and the third header part containing data Rc describing the decoding artificial neural network.

[0076] At step E16, an indicator IND' is coded in the data stream indicating that the decoding artificial neural network has been coded in the data stream, ie represented by the above-mentioned description data Rc.

[0077] The decoding artificial neural network is for example coded (i.e. represented) by description data (or data encoding the decoding artificial neural network) Rc according to a standard such as MPEG-7 part 17 or a format such as JSON.

[0078] For that purpose, please refer to the article "DeepCABAC: Context-adaptive binary arithmetic coding for deep neural network compression" by S. Wiedemann et al. in Proceedings of the 36th International Conference on Machine Learning, Long Beach, California, PMLR 97, 2019, or the article "Compact and Computationally Efficient Representation of Deep Neural Networks" by S. Wiedemann et al. in IEEE Transaction on Neural Networks and Learning Systems (Vol. 31, Issue 3), March 2020.

[0079] After step E16, at step E14, the method of FIG. 2 continues with step E18 in which the electronic decoding device identifies the possibilities for carrying out the decoding process using a decoding artificial neural network.

[0080] The processor 4 identifies this possibility, for example, by determining whether the electronic decoding device includes (perhaps through a previous exchange between the electronic encoding device and the electronic decoding device) a module designed to perform this decoding process or software that is suitable for the electronic decoding device to perform this decoding process when the software is executed by the processor of this electronic decoding device.

[0081] If the processor 4 determines that the electronic decoding device is capable of carrying out the decoding process, the method continues with step E22, which will be described below.

[0082] If the processor 4 determines that the electronic decoding device is not capable of carrying out the decoding process, the method carries out step E20 described below (before proceeding to step E22).

[0083] Alternatively, the choice of whether to perform step E20 (before performing step E22) may depend on other criteria, for example on a dedicated indicator stored in the electronic encoding device 2 (and possibly adjustable by the user via the user interface of the electronic encoding device 2) or on a selection made by the user (for example obtained via the user interface of the electronic encoding device 2).

[0084] The processor 4 encodes, in step E20, a fourth header part in the data stream containing a computer program Exe (or code) executable by the processor of the electronic decoding device (the use of the computer program Exe in the electronic decoding device will be explained below with reference to Figure 8).

[0085] In order to be adapted for execution within the electronic decoding device, the computer program is selected within the library depending, for example, on information regarding the hardware configuration of the electronic decoding device (e.g., information received from a previous exchange between the electronic encoding device 2 and the electronic decoding device).

[0086] The processor 4 therefore proceeds to step E22, in which it encodes the compressed stream Fnn on the basis of the compressed data C obtained at step E8.

[0087] In this respect, it can be seen that in the above description, step E8 is described before the step of encoding the header Fet (steps E10 to E20), but in practice step E8 can be carried out just before step E22.

[0088] In particular, if step E8 allows processing of only a part of the audio or video content to be compressed (for example, if step E8 performs processing of a block, component or image of a video sequence to be compressed), it is possible to repeat the execution of steps E8 (to obtain compressed data relating to successive parts of the content) and E22 (to encode the compressed data obtained in the data stream).

[0089] The processor 4 is therefore able to construct, in step E24, the complete data stream including the header Fet and the compressed stream Fnn.

[0090] The complete data stream is constructed in such a way that the header Fet and the compressed stream Fnn are separately identifiable.

[0091] According to a possible embodiment, the header Fet contains an indicator of the start of the compressed stream Fnn in the complete data stream, for example the bit position of the beginning of the compressed stream Fnn from the start of the compressed data stream (in other words, the header in this case has a predetermined fixed length).

[0092] Other ways of identifying the header Fet and the compressed stream Fnn can alternatively be envisaged, such as a marker (i.e. a binary combination used to indicate the start of the compressed stream Fnn, the use of which is hidden in the rest of the data stream or at least in the header Fet).

[0093] 3-6 show examples of complete data streams that can be obtained by the method of FIG.

[0094] As already explained, these data streams contain a header Fet and a compressed stream Fnn.

[0095] In the case of FIG. 3 (corresponding to the situation where step E14 has been carried out but step E20 has not), the header - a first part Fc containing data characteristics of the representation format of the audio or video content; - a second part containing an indicator IND indicating that the decoded artificial neural network belongs to a predetermined set of artificial neural networks; - a third part containing an identifier of the decoding artificial neural network, Inn; Includes:

[0096] In the case of FIG. 4 (corresponding to the situation where step E16 has been carried out but step E20 has not), the header - a first part Fc containing data characteristics of the representation format of the audio or video content; - a second part containing an indicator IND' indicating that a decoding artificial neural network is coded in the data stream; a third part containing data Rc describing the decoding artificial neural network (here the data encoding it), Includes:

[0097] In the case of FIG. 5 (corresponding to the situation where steps E14 and E20 have been executed), the header - a first part Fc containing data characteristics of the representation format of the audio or video content; - a second part containing an indicator IND indicating that the decoded artificial neural network belongs to a predetermined set of artificial neural networks; - a third part containing an identifier of the decoding artificial neural network, Inn; - a fourth part containing a computer program Exe; Includes:

[0098] In the case of FIG. 6 (corresponding to the situation where steps E16 and E20 have been executed), the header - a first part Fc containing data characteristics of the representation format of the audio or video content; - a second part containing an indicator IND' indicating that a decoding artificial neural network is coded in the data stream; a third part containing data Rc describing the decoding artificial neural network (here the data encoding it), - a fourth part containing a computer program Exe; Includes:

[0099] The data stream constructed in step E24 can be encapsulated in a transmission format known per se, such as the "Packet-Transport System" format or the "Byte-Stream" format.

[0100] In the "Packet-Transport System" format (for example, as proposed by the RTP protocol), data is encoded in identifiable packets and transmitted over a communication network. The network can easily identify the boundaries of the data (images, image collections, and here the header Fet and the compressed stream Fnn) using packet identification information provided by the network layer.

[0101] In the case of the "Byte-Stream" format, there are no specific packets and the construction of step E24 must be able to identify the boundaries of the relevant data (e.g. between the parts of the stream corresponding to each image and here between the header Fet and the compressed stream Fnn) using additional means such as the use of Network Abstraction Layer (NAL) units, in which case it is possible to identify the boundaries between data by a unique binary combination (e.g. 0x00000001).

[0102] The complete data stream constructed in step E24 can then be emitted in step E26 to the electronic decoding device 26 (by communication means not shown and / or through at least one communication network) or stored in the electronic encoding device 2 (for later transmission or alternatively for later decoding, for example, within the electronic encoding device itself (in this case designed to further carry out the decoding method described below in relation to Figure 8)).

[0103] When the audio or video content comprises several parts (e.g. several images if the content is a video sequence), the method of steps E4 to E24 can possibly be performed in such a way as to obtain, for each part of the content (e.g. for each image), a data stream as shown in one of Figures 3 to 6. The compressed stream Fnn for each image group can thus be decoded using an artificial neural network appropriate for the image group concerned, possibly different from the artificial neural networks used for the other images, as described below. The artificial neural networks can probably have the same structure (and only differ in their weights and / or the activation functions that define the particular artificial neural network).

[0104] FIG. 7 shows an electronic decoding device 10 that uses at least one artificial neural network 18 .

[0105] The electronic decoding device 10 comprises a receiving unit 11, a processor 14 (eg a microprocessor) and a parallel processing unit 16, for example a graphics processing unit, ie GPU, or a tensor processing unit, ie TPU.

[0106] The receiving unit 11 is, for example, a communication circuit (such as a radio frequency communication circuit) that makes it possible to receive data (and in particular here an encoded data stream) from an external electronic device, such as the electronic encoding device 2, and to communicate these data to a processor 14 (to which the receiving unit 11 is connected, for example, by a bus).

[0107] The electronic decoding device 10 also includes a storage unit 12, such as a memory (which may be a rewritable non-volatile memory) or a hard drive. Although the storage unit 12 is shown in Figure 7 as a separate element from the processor 14, the storage unit 12 may alternatively be integrated into (i.e., included within) the processor 14.

[0108] The processor 14 is in this case adapted to successively execute a plurality of instructions of a computer program stored, for example, in the storage unit 12 .

[0109] The parallel processing unit 16, after being configured by the processor 14, is designed to implement the artificial neural network 18. To that end, the parallel processing unit 16 is designed to perform multiple operations of the same type in parallel at any one time.

[0110] As shown schematically in FIG. 7, the processor 14 receives (e.g., via communication means, not shown, of the electronic decoding device 10) a data stream including a first data set, here a header Fet, and a second data set, here a compressed stream Fnn, representing audio or video content.

[0111] As will be explained later, the artificial neural network 18 is used to obtain the audio or video content corresponding to the initial audio or video content B within the framework of processing the second data set (i.e. here the compressed data Fnn).

[0112] The storage unit 12 can store a plurality of parameter sets, each parameter set defining a decoding artificial neural network, and as will be described below, the processor 14 can then configure the parallel processing units 16 with a particular one of these parameter sets in such a way that the parallel processing units 16 can implement the artificial neural network defined by this particular parameter set.

[0113] The storage unit 12 may in particular store a first set of parameters defining a first artificial neural network forming a random access decoder and / or a second set of parameters defining a second artificial neural network forming a low latency decoder.

[0114] In this case, the electronic decoding device 10 has a pre-decoding option for both situations where random access to the content is desired and situations where the content is to be displayed without delay.

[0115] With reference now to FIG. 8, a decoding method performed within the electronic decoding device 10 using an artificial neural network 18 implemented by the parallel processing unit 16 will be described.

[0116] The method may begin with an optional step of transmission by the electronic decoding device 10 of a list L of artificial neural networks available to it, to a device controlling the transmission of the data stream to be decoded. The data stream transmission control device may, for example, be the electronic encoding device 2. (The electronic encoding device 2 in this case receives this list L in step E2 described above with reference to FIG. 2.) Alternatively, the data stream transmission control device may also be a dedicated server cooperating with the electronic encoding device 2.

[0117] An artificial neural network accessible to the electronic decoding device 10 is one for which the electronic decoding device 10 stores a set of parameters defining the relevant artificial neural network (as described above) or can access this set of parameters by connection with a remote electronic facility such as a server (as described below).

[0118] The method of Figure 8 includes a step E52 of receiving (by the electronic decoding device 10, more precisely here by the receiving unit 11) a data stream comprising a first set of parameters, i.e. a header Fet, and a second set of parameters, i.e. a compressed stream Fnn. The receiving unit 11 transmits the received data stream to the processor 14.

[0119] The processor 14 therefore proceeds to step E54 in which it identifies the first data set (header Fet) and the second data set (compressed stream Fnn) in the received data stream, for example by means of an indicator of the beginning of the compressed stream (already mentioned in the description of step E24).

[0120] In step E54, the processor 14 may also identify different parts within the first data set (header), i.e. here the header Fet, which are a first part Fc (containing data characteristic of the format of the content encoded by the data stream), a second part (indicator IND or IND'), a third part (Inn or encoded data Rc) and possibly a fourth part (computer program Exe), as shown in the above figures 3 to 6.

[0121] If executable instructions (e.g. instructions of a computer program Exe) are identified (i.e. detected) in the first data in step E54, the processor 14 may execute these executable instructions in step E56 to initiate the execution of at least certain of the steps of processing data from the first data set. These instructions may be executed by the processor 14 or alternatively by a virtual machine instantiated in the electronic decoding device 10.

[0122] The method of Figure 7 continues with a step E58 of decoding data Fc characteristic of the representation format of the audio or video content in such a way as to obtain the characteristics of this format. In the case of video content, for example, decoding the data portion Fc makes it possible to obtain the image size (in pixels), and / or the frame rate, and / or the binary depth of the luminance information and / or the binary depth of the chrominance information.

[0123] The processor 14 then proceeds to step E60 of decoding (in the particular embodiment by executing the instructions identified in the first data set at step E54 as described above) the indicators IND, IND', here contained in the second part of the header Fet.

[0124] If the decoding of the indicators IND, IND' present in the received data stream indicates that the artificial neural network 18 used for decoding belongs to a predetermined set of artificial neural networks (i.e. if the indicator present in the first data set is an indicator IND indicating that the decoding artificial neural network 18 belongs to a predetermined set of artificial neural networks), the method continues with step E62, which will be described below.

[0125] If the decoding of the indicator IND, IND' present in the received data stream indicates that the artificial neural network 18 used for decoding is encoded in the data stream (i.e. if the indicator present in the first data set is an indicator IND' indicating that the decoding artificial neural network 18 is encoded in the data stream), the method continues with step E66, which will be described below.

[0126] In step E62, the processor 14 proceeds (in a particular embodiment by executing the instructions identified in the first data set in step E54 as described above) to decode the identifier Inn (here contained in the third part of the header Fet). As described above, this identifier Inn is, for example, an identifier designating the decoding artificial neural network 18 within the set of predetermined artificial neural networks described above.

[0127] The processor 14 can therefore proceed to step E64 of reading (by execution of instructions identified in the first data set in step E54 as described above in the particular embodiment) from the storage unit 12 the parameter set associated with the decoded identifier Inn (this parameter set defining the artificial neural network identified by the decoded identifier Inn).

[0128] According to a possible embodiment, the processor 14 may be adapted to generate an error message if there are no data (here in the storage unit 12) relating to this artificial neural network identified by the decoded identifier Inn (in particular the parameters).

[0129] Alternatively (or if no parameter set is stored in the storage unit 12 for the artificial neural network identified by the decoded identifier Inn), the electronic decoding device 10 may (in certain embodiments, by executing instructions identified in the first data set in step E54 as described above) send a request for a parameter set to a remote server (this request including, for example, the coded identifier Inn) and receive as answer, in step E64, a parameter set defining the artificial neural network identified by the decoded identifier Inn.

[0130] The method then continues with step E68, which will be described below.

[0131] In step E66, the processor 14 proceeds to decode (in the particular embodiment by executing the instructions identified in the first data set in step E54 as described above) the data Rc describing the artificial neural network 18 (here contained in the third part of the header Fet).

[0132] As mentioned above, these description data (or encoded data) Rc are encoded according to a standard such as MPEG-7 part 17 or a format such as JSON.

[0133] The encoding of the description data Rc makes it possible to obtain the parameters that define the artificial neural network used to decode the data from the second data set, ie here the data of the compressed stream Fnn.

[0134] The method then continues with step E68, which is described below.

[0135] The processor 14 then proceeds to step E68, in which it configures (in the particular embodiment by executing the instructions identified in the first data set at step E54 as described above) the parallel processing units 16 with the parameters defining the decoding artificial neural network 18 (the parameters obtained at step E64 or step E66) in such a way that the parallel processing units 16 are able to implement the decoding artificial neural network 18.

[0136] This configuration step E68 notably involves instantiating a decoding artificial neural network 18 in the parallel processing unit 16, here using the parameters obtained in step E64 or step E66.

[0137] This instantiation, in particular, - reserving, in the parallel processing unit 16, the memory space necessary to implement the decoding artificial neural network 18; and / or - programming the parallel processing units 16 with parameters (including, for example, the weights W' and the activation function) defining the decoding artificial neural network 18 (parameters obtained at step E64 or step E66), and / or - loading at least part of the data from the second data set (i.e. at least part of the data of the compressed stream Fnn) into the local memory of the parallel processing unit 16; may include:

[0138] As can be seen from the above description of steps E58 to E68, the data of the first data set Fet are therefore processed in this way by the processor 14.

[0139] The processor 14 can then, in step E70, apply (i.e. present) the data from the second data set (here the data of the compressed stream Fnn) to the artificial neural network 18 implemented by the parallel processing unit 16 in such a way that these data are processed by a decoding process that at least partly uses the artificial neural network 18.

[0140] In the example described herein, the artificial neural network 18 receives as input the data of the second data set Fnn and generates as output a representation I of the encoded content adapted for playback on an audio or video playback device. In other words, at least certain pieces of the data of the second data set Fnn are applied to the input layer of the artificial neural network 18, and the output layer of the artificial neural network 18 generates said representation I of the encoded content. In the case of video content (comprising one image or a sequence of images), the artificial neural network 18 therefore generates as output (i.e. at its output layer) at least one matrix representation I of the image.

[0141] In certain embodiments, to process certain data of the compressed stream Fc (e.g. corresponding to a certain block or a certain image), the artificial neural network 18 may receive as input at least certain pieces of data generated by the artificial neural network 18 during the processing of previous data in the compressed stream Fc (e.g. corresponding to a previous block or image). In this case, it proceeds to step E72, in which the data generated at the output of the artificial neural network 18 is re-injected into the input of the artificial neural network 18.

[0142] Furthermore, according to another possible embodiment, the decoding process may use multiple artificial neural networks, as already mentioned above for the processing of content data B.

[0143] The data from the second set (here at least certain data of the compressed stream Fnn) is therefore processed using an artificial neural network 18 implemented by a parallel processing unit 16, by a process that depends on the portion of the data from the first set (here a process that depends on the identifier Inn of the encoded data Rc).

[0144] Processor 14 then determines, in step 74, whether artificial neural network 18 has finished processing compressed stream Fnn.

[0145] If the identification result is negative (N), the method returns to step E70 and applies other data of the compressed stream Fnn to the artificial neural network 18.

[0146] If the determination result is positive (P), the method continues at step E76, where the processor 14 determines whether there is data remaining to be processed in the received data stream.

[0147] If the result of the determination at step E76 is negative (N), the method ends at step E78.

[0148] If the determination at step E76 is positive (P), the method returns to step E52 and processes a new portion of the data stream shown in one of the figures 3 to 6.

[0149] As shown above, as a repetition of the encoding steps E4 to E24, this other part of the data stream therefore also comprises the first set of data and a second set of data representing other audio or video content (e.g., in the case of video content, another group of pictures related to the representation format of the content used). In this case, as described above in steps E54 to E66, a further artificial neural network can be identified based on certain of these first data, and the parallel processing unit 16 can then be configured to implement this further artificial neural network (in accordance with the above-mentioned step E68). Data from the second set of data of this other part of the data stream (e.g., related to said other group of pictures) can then be decoded by this further artificial neural network (as described above in step E70).

[0150] The alternative artificial neural network may have a structure identical to that of the artificial neural network 18 described above, thereby simplifying the step of configuring the parallel processing unit 16 (e.g., only the weights and / or activation functions defining the current artificial neural network are updated).

Claims

1. 1. A method for decoding a data stream containing indicators (IND, IND') and presentation data (Fnn) representing audio or video content, comprising: a step (E60) of decoding said indicators (IND, IND') to identify whether the artificial neural network (18) used to decode said representation data (Fnn) is coded in said data stream or belongs to a set of predetermined artificial neural networks; a step (D70) of decoding said expression data (Fnn) by said artificial neural network (18); A decoding method comprising:

2. 2. The decoding method of claim 1, wherein the audio or video content is a first portion of a video sequence, the video sequence including the first portion and a second portion, the step of decoding the representation data produces the first portion, and the method further comprises the step of decoding other data by another artificial neural network to produce the second portion.

3. 3. A method of decoding according to claim 2, wherein said other artificial neural network has a structure identical to that of said artificial neural network.

4. 4. A decoding method according to claim 2 or 3, wherein the first and second parts respectively form two groups of pictures for the representation format of the audio or video content used.

5. Decoding method according to any one of claims 1 to 3, comprising a step (E50) of transmitting the list (L) of artificial neural networks to a device for controlling the transmission of the data stream.

6. 4. The method of claim 1, further comprising a step (E62) of decoding an identifier (Inn) of said artificial neural network (18) if decoding said indicator (IND) identifies said artificial neural network (18) as belonging to said predetermined set.

7. 7. A decoding method according to claim 6, comprising the step of reading, in a storage unit (12), the parameters of the artificial neural network (18) identified by the decoded identifier (Inn).

8. 8. The decoding method of claim 7, wherein the storage unit (12) stores a first set of parameters representing a first artificial neural network forming a random access decoder and a second set of parameters representing a second artificial neural network forming a low latency decoder.

9. 7. A method of decoding as claimed in claim 6, including the step of generating an error message if there is no data relating to the artificial neural network identified by the decoded identifier.

10. 7. A method of decoding according to claim 6, comprising the step of receiving, from a remote server, the parameters of said artificial neural network (18) identified by said decoded identifier (Inn).

11. 4. The method of decoding according to claim 1, further comprising a step (E66) of decoding data (Rc) contained in the data stream encoding said artificial neural network (18) to obtain the parameters of said artificial neural network (18), if decoding said indicator (IND') identifies that said artificial neural network (18) is encoded in said data stream.

12. A decoding device, comprising: a unit (11) for receiving a data stream containing indicators (IND, IND') and presentation data (Fnn) representative of an audio or video content; a decoding assembly (14, 16) designed to determine, by decoding said indicators (IND, IND'), whether the artificial neural network (18) used to decode said representation data (Fnn) belongs to a set of predetermined artificial neural networks or is coded in said data stream, and to decode said representation data by said artificial neural network (18); A decoding device comprising:

13. 1. A data stream comprising representation data (Fnn) representing audio or video content and indicators (IND, IND') indicating whether an artificial neural network (18) used to decode said representation data (Fnn) is encoded in said data stream or belongs to a set of predetermined artificial neural networks.

Images