Method and apparatus for updating a neural network

Abstract

Described herein is a method of generating a media bitstream for transmitting parameters for updating a neural network implemented in a decoder, wherein the method comprises the steps of: (a) determining at least one set of parameters for updating the neural network; (b) encoding the at least one set of parameters and media data to generate the media bitstream; and (c) transmitting the media bitstream to the decoder for updating the neural network with the at least one set of parameters. Also described herein is a method for updating a neural network implemented in a decoder, an apparatus for generating a media bitstream for transmitting parameters for updating a neural network implemented in a decoder, an apparatus for updating a neural network implemented in a decoder, and a computer program product comprising a computer readable storage medium having instructions adapted to cause a device having processing capabilities to carry out the above method when executed by the device.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the following priority applications: U.S. Provisional Application No. 62 / 818,879 filed on March 15, 2019 (Docket No.: D19009USP1) and European Application No. 19174542.1 filed on May 15, 2019 (Docket No.: D19009EP), which are hereby incorporated by reference into this application. Technical Field

[0003] The present disclosure generally relates to methods and apparatus for generating a media bitstream for transmitting parameters for updating a neural network in a decoder and for updating the neural network by the decoder.

[0004] Although some embodiments will be described herein with particular reference to the present disclosure, it will be understood that the present disclosure is not limited to such areas of use and is applicable in a broader context. Background Art

[0005] Any discussion of the background art throughout the disclosure should not be considered as an admission that such art is widely known or forms part of the common general knowledge in the field.

[0006] Recently, neural networks have attracted increasing attention due to their increasing reliability.

[0007] Typically, a deep neural network generator is trained by interacting with a so-called discriminator on a real-fake basis. Specifically, during training, the generator is used to generate so-called fake data, while the discriminator tries to distinguish this fake data from real data. By iterating this procedure, the generator is eventually trained to generate fake data that is close to real data (to the extent that the discriminator cannot distinguish it from real data). Recent work is mainly based on deep convolutional generative adversarial networks (GANs). GANs have been successfully used, for example, in various computer vision and image processing tasks, as well as speech and audio related applications.

[0008] Once a deep neural network (e.g., a deep neural network generator) has been successfully trained on a specific task, it can be applied to that specific task, for example, at a decoder. However, for a trained neural network implemented in a decoder and acting on media content that has been extracted from a bitstream provided by an encoder, the neural network may or may not produce satisfactory results, depending on the specific encoding constraints that existed when encoding the media content at the encoder. This is especially true when the encoding constraints may change over time. For example, changes in the bitrate or frame rate of the encoded media content may have an impact on the quality of the results of the neural network processing the media content.

[0009] Therefore, there is a need for methods and apparatus that allow for greater flexibility in adapting a decoder-implemented neural network to specific requirements. In particular, there is a need for methods and apparatus that enable a decoder-implemented neural network to process media content to handle variable coding constraints during the encoding phase of the media content. Summary of the Invention

[0010] In view of the above needs, the present disclosure provides a method for generating a media bitstream for transmitting parameters for updating a neural network implemented in a decoder, a method for updating a neural network implemented in a decoder, and corresponding devices and computer program products, wherein the above method, device and computer program product respectively have the features of the corresponding independent claims.

[0011] According to a first aspect of the present disclosure, a method for generating a media bitstream for transmitting parameters for updating a neural network implemented in a decoder (e.g., a method for transmitting such a media bitstream) is provided. The method may include the step of (a) determining at least one set of parameters for updating the neural network. The method may also include the step of (b) encoding the at least one set of parameters and media data to generate a media bitstream. This may involve encoding the at least one set of parameters and media data into a bitstream. The method may also include the step of (c) transmitting (outputting) the media bitstream to a decoder so as to utilize (e.g., use) the at least one set of parameters to update the neural network. Outputting / transmitting the media bitstream may involve transmitting the media bitstream via a wired or wireless connection, or storing the media bitstream on a tangible medium and providing the tangible medium to the decoder.

[0012] In some embodiments, the media data may include one or more of audio data and / or video data.

[0013] In some embodiments, the at least one set of parameters may be encoded (into the media bitstream) based on a set of syntax elements. This may involve generating (creating) syntax elements.

[0014] In some embodiments, in step (a), two or more sets of parameters for updating the neural network may be determined, and the sets of syntax elements may include one or more syntax elements identifying a respective set of parameters for a respective update of the neural network to be performed.

[0015] In some embodiments, a neural network implemented in a decoder can be used to process media data, and, in a media bitstream, the at least one set of parameters used to update the neural network can be time-aligned with (the portion of) the media data processed by the neural network (e.g., after the neural network is updated using the at least one set of parameters). For example, the media data can be divided into portions of the media data (e.g., frames of the media data). Then, for each portion, the corresponding at least one set of parameters used to update the neural network processing the portion can be time-aligned with the portion. For frame-based media data, each frame can include media data and at least one corresponding set of parameters that can be used to process the media data.

[0016] In some embodiments, the at least one set of parameters may be determined based on one or more of a codec mode, the content of the media data, and encoding constraints. In some embodiments, the codec mode may include one or more of a bit rate, a video and / or audio frame rate, and a core codec used (e.g., ASF or SSF, etc.). In some embodiments, the content of the media data may include one or more of speech, music, and applause. In some embodiments, the encoding constraints may include one or more of constraints for performance scalability and constraints for adaptive processing (e.g., for adaptive streaming). In some embodiments, the at least one set of parameters is included in the media bitstream prior to the media data to be processed by the corresponding updated neural network.

[0017] In some embodiments, the media data may be in MPEG-H Audio format or MPEG-I Audio format, and the media bitstream may be a packetized media bitstream in MHAS format. The at least one set of parameters (update parameters) and the corresponding media data may be packaged together in the media bitstream.

[0018] In some embodiments, the at least one set of parameters may be encoded by encapsulating the at least one set of parameters into one or more MHAS packets of a (possibly new) MHAS packet type.

[0019] In some embodiments, the media data may be in AC-4, AC-3, or EAC-3 format.

[0020] In some embodiments, the at least one set of parameters may be encoded as one or more payload elements in the media bitstream.

[0021] In some embodiments, the media data may be in MPEG-4 or MPEG-D USAC format.

[0022] In some embodiments, the at least one set of parameters may be encoded in the media bitstream as one or more payload elements or one or more data stream elements.

[0023] In some embodiments, the at least one set of parameters includes parameters for updating weights of one or more layers of the neural network, including an identifier identifying whether the parameters for updating the weights represent relative values ​​or absolute values.

[0024] In some embodiments, the neural network may have a layer structure and the at least one set of parameters may include parameters for updating weights of at least a media data-oriented layer (i.e., the first layer of the plurality of layers) and / or an output layer (i.e., the last layer of the plurality of layers) of the neural network. In one example, the neural network has three or more layers, with at least one layer between the media data-oriented layer and the output layer.

[0025] In some embodiments, a media bitstream is generated by encoding only the parameters for updating the weights of the layers facing the media data and / or the output layer, of the at least one set of parameters for updating the weights of the plurality of layers of the neural network, and the media data. In other words, the parameters for updating the weights of the other layers are not encoded in the bitstream. In a first example, the parameters for updating the weights of all layers of the neural network implemented at the decoder are determined, but only the parameters for updating the weights of the layers facing the media data and / or the output layer are encoded in the media bitstream. In a second example, only the parameters for updating the weights of the layers facing the media data and / or the output layer are determined, i.e., no parameters are determined for the other layers.

[0026] According to a second aspect of the present disclosure, a method for updating a neural network implemented in a decoder is provided. The method may include (a) receiving a coded media bitstream including media data and at least one set of parameters for updating the neural network. The method may also include (b) decoding the received media bitstream to obtain decoded media data and the at least one set of parameters for updating the neural network. The method may also include (c) utilizing (e.g., using) the at least one set of parameters by the decoder to update the neural network.

[0027] In some embodiments, the media data may include one or more of audio data and / or video data.

[0028] In some embodiments, the method may further include receiving a set of syntax elements and updating the neural network using (e.g., using) the at least one set of parameters by the decoder. This may be based on the received set of syntax elements.

[0029] In some embodiments, in step (a), two or more sets of parameters for updating the neural network may be included in the received coded media bitstream, and the received set of syntax elements may include one or more syntax elements identifying a respective set of parameters for a respective update of the neural network to be performed.

[0030] In some embodiments, a neural network may be used for processing of media data, and in a received coded media bitstream, the at least one set of parameters may be time-aligned with (the portion of) the media data processed by the neural network (e.g., after using the at least one set of parameters).

[0031] In some embodiments, the neural network may have a layer structure and the weights of at least the media data-oriented layer and / or the output layer of the neural network may be updated by the decoder using (e.g., using) the at least one set of parameters.

[0032] In some embodiments, the received media bitstream includes parameters for updating only the media data-oriented layer and / or the output layer, i.e., parameters for updating other layers of the neural network are not included in the media bitstream. The decoding step may include decoding the received media bitstream to obtain decoded media data and parameters for updating weights of the media data-oriented layer and / or the output layer. The step of updating the neural network may include using the received parameters to update weights of the media data-oriented layer and / or the output layer of the neural network. For example, only the media data-oriented layer and / or the output layer are updated, while other layers are not updated.

[0033] According to a third aspect of the present disclosure, an apparatus for generating a media bitstream for transmitting parameters for updating a neural network implemented in a decoder (e.g., an apparatus for transmitting such a media bitstream) is provided. The apparatus may include a processor configured to perform a method comprising the steps of (a) determining at least one set of parameters for updating the neural network. The method may also include the step of (b) encoding the at least one set of parameters and media data to generate a media bitstream. The method may further include the step of (c) transmitting the media bitstream to a decoder so as to utilize (e.g., use) the at least one set of parameters to update the neural network.

[0034] According to a fourth aspect of the present disclosure, an apparatus for updating a neural network implemented in a decoder is provided. The apparatus may include (a) a receiver for receiving a coded media bitstream, the coded media bitstream including media data and at least one set of parameters for updating the neural network. The apparatus may also include (b) a decoder for decoding the received media bitstream to obtain decoded media data and the at least one set of parameters for updating the neural network. The apparatus may further include (c) an updater for updating the neural network using (e.g., using) the at least one set of parameters.

[0035] In some embodiments, the receiver may also receive a set of syntax elements, and the updater may utilize (e.g., use) the at least one set of parameters to update the neural network based on the set of syntax elements.

[0036] In some embodiments, two or more sets of parameters for updating a neural network may be included in a coded media bitstream received by a receiver, and the received set of syntax elements may include one or more syntax elements identifying a respective set of parameters for a respective update of the neural network to be performed.

[0037] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer-readable storage medium having instructions, wherein the instructions are adapted to, when executed by a device having processing capabilities (e.g., a processor), cause the device to implement a method for generating a media bitstream for transmitting a method for updating parameters of a neural network implemented in a decoder.

[0038] According to a sixth aspect of the present disclosure, there is provided a computer program product comprising a computer-readable storage medium having instructions, the instructions being adapted to, when executed by a device having processing capabilities (e.g., a processor), cause the device to implement a method for updating a neural network implemented in a decoder.

[0039] It should be understood that method steps and device features can be interchanged in a variety of ways. In particular, as will be appreciated by those skilled in the art, the details of the disclosed method can be implemented as a device suitable for performing some or all of the method or the steps thereof, and vice versa. In particular, it should be understood that the method according to the present disclosure relates to a method for operating a corresponding device. It should also be understood that corresponding statements regarding the method are equally applicable to the corresponding device. It should further be understood that corresponding statements regarding the encoder / transmitter side are equally applicable to the decoder / receiver side. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Example embodiments of the present disclosure will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0041] Figure 1A flowchart illustrating an example of a method of generating a media bitstream for transmitting parameters for updating a neural network implemented in a decoder,

[0042] Figure 2 A flowchart illustrating an example of a method for updating a neural network implemented in a decoder is shown, and

[0043] Figure 3 An example of a neural network implemented as a multi-layer generator including an encoder stage and a decoder stage is schematically illustrated. DETAILED DESCRIPTION

[0044] Overview

[0045] In the context of the present disclosure, deep neural networks can be used to process audio and / or video media data. For audio, possible applications of neural networks include media analysis, media enhancement (e.g., encoding audio enhancement), media classification (speech, movie, music, applause, etc.), media generation (e.g., generating deep generative models for missing or new media data), or dialogue enhancement. For video, possible applications of neural networks include deblocking or motion enhancement (e.g., for sports programs).

[0046] A neural network may receive as input media content, which may be divided into suitable parts (e.g., frames), and output one or more processed (e.g., enhanced) media content (audio and / or video), analysis results of the media content, classifications of the media content, or newly generated media content.

[0047] As has been found, for example, for speech content, small changes to the weights of an existing neural network in the decoder / client can often significantly improve the quality of the processed media content. Using generative adversarial networks (GANs) as an example of neural networks, it has been observed that when using a deep neural network generator (G) to enhance encoded audio at bitrate X, only minimal training time is required (starting from a model pre-trained for bitrate X) to also enable it to work at a (different) bitrate Y. In other words, the pre-trained model already works at bitrate Y, and merely incremental training appears to be sufficient to achieve optimal performance. Similar observations apply to different types of media content (e.g., pre-training on speech and adapting to applause).

[0048] To determine incremental updates, GAN training can be based on transfer learning, a method in which a neural network trained to perform a specific task is adapted to a different task.

[0049] In the context of the present disclosure, a neural network implemented by a decoder is updated based on update parameters transmitted along with corresponding media data in an encoded (normalized) media bitstream. The update parameters may be packaged into the media bitstream along with the corresponding media data. The media bitstream is decoded at the decoder, and the update parameters and corresponding media data are extracted. The update parameters are used to update the neural network before processing the corresponding media data, and the updated neural network processes the corresponding media data.

[0050] In other words, the present disclosure relates, at least in part, to delivering updates (update parameters) to be applied to an existing neural network in a receiving device (e.g., a decoder) over a (standardized) media bitstream, along with the corresponding media data to be processed by the neural network. Preferably, these updates are packaged with the media data, where the updated neural network applies its enhancements on the media data. More preferably, this involves time-aligned transmission of the updates (update parameters) along with the media data, which also means that the media enhancement neural network can be fine-tuned based on the current media content.

[0051] Thus, broadly speaking, the present disclosure relates to transmitting neural network updates along with the encoded media to a decoding system in a media bitstream. This applies to video, audio, and other media-related content (e.g., haptics, etc.). A list of non-limiting examples of where parameter updates can be transmitted within different types of bitstreams is further provided below.

[0052] Moreover, as will be described in more detail below, the methods and apparatus described in the present disclosure, as well as the underlying transport mechanisms for updating parameters of a neural network, may be relevant to ETSI AC-4, ETSI AC-3 / EAC-3, ISO / IEC 23008-3 (MPEG-H Audio), ISO / IEC 14496-3 (MPEG-4 Audio) and all related standards (e.g., 3GPP, etc.), ISO / IEC 23003-3 (USAC), and ISO / IEC 23090-4 (MPEG-I Audio).

[0053] First, refer to Figure 1, which illustrates a method for generating a media bitstream for transmitting parameters for updating a neural network implemented in a decoder. In step S101, at least one set of parameters (update parameters) is determined for updating the neural network implemented in the decoder. The decoder can be understood as a device that receives the at least one set of parameters for updating the neural network. To transmit the at least one set of parameters to the decoder, in step 102, the at least one set of parameters and media data are encoded to generate a media bitstream. That is, the at least one set of parameters is encoded into the media bitstream. Corresponding media data to be processed by the appropriately updated neural network (i.e., updated using the at least one set of parameters) is encoded into the media bitstream along with the at least one set of parameters. For packetized media bitstreams, the media data and the at least one set of parameters can be packaged together. Typically, the at least one set of parameters is preferably time-aligned with the corresponding media data in the media bitstream. The generated media bitstream is then provided (e.g., output or transmitted) to the decoder for updating the neural network in step 103. By generating a media bitstream including the at least one set of parameters and the media data, the at least one set of parameters for updating the neural network can be delivered to the decoder together with the media data. In some embodiments, the media data may include one or more of audio data and / or video data.

[0054] Next reference Figure 2 , which illustrates a method for updating a neural network implemented in a decoder. In step S201, a coded media bitstream including media data and at least one set of parameters for updating the neural network is received. In step S202, the coded media bitstream is decoded to obtain decoded media data and the at least one set of parameters. Then, in step S203, the decoder uses the at least one set of parameters to update the neural network.

[0055] In some embodiments, a neural network may be used for processing media data, and in a received coded media bitstream, the at least one set of parameters may be time-aligned with the media data processed by the neural network. Figure 2 As shown in step S204 of the example, when the at least one set of parameters is time-aligned with the media data, the neural network can be updated by the decoder using the at least one set of parameters to subsequently process the decoded media data. In some embodiments, the media data can include one or more of audio data and / or video data.

[0056] The above method can be implemented in respective devices or through corresponding computer program products.

[0057] Transmission of update parameters in the media bitstream

[0058] In some embodiments, the at least one set of parameters may be encoded based on a set of syntax elements. The syntax elements may allow the at least one set of parameters to be transmitted (communicated) in the generated media bitstream and may further allow the decoder to update the neural network. In some embodiments, the set of syntax elements may be received by the decoder, and the decoder's update of the neural network using the at least one set of parameters may be based on the received set of syntax elements.

[0059] For example, the following syntax elements may be used to transmit the at least one set of parameters in the media bitstream:

[0060] Table 1: Examples of syntax elements for transmitting at least one set of parameters for updating a neural network in a media bitstream.

[0061]

[0062] Referring to the example of Table 1, the syntax elements described therein can be used to pack the required bits of the at least one set of parameters used for neural network updating into the media bitstream. However, the syntax elements can be modified according to the codec used to transmit the corresponding media bitstream of the at least one set of parameters. For example, the variable_bits() syntax specified in ETSI 103 190 can be replaced with the escaped_value() syntax specified in ISO / IEC 23003-3 and USAC.

[0063] Referring again to the example of Table 1, the semantics used with the syntax elements may be as follows:

[0064] neural_network_id can be used to identify the neural network that must be updated in the decoder.

[0065] nn_update_byte may be used to carry the at least one set of parameters for updating the neural network in a specific format.

[0066] upd_type may be used to identify the type of update to be performed using the at least one set of parameters, for example:

[0067] 0 – relative update (transferred values ​​added or subtracted from the original weights, including gradients)

[0068] 1 – Absolute update (transferred values ​​can replace original weights)

[0069] It is worth noting that assigning 0 and 1 to relative updates and absolute updates is a non-limiting example, and in some embodiments the assignment may be reversed. What is important in the current context is that upd_type can indicate both relative updates and absolute updates in some way.

[0070] The update_id may be used to identify the set of parameters for the update to be performed, i.e., to distinguish between different sets of update parameters. This allows for avoiding confusion between parameters from different sets of update parameters. In general, the syntax elements may include one or more syntax elements that identify a respective set of parameters for a respective update to be performed on a neural network.

[0071] number_of_upd_packages may indicate the total number of update packages for the associated neural_network_id. The update using the at least one set of parameters is applied only when the decoder (client) has received all the associated update packages.

[0072] The upd_sequence_counter can be used to identify a specific grouping of related updates.

[0073] nn_update_byte can be used to carry the at least one set of parameters (e.g., weight updates). The format of these bytes may depend on neural_network_id and other values ​​(e.g., upd_type).

[0074] While the structure of the neural network to be updated is not limited, in some embodiments, the neural network may have a (multi-)layer structure, and the at least one set of parameters may include parameters for updating at least the weights of the media data-oriented layer and / or output layer of the neural network. In some embodiments, the weights of at least the media data-oriented layer and / or output layer of the neural network can therefore be updated by the decoder using the at least one set of parameters. In some embodiments, only the media data-oriented layer and / or output layer of the neural network can be updated. For example, for a discrimination model (i.e., a neural network that classifies input media content), only the output layer can be updated.

[0075] For example, the following syntax elements shown in Table 2 may be used to update the neural network using the at least one set of parameters:

[0076] Table 2: Examples of syntax elements for updating a neural network using the at least one set of parameters.

[0077]

[0078] Referring to the example of Table 2, the semantics used with the syntax elements may be as follows:

[0079] layer_id can be used to identify the layer whose weights have to be updated.

[0080] weight_num_in_layer can be used to identify which weight in a layer has to be updated.

[0081] weight_value can be used to carry the value of the weight that must be updated. It can be interpreted according to the value of upd_type, that is, relative update vs. absolute update:

[0082] If upd_type = 0 signed byte

[0083] If upd_type = 1 unsigned byte

[0084] Likewise, the assignment of 0s and 1s is to be understood as a non-limiting example.

[0085] In some embodiments, a neural network implemented in a decoder can be used for processing (e.g., enhancement, analysis, classification, etc.) of media data. The at least one set of parameters in the media bitstream used to update the neural network (the update parameters) can then be time-aligned with the media data being processed by the neural network at the time of the update using the at least one set of parameters. In other words, if media data to be processed arrives at the decoder, the neural network implemented by the decoder must be updated with the parameters intended for that particular portion of media data. Therefore, each update parameter must arrive at least concurrently with, and possibly before, the corresponding media data (the processing of which requires these update parameters).

[0086] In this context, time alignment may refer to the order in which the at least one set of parameters and the media data are encoded into the media bitstream. For example, the at least one set of parameters may be encoded in the media bitstream before the media data to be processed by the corresponding updated neural network. In the case where the media data is audio data representing applause, the at least one set of parameters determined to be used to update the neural network to process the media data representing applause is encoded in the bitstream before the media data. Alternatively or additionally, the at least one set of parameters may be encoded in the media bitstream such that the corresponding media data to be processed by the neural network updated using the at least one set of parameters is delayed relative to the at least one set of parameters. Thereby, any delay in the neural network processing the media data due to incomplete updates may be avoided. If necessary, the updated parameters may be cached in the decoder and used to update the neural network when the corresponding media data arrives at the decoder.

[0087] For example, for a frame-based format, the at least one set of parameters can be encoded in the same frame as the media data to be processed by the neural network updated using the at least one set of parameters. Alternatively, the at least one set of parameters can be encoded in an earlier frame, with a fixed offset in frames between the frame including the at least one set of parameters and the frame including the media data to be processed by the neural network updated using the at least one set of parameters.

[0088] In some embodiments, the at least one set of parameters may be determined based on one or more of a codec mode, content of the media data, and encoding constraints.

[0089] In some embodiments, the codec mode may include one or more of a bit rate, a video and / or audio frame rate, and a core codec used. If, for example, the at least one set of parameters is determined based on the bit rate, this can be used to time-align the at least one set of parameters in the media bitstream relative to corresponding media data having that bit rate. The same applies to the video and / or audio frame rate and the core codec used.

[0090] In some embodiments, the content of the media data may be one or more of speech, music, and applause in the audio data. Similarly, the content of the media data may be video data. If, for example, the at least one set of parameters is determined based on the corresponding content of the audio data and / or video data, this can be used to time-align the at least one set of parameters in the media bitstream relative to the corresponding media data having the content.

[0091] In some embodiments, the coding constraints may include one or more of constraints on scalability performance and constraints on adaptive processing. If, for example, the at least one set of parameters is determined based on the corresponding coding constraints, this may also be used to time-align the at least one set of parameters in the media bitstream relative to the corresponding media data to which the coding constraints apply.

[0092] While the format of the media data is not limited, in some embodiments, the media data may be in the MPEG-H Audio format (ISO / IEC 23008-3) or the MPEG-I Audio format, and the media bitstream may be a packetized bitstream in the MHAS format. In some embodiments, the at least one set of parameters may then be encoded by encapsulating the at least one set of parameters in one or more MHAS packets of a new MHAS packet type. For example, one possible way to transmit the required payload may be to assign a new MHAS packet type, PACTYP_NN_UPDATE.

[0093] The decoder can identify the payload for updating the corresponding (predefined) neural network by interpreting the MHAS packet type. To this end, in the MPEG-H Audio standard, the syntax elements can be modified as shown in Table 3 below.

[0094] Table 3: Examples of modified syntax elements for transmitting the at least one set of parameters in the media bitstream in the MPEG-H Audio standard.

[0095]

[0096] Referring to the example of Table 3, neural_network_update_byte may carry the bytes of the syntax element neural_network_update() described previously. Additionally, depending on the (nature of) the neural network to be updated, the bitstream extension mechanism usacExtElement may be used.

[0097] In some embodiments, the media data may be in AC-4, AC-3, EAC-3, MPEG-4, or MPEG-D USAC format. In addition, in some embodiments, the at least one set of parameters may then be encoded in the media bitstream as one or more payload elements or one or more data stream elements (DSEs).

[0098] In the case of AC-4 (ETSI 103 190), an example of transmitting the at least one set of parameters in the media bitstream may be transmitting (or delivering) a neural_network_update() element via an emdf_payloads_substream() element (see Table 1. To achieve this, a free emdf_payload_id may be used in order to be able to identify this EMDF payload. Each byte contained in the neural_network_update() element may then be transmitted (delivered) as an emdf_payload_byte.

[0099] In the case of AC-3 and EAC-3 (ETSI 102 366), an example of transmitting the at least one set of parameters in the media bitstream may be to transmit (or transfer) a neural_network_update() element via an emdf_container() element as described in Annex H of ETSI 102 366. To achieve this, a free emdf_payload_id may be used to enable identification of the EMDF payload. This ID may be the same as in the case of AC-4 in order to implement the principles of the EMDF framework. Each byte contained in the neural_network_update() element may then be transmitted (or transferred) as an emdf_payload_byte.

[0100] In the case of MPEG-4 (AAC, HE-AACv1, HE-AACv2), the at least one set of parameters for updating the neural network in the media bitstream can be transmitted using at least two independent extension mechanisms defined by MPEG-4. When the first extension mechanism is used, the at least one set of parameters can be transmitted as a data stream within a data stream element (DSE), as shown in the following example:

[0101]

[0102]

[0103] When the second extension mechanism is used, the MPEG-4 defined extension mechanism (especially for non-elastic payloads) can be utilized, and its top-level syntax element extension_payload is contained in the filler element ID_FIL, as shown in the following example:

[0104] Alternatively or additionally, the MPEG-4 extension_type EXT_DATA_ELEMEN may be used. In this case, the bytes of the neural_network_update() syntax element may be transmitted as extension_payload().

[0105] In the case of MPEG-D USAC, the at least one set of parameters for updating the neural network can be transmitted based on the USAC extension element defined in 23003-3. The new usacExtElementType can be named ID_EXT_ELE_NN_UPDATE, and the corresponding syntax elements in Table 17 "Syntax of UsacExtElementConfig()" of 23003-3 can be modified as shown in the following example:

[0106]

[0107] Neural network structure

[0108] Although the structure of the neural network is not limited, the neural network can have a (multi-) layer structure. Such a layer structure will be described based on the following non-limiting example.

[0109] refer to Figure 3 As an example, a neural network can be a so-called generator trained to process audio data, for example, in a generative adversarial network setting. Although the structure of such a generator is generally not limited, the generator can include an encoder stage and a decoder stage. The encoder and decoder stages of the generator can be fully convolutional. The decoder stage can mirror the encoder stage, and the encoder and decoder stages can each include L layers, where each layer L has N filters. L can be > 1 and N can be a natural number > 1. The size of the N filters (also called the kernel size) is not restricted and can be selected according to the application requirements. However, the filter size in each of the L layers can be the same.

[0110] exist Figure 3In the example of , the raw audio data to be processed can be input to the generator in a first step 1. The first encoder layer 3 shown can therefore be a media data-oriented layer. In each of the L layers of the encoder stage, the number of filters can increase. Each filter can operate on the audio data input to each encoder layer with a step size of 2. In this case, the depth increases as the width (duration of the signal) becomes narrower. Therefore, a learnable downsampling by a factor of 2 can be performed. Alternatively, the filters can be operated with a step size of 1 in each encoder layer and then downsampled by a factor of 2 (as in known signal processing).

[0111] In at least one encoder layer and at least one decoder layer, a nonlinear operation may be additionally performed as an activation. The nonlinear operation may include one or more of a parameterized rectified linear unit (PReLU), a rectified linear unit (ReLU), a leaky rectified linear unit (LReLU), an exponential linear unit (eLU), and a scaled exponential linear unit (SeLU). However, other activations are also conceivable depending on the application of the generator.

[0112] like Figure 3 As shown in the example of , the respective decoder layers 7, 8, and 9 can mirror the encoder layers 5, 4, and 3. While the number of filters in each layer and the filter width in each layer can be the same in the decoder level as in the encoder level, upsampling of the audio signal starting from a narrow width (duration of the signal) can be performed in two alternative ways. Fractional stride convolution (also known as transposed convolution) operations can be used in the layers of the decoder level to increase the width of the audio signal to the full duration, i.e., the frame of the audio signal input to the generator.

[0113] Alternatively, in each layer of the decoder stage, after upsampling with an upsampling factor of 2 and interpolation are performed as in conventional signal processing, a filter may operate on the audio data input into each layer with a stride of 1.

[0114] Before the processed audio data is output in the final step 11, an output layer (convolutional layer) 10 can follow the decoder stage. In the output layer, the activation can be different from the activation performed in at least one encoder layer and at least one decoder layer. The activation can be any nonlinear function that is constrained to be within the same range as the audio signal input to the generator. For example, the time signal to be enhanced can be constrained to be between + / - 1. The activation can then be based on, for example, a tanh operation.

[0115] Between the encoder and decoder stages, the audio data may be modified. The modification may be based on the encoded audio feature space (also referred to as the bottleneck layer) 12. The modification in the encoded audio feature space may be accomplished, for example, by concatenating a random noise vector (z) with the vector representation (c) of the original audio data as the output of the last layer in the encoder stage. The random noise vector may be set to z = 0.

[0116] A skip connection 2 may be present between homogeneous layers at the encoder and decoder levels to preserve the temporal structure or texture of the encoded audio, since the encoded audio feature space can be bypassed, thereby preventing information loss. Skip connections may be implemented using one or more of concatenation and signal addition. Due to the implementation of skip connections, the number of filter outputs may be "almost" doubled.

[0117] refer to Figure 3 Taking the example in , the structure of the generator as a non-limiting example of a neural network can be as follows:

[0118] 1 / Input: original audio data

[0119] 3 / Encoder layer L=1: number of filters N=16, filter size=31, activation=PreLU

[0120] 4 / Encoder layer L=2: number of filters N=32, filter size=31, activation=PreLU . . .

[0124] 5 / Encoder layer L=11: number of filters N=512, filter size=31

[0125] 6 / Encoder layer L = 12: number of filters N = 1024, filter size = 31

[0126] 12 / Encoded Audio Feature Space

[0127] 7 / Decoder layer L=1: number of filters N=512, filter size=31 . . .

[0131] 8 / Decoder layer L=10: number of filters N=32, filter size=31, PreLU activation

[0132] 9 / Decoder layer L=11: number of filters N=16, filter size=31, PreLU activation

[0133] 10 / Output layer: number of filters N = 1, filter size = 31, activation tanh

[0134] 11 / Output: Enhanced audio data

[0135] 2 / Skip Connection

[0136] It is worth noting that the above structures represent only non-limiting examples. Depending on the application of such a generator, the number of layers in the encoder level and in the decoder level can be reduced or enlarged, respectively. In addition, the neural network in the context of the present disclosure is understood to be not limited to GAN-related models. For example, although the above examples refer to a generative network model, the neural network can also be a neural network based on a discriminative network model. In general, the present disclosure is applicable to any type of neural network suitable for processing media content (e.g., audio and / or video content).

[0137] Back to Figure 3 The example of a generator shown in , can facilitate the training of the generator in a generative adversarial network setting based on true-false interactions with a discriminator. In order to determine at least one set of parameters for updating such a neural network, the inventors have found that the at least one set of parameters can be determined based on a generative adversarial network setting including a pre-trained generator and an untrained discriminator. In this setting, the generator can be incrementally updated with very limited (differently configured) training data and time, thereby improving the performance of its new (updated) configuration. Using this approach, all weights in the generator can be incrementally updated. However, for a neural network operating on a new domain, the most important layers may be signal-oriented (e.g., media data-oriented) layers. Therefore, in order to limit the update parameters to only a few weights (and thereby reduce the bitrate burden), all weights of the pre-trained generator can be frozen, and only the media data (signal)-oriented layers and / or output layers of the generator can be updated.

[0138] explain

[0139] Unless otherwise specifically noted, as will be apparent from the following discussion, it should be understood that throughout this disclosure, discussions utilizing terms such as "process," "calculate," "calculate," "determine," "analyze," etc. refer to the actions and / or processes by which a computer or computing system or similar electronic computing device manipulates and / or transforms data represented as physical (e.g., electronic) quantities into other data similarly represented as physical quantities.

[0140] In a similar manner, the term "processor" may refer to any device or portion of a device that processes electronic data, for example, from registers and / or memory, to transform that electronic data into other electronic data, for example, which may be stored in registers and / or memory. A "computer" or "computing machine" or "computing platform" may include one or more processors.

[0141] In one example embodiment, the methods described herein may be performed by one or more processors that accept computer-readable (also referred to as machine-readable) code comprising a set of instructions that, when executed by one or more of the processors, implement at least one of the methods described herein. Any processor capable of executing (sequentially or otherwise) a set of instructions specifying an action to be taken is encompassed. Thus, an example is a typical processing system comprising one or more processors. Each processor may comprise one or more of a CPU, a graphics processing unit, and a programmable DSP unit. The processing system may also include a memory subsystem comprising main RAM and / or static RAM and / or ROM. A bus subsystem may be included for communication between components. The processing system may also be a distributed processing system having processors coupled via a network. If the processing system requires a display, such a display may be included, such as a liquid crystal display (LCD) or a cathode ray tube (CRT) display. If manual data entry is required, the processing system may also include an input device, such as one or more of an alphanumeric input unit such as a keyboard, a pointing control device such as a mouse, and the like. The processing system may also include a storage system such as a disk drive unit. In some configurations, the processing system may include an audio output device and a network interface device. Thus, the memory subsystem includes a computer-readable carrier medium carrying computer-readable code (e.g., software), which includes a set of instructions to cause one or more of the methods described herein to be performed when executed by one or more processors. Please note that when a method includes several elements (e.g., several steps), the ordering of these elements is not implied unless otherwise specified. The software can reside on a hard disk, or it can reside completely or at least partially in RAM and / or a processor during execution by the computer system. Thus, the memory and the processor also constitute a computer-readable carrier medium carrying the computer-readable code. In addition, the computer-readable carrier medium can form or be included in a computer program product.

[0142] In alternative example embodiments, the one or more processors operate as standalone devices or may be connected (e.g., networked) to other processors in a networked deployment, the one or more processors may operate as a server or user machine in a server-user network environment, or as a peer machine in a peer-to-peer or distributed network environment. The one or more processors may form a personal computer (PC), a tablet computer, a personal digital assistant (PDA), a cellular phone, a network appliance, a network router, a switch or bridge, or any machine capable of executing (sequentially or otherwise) a set of instructions that specify actions to be taken by the machine.

[0143] Note that the term "machine" shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

[0144] Thus, an example embodiment of each method described herein is in the form of a computer-readable carrier medium carrying a set of instructions, for example, a computer program for execution on one or more processors (e.g., one or more processors as part of a network server arrangement). Thus, as will be appreciated by those skilled in the art, example embodiments of the present disclosure may be embodied as a method, a device such as a dedicated device, a device such as a data processing system, or a computer-readable carrier medium (e.g., a computer program product). A computer-readable carrier medium carries a computer-readable code comprising a set of instructions that, when executed on one or more processors, causes the one or more processors to implement a method. Thus, aspects of the present disclosure may take the form of a method, an example embodiment of a fully hardware embodiment, an example embodiment of a fully software embodiment, or an example embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a carrier medium (e.g., a computer program product on a computer-readable storage medium) carrying a computer-readable program code embodied in a medium.

[0145] The software may also be transmitted or received over a network via a network interface device. Although the carrier medium is a single medium in the example embodiments, the term "carrier medium" should be understood to include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) that store the one or more sets of instructions. The term "carrier medium" should also be understood to include any medium that can store, encode, or carry a set of instructions for execution by one or more of the processors and cause the one or more processors to perform any one or more methods of the present disclosure. The carrier medium may take a variety of forms, including but not limited to non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical disks, magnetic disks, and magneto-optical disks. Volatile media include dynamic memory, such as main memory. Transmission media include coaxial cables, copper wires, and optical fibers, including wires that constitute a bus subsystem. Transmission media may also take the form of sound waves or light waves, such as those generated during radio wave and infrared data communications. For example, the term "carrier medium" is accordingly understood to include, but is not limited to: solid-state memory, computer products embodied in optical and magnetic media; media carrying propagated signals detectable by at least one processor or one or more processors and representing a set of instructions that, when executed, implement a method; and transmission media in a network carrying propagated signals detectable by at least one of the one or more processors and representing the set of instructions.

[0146] It will be appreciated that, in one example embodiment, the steps of the method discussed are performed by one (or more) suitable processors of a processing (e.g., computer) system executing instructions (computer-readable code) stored in a storage device. It will also be appreciated that the present disclosure is not limited to any particular implementation or programming technique, and that the present disclosure may be implemented using any suitable technique for implementing the functionality described herein. The present disclosure is not limited to any particular programming language or operating system.

[0147] References throughout this disclosure to "an example embodiment," "some example embodiments," or "example embodiments" mean that a particular feature, structure, or characteristic described in connection with the example embodiment is included in at least one example embodiment of the present disclosure. Thus, appearances of the phrases "in an example embodiment," "in some example embodiments," or "in an example embodiment" throughout this disclosure are not necessarily all referring to the same example embodiment. Furthermore, in one or more example embodiments, the particular features, structures, or characteristics may be combined in any suitable manner, as would be apparent to one of ordinary skill in the art from this disclosure.

[0148] As used herein, unless otherwise indicated, the use of ordinal adjectives "first," "second," "third," etc. to describe a common object merely indicates that different instances of similar objects are being referred to and is not intended to imply that the objects so described must be in a given order in time, space, order, or in any other manner.

[0149] In the following claims and the description herein, any of the terms "comprising", "consisting of" or "which comprises" is an open term, which means including at least the elements / features that follow, but not excluding other elements / features. Therefore, when used in the claims, the term "comprising" should not be interpreted as being limited to the means or elements or steps listed thereafter. For example, the scope of the expression of a device that includes A and B should not be limited to a device consisting of only elements A and B. Any of the terms "including" or "which comprises" used herein is also an open term, which also means including at least the elements / features that follow the term, but not excluding other elements / features. Therefore, "including" is synonymous with "comprising", and "including" means "comprising".

[0150] It should be understood that in the above description of example embodiments of the present disclosure, in order to clarify the present disclosure and aid in understanding one or more of the various inventive aspects, various features of the present disclosure are sometimes combined in a single example embodiment, figure, or description thereof. However, this method of disclosure should not be interpreted as reflecting an intention that the claims require more features than those expressly recited in each claim. On the contrary, as reflected in the claims below, the inventive aspects lie in less than all the features of a single aforementioned disclosed example embodiment. Therefore, the claims following the specification are hereby expressly incorporated into this specification, with each claim standing on its own as a separate example embodiment of the present disclosure.

[0151] Furthermore, while some example embodiments described herein include some features of other example embodiments but not others, combinations of features from different example embodiments are intended to be within the scope of this disclosure and to form different example embodiments, as will be understood by those skilled in the art. For example, in the claims below, any of the claimed example embodiments may be used in any combination.

[0152] In the description provided herein, numerous specific details are set forth. However, it should be understood that the exemplary embodiments of the present disclosure may be practiced without these specific details. In other cases, well-known methods, structures, and techniques are not shown in detail in order not to obscure the understanding of this specification.

[0153] Thus, while what is believed to be the best mode of the present disclosure has been described, those skilled in the art will recognize that other and further modifications may be made thereto without departing from the spirit of the present disclosure, and it is intended that all such variations and modifications be claimed as falling within the scope of the present disclosure. For example, any formulas given above are merely representative of procedures that may be used. Functions may be added or deleted from the block diagrams, and operations may be interchanged between functional blocks. Steps may be added or deleted from the methods described within the scope of the present disclosure.

[0154] Various aspects of the present invention may be understood from the following enumerated example embodiments (EEE):

[0155] 1. A method for generating a media bitstream for transmitting parameters for updating a neural network implemented in a decoder, wherein the method comprises the following steps:

[0156] (a) determining at least one set of parameters for updating a neural network;

[0157] (b) encoding the at least one set of parameters and the media data to generate a media bitstream; and

[0158] (c) transmitting the media bitstream to the decoder for updating the neural network using the at least one set of parameters.

[0159] 2. The method according to EEE 1, wherein the media data comprises one or more of audio data and / or video data.

[0160] 3. The method according to EEE 1 or EEE 2, wherein the at least one set of parameters is encoded based on a set of syntax elements.

[0161] 4. The method of EEE 3, wherein in step (a) two or more sets of parameters for updating the neural network are determined, and wherein the set of syntax elements includes one or more syntax elements identifying a respective set of parameters for a respective update of the neural network to be performed.

[0162] 5. The method of any one of EEEs 1-4, wherein a neural network implemented in a decoder is used for processing media data, and wherein, in a media bitstream, the at least one set of parameters used to update the neural network is time-aligned with the media data processed by the neural network.

[0163] 6. The method according to EEE 5, wherein the at least one set of parameters is determined based on one or more of a codec mode, content of the media data, and coding constraints.

[0164] 7. The method according to EEE 6, wherein the codec mode comprises one or more of bit rate, video and / or audio frame rate, and used core codec.

[0165] 8. The method according to EEE 6 or EEE 7, wherein the content of the media data includes one or more of speech, music, and applause.

[0166] 9. The method according to any one of EEEs 6-8, wherein the coding constraints include one or more of constraints on performance scalability and constraints on adaptive processing.

[0167] 10. The method according to any one of EEEs 1-9, wherein the media data is in MPEG-H Audio format or MPEG-I Audio format, and the format media bitstream is a packetized media bitstream in MHAS format.

[0168] 11. The method according to EEE 10, wherein the at least one set of parameters is encoded by encapsulating the at least one set of parameters into one or more MHAS packets of a new MHAS packet type.

[0169] 12. The method according to any one of EEEs 1-11, wherein the media data is in AC-4, AC-3 or EAC-3 format.

[0170] 13. The method according to EEE 12, wherein the at least one set of parameters is encoded as one or more payload elements in the media bitstream.

[0171] 14. The method according to any one of EEEs 1-13, wherein the media data is in MPEG-4 or MPEG-D USAC format.

[0172] 15. The method according to EEE 14, wherein the at least one set of parameters is encoded in the media bitstream as one or more payload elements or one or more data stream elements.

[0173] 16. The method according to any one of EEEs 1-15, wherein the neural network has a layer structure and the at least one set of parameters includes parameters for updating weights of at least a media data-oriented layer and / or an output layer of the neural network.

[0174] 17. A method for updating a neural network implemented in a decoder, the method comprising the steps of:

[0175] (a) receiving an encoded media bitstream comprising media data and at least one set of parameters for updating a neural network;

[0176] (b) decoding the received media bitstream to obtain decoded media data and the at least one set of parameters for updating the neural network; and

[0177] (c) Updating the neural network by the decoder using the at least one set of parameters.

[0178] 18. The method according to EEE 17, wherein the media data comprises one or more of audio data and / or video data.

[0179] 19. The method according to EEE 17 or EE 18, wherein the method further comprises receiving a set of syntax elements, and wherein the decoder updates the neural network based on the received set of syntax elements using the at least one set of parameters.

[0180] 20. The method of claim 19, wherein in step (a), two or more sets of parameters for updating the neural network are included in the received coded media bitstream, and wherein the received set of syntax elements includes one or more syntax elements identifying a respective set of parameters for a respective update of the neural network to be performed.

[0181] 21. The method of any one of EEEs 17-20, wherein a neural network is used for processing of media data, and wherein, in the received coded media bitstream, the at least one set of parameters is time-aligned with the media data processed by the neural network.

[0182] 22. The method according to any one of EEEs 17-21, wherein the neural network has a layer structure, and wherein weights of at least a media data-oriented layer and / or an output layer of the neural network are updated by the decoder using the at least one set of parameters.

[0183] 23. An apparatus for generating a media bitstream for transmitting parameters for updating a neural network implemented in a decoder, wherein the apparatus comprises a processor configured to perform a method comprising the steps of:

[0184] (a) determining at least one set of parameters for updating a neural network;

[0185] (b) encoding the at least one set of parameters and the media data to generate a media bitstream; and

[0186] (c) transmitting the media bitstream to the decoder so as to update the neural network using the at least one set of parameters.

[0187] 24. An apparatus for updating a neural network implemented in a decoder, the apparatus comprising:

[0188] (a) a receiver configured to receive a coded media bitstream, wherein the coded media bitstream includes media data and at least one set of parameters for updating a neural network;

[0189] (b) a decoder for decoding the received media bitstream to obtain decoded media data and the at least one set of parameters for updating the neural network; and

[0190] (c) an updater for updating the neural network using the at least one set of parameters.

[0191] 25. The apparatus according to EEE 24, wherein the receiver further receives a set of syntax elements, and the updater updates the neural network using the at least one set of parameters based on the set of syntax elements.

[0192] 26. An apparatus according to EEE 25, wherein two or more sets of parameters for updating a neural network are included in a coded media bitstream received by a receiver, and wherein the received set of syntax elements includes one or more syntax elements that identify a respective set of parameters for a respective update of the neural network to be performed.

[0193] 27. A computer program product comprising a computer-readable storage medium having instructions adapted to, when executed by a device having processing capabilities, cause the device to implement the method according to any one of EEE 1-16.

[0194] 28. A computer program product comprising a computer-readable storage medium having instructions adapted to, when executed by a device having processing capabilities, cause the device to implement the method according to any one of EEE 17-22.

Claims

1. A method for generating a media bitstream for transmitting parameters of a neural network implemented in a decoder, wherein the neural network has a plurality of layers, wherein a media data-oriented layer is the first layer of the plurality of layers and an output layer is the last layer of the plurality of layers, wherein the method comprises the following steps: (a) determining at least one set of parameters for updating weights of the plurality of layers of the neural network, the at least one set of parameters including parameters for updating weights of the media data-oriented layer and / or the output layer; (b) generating the media bitstream by encoding only parameters for updating weights of the media data-oriented layer and / or the output layer among the at least one set of parameters for updating weights of the plurality of layers of the neural network, and media data, the media data comprising one or more of audio data and / or video data; and (c) transmitting the media bitstream to the decoder so as to update the neural network using parameters included in the media bitstream, thereby updating the weights of the media data-oriented layer and / or the output layer. 2 . The method of claim 1 , wherein the at least one set of parameters is encoded based on a set of syntax elements.

3. The method of claim 2 , wherein in step (a), two or more sets of parameters for updating the neural network are determined, and wherein the set of syntax elements includes one or more syntax elements identifying a respective set of parameters for a respective update of the neural network to be performed.

4. The method of any one of claims 1-3, wherein the neural network implemented in the decoder is used for processing of media data, and wherein in the media bitstream, the at least one set of parameters used to update the neural network is time-aligned with the media data processed by the neural network. 5 . The method of claim 4 , wherein the at least one set of parameters is determined based on one or more of a codec mode, content of the media data, and coding constraints.

6. The method of claim 5, wherein the codec mode comprises one or more of a bit rate, a video and / or audio frame rate, and a core codec used. 7 . The method according to claim 5 , wherein the content of the media data comprises one or more of voice, music, and applause.

8. The method of claim 5 or claim 6, wherein the encoding constraints include one or more of constraints on performance scalability and constraints on adaptive processing.

9. The method of claim 5 or claim 6, wherein the at least one set of parameters is included in the media bitstream prior to the media data to be processed by the corresponding updated neural network.

10. The method according to any one of claims 1 to 3, wherein the media data is in MPEG-H Audio format or MPEG-I Audio format, and the media bitstream is a packetized media bitstream in MHAS format.

11. The method of claim 10, wherein the at least one set of parameters is encoded by encapsulating the at least one set of parameters into one or more MHAS packets of a new MHAS packet type.

12. The method according to any one of claims 1 to 3, wherein the media data is in AC-4, AC-3, EAC-3 format, MPEG-4 or MPEG-D USAC format.

13. The method of claim 12, wherein the at least one set of parameters is encoded in the media bitstream as one or more payload elements.

14. The method of claim 13, wherein the at least one set of parameters is encoded in the media bitstream as one or more data stream elements.

15. The method according to any one of claims 1 to 3, wherein the at least one set of parameters comprises an identifier identifying whether the parameter used to update the weight represents a relative value or an absolute value.

16. A method for updating a neural network implemented in a decoder, the neural network having a plurality of layers, wherein a media data-oriented layer is a first layer of the plurality of layers and an output layer is a last layer of the plurality of layers, the method comprising the following steps: (a) receiving a coded media bitstream, wherein the coded media bitstream includes media data and parameters for updating weights of the media data-oriented layer and / or the output layer of the neural network; (b) decoding the received media bitstream to obtain decoded media data and parameters for updating weights of the media data-oriented layer and / or the output layer of the neural network; and (c) Updating the media data oriented layer and / or the output layer of the neural network using the received parameters included in the media bitstream for updating the weights of the media data oriented layer and / or the output layer by the decoder.

17. An apparatus for generating a media bitstream for transmitting parameters for updating a neural network implemented in a decoder, the neural network having a plurality of layers, wherein a media data-oriented layer is a first layer of the plurality of layers and an output layer is a last layer of the plurality of layers, wherein the apparatus comprises a processor configured to perform a method comprising the steps of: (a) determining at least one set of parameters for updating weights of the plurality of layers of the neural network, the at least one set of parameters including parameters for updating weights of the media data-oriented layer and / or the output layer; (b) generating the media bitstream by encoding only parameters for updating weights of the media data-oriented layer and / or the output layer among the at least one set of parameters for updating weights of the plurality of layers of the neural network, and media data, the media data comprising one or more of audio data and / or video data; and (c) transmitting the media bitstream to the decoder so as to update the neural network using the parameters included in the media bitstream, thereby updating the weights of the media data-oriented layer and / or the output layer.

18. An apparatus for updating a neural network implemented in a decoder, the neural network having a plurality of layers, wherein a media data-oriented layer is a first layer of the plurality of layers and an output layer is a last layer of the plurality of layers, the apparatus comprising: (a) a receiver configured to receive a coded media bitstream, wherein the coded media bitstream includes media data and parameters for updating weights of the media data-oriented layer and / or the output layer of the neural network; (b) a decoder configured to decode the received media bitstream to obtain decoded media data and parameters for updating weights of the media data-oriented layer and / or the output layer of the neural network; and (c) an updater configured to update the weights of the media data-oriented layer and / or the output layer of the neural network using the received parameters for updating the weights of the media data-oriented layer and / or the output layer included in the media bitstream.

19. A computer program product comprising a computer-readable storage medium having instructions adapted, when executed by a device having processing capabilities, to cause the device to carry out the method according to any one of claims 1 to 15.

Images