Apparatus, system, codec and method for neural coding

Abstract

A hybrid codec for training a neural coding system, the hybrid codec comprising: a real codec for encoding data pre-processed by a first neural network and for decoding the encoded data for post-processing by a second neural network; and a proxy codec being a differentiable representation of the real codec and configured to back propagate a loss function representing the coding bitrate and coding distortion between the post-processed data and the original data, for training weights of at least the first neural network.

Description

Technical Field

[0001] The present invention relates to devices, systems, codecs and methods for neural coding. Background Art

[0002] The neural network can be used independently to pre-process and / or post-process data for streaming to improve the efficiency or quality of the streamed data / content. For example, a post-processing neural network can be trained to remove compression or streaming artifacts, and / or perform operations such as super-resolution imaging to increase the output resolution of the streamed content.

[0003] At the same time, a pre-processing neural network can, for example, be used to remove higher frequencies from the data / content before it is encoded, which can achieve more efficient encoding with fewer compression artifacts.

[0004] However, these pre-processing and post-processing are independent of each other and can therefore be enabled / disabled independently. If the pre-processing neural network and the post-processing neural network are trained together, the pre-processing neural network and the post-processing neural network can be more efficient, and in some cases, it may be possible to use the pre-processing neural network to convert data from a type not supported by the encoder / decoder to a data type supported by the encoder / decoder, and after encoding / decoding, reconstruct the data into a type not supported by the encoder / decoder by using the post-processing neural network.

[0005] It is under such circumstances that the present disclosure was produced. Summary of the invention

[0006] In a first aspect, there is provided in claim 1 a hybrid codec.

[0007] In another aspect, a neural decoding system for streaming is provided in claim 2.

[0008] In another aspect, a method for streaming using neural coding is provided in claim 11.

[0009] In another aspect, a method for training a neural network for neural decoding is provided in claim 12.

[0010] Further individual aspects and features of the invention are defined in the dependent claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Embodiments of the present invention will now be described by way of example with reference to the accompanying drawings, in which:

[0012] - Figure 1 An example neural decoding system is schematically illustrated;

[0013] - Figure 2 Another example neural decoding system is schematically illustrated;

[0014] - Figure 3 schematically illustrates an example method for streaming using neural coding;

[0015] - Figure 4 An example method for training a neural network is schematically illustrated;

[0016] - Figure 5 An example management device for a neural decoding streaming system is schematically shown;

[0017] - Figure 6 Schematically illustrates an example neural decoding transmission device 600 for streaming;

[0018] - Figure 7 schematically illustrates an example neural decoding receiving device 700 for receiving streaming data; and

[0019] - Figure 8 An example method 800 for managing a neural decoding system is schematically illustrated. DETAILED DESCRIPTION

[0020] In the following description, many specific details are presented in order to provide a thorough understanding of embodiments of the present invention. However, it will be apparent to those skilled in the art that these specific details do not need to be adopted to practice the present invention. On the contrary, for the sake of clarity, specific details known to those skilled in the art have been omitted where appropriate.

[0021] For streaming data (such as video content, audio content, and / or video game content), the encoder / decoder typically uses a codec (codec) (such as HEVC, JPEG, FLAC, H.264, etc.) to encode / decode the streaming data for transmission. Neural decoding is a technology that can improve the streaming of data by optimizing the streaming of data using a neural network (NN). In some cases, a neural network can be trained to perform the entire decoding process. However, since many frequently used decoding schemes are implemented in hardware circuits, it may be more efficient to use NN to pre-process and post-process data that is decoded using a standard codec. For example, the input of the encoder can be processed by a pre-processing neural network, and the output of the decoder can be processed by a post-processing neural network.

[0022] In addition, the pre-processing NN and post-processing NN can be more efficient when they are trained together than when they are trained independently of each other because the pre-processing NN and post-processing NN can learn to pass additional information within the encoded information from the pre-processing NN to the post-processing NN, and this learned information can allow more efficient encoding / decoding and can enable information that is normally in a format not supported by a given codec to be encoded by the codec.

[0023] Reference now Figure 1 , shows an example of a neural decoding system 100 for streaming according to an embodiment of the present disclosure. In this example, streaming data 102 is data to be streamed by the neural decoding system 100. The streaming data 102 may be one or more of video data, audio data, video game data, and / or any other type of data suitable for streaming.

[0024] We will now describe Figure 1 The basic operation of the neural decoding system 100 shown. The stream data 102 is input into the preprocessing unit 110, which is configured to preprocess the stream data 102 using a first neural network trained to preprocess the stream data to obtain preprocessed data. The preprocessed data is in a data format supported by the codec used by the encoder 120 and the decoder 130.

[0025] The pre-processed data is input into the encoder 120, which is configured to encode the pre-processed data using / according to a codec. The codec used by the encoder 120 may be predetermined or may be selected by, for example, a user of the streaming system. It should be noted that the techniques of the present invention are applicable to lossy or lossless codecs.

[0026] The encoded pre-processed data stream may then be transmitted to the decoder 130 via the network. The decoder 130 is configured to decode the encoded pre-processed data using / according to a codec (which may be the same or complementary to the codec used by the encoder 120) to obtain decoded pre-processed data. It should be noted that the decoded pre-processed data may be different from the pre-processed data before encoding (e.g., in the case of using a lossy decoding scheme). However, in some cases (such as when a lossless decoding scheme is used), the decoded pre-processed data may be the same as the pre-processed data before encoding.

[0027] The decoded pre-processed data may be input to a post-processing unit 140, which is configured to post-process the decoded pre-processed data using a second neural network trained to post-process the decoded pre-processed data to obtain a reconstruction of the stream data 142. The reconstruction of the stream data 142 may then be output by a suitable output device (e.g., a display screen, a head mounted display, an audio output system, etc.).

[0028] Figure 2 Another example of a neural decoding system 200 for streaming is shown. It should be noted that Figure 1 Features discussed Figure 2 Features shown in the drawings share reference numerals, and repeated discussion of these features will be omitted.

[0029] In this example, the encoder 120 and the decoder 130 perform encoding / decoding using a hybrid codec 210. The hybrid codec 210 includes a real codec 212 (ie, any standard codec that can be used for encoding / decoding as discussed elsewhere herein);

[0030] A reconstruction of the stream data 142 that has been encoded using the real codec 212 and then decoded cannot be back-propagated using the real codec 212 to further train the neural network 116, 146 because the real codec 212 is not differentiable (i.e., derivatives cannot be propagated using the real codec 212), which is essential for back-propagation.

[0031] Therefore, in some embodiments of the present disclosure, a proxy codec 214 is provided. The proxy codec 214 is a differentiable model of the real codec 212. In particular, the proxy codec 214 represents the decoding bit rate (ie, compression rate) of the real codec 212 and the decoding distortion of the real codec 212.

[0032] The decoding distortion can be modeled by replacing any non-differentiable step size of the real codec 212 with a differentiable mathematical approximation of the non-differentiable step size. For example, for the JPEG codec, the quantization step size is non-differentiable because it includes a non-differentiable rounding function. However, the rounding function can be replaced by using a differentiable mathematical approximation (such as a polynomial approximation, Fourier series, Taylor series, etc.). Therefore, the modified JPEG codec in which the rounding function of the quantization step size has been replaced by a differentiable mathematical approximation of the rounding function is an example of a proxy codec that models the decoding distortion of the real codec 212 (in this example, the JPEG codec).

[0033] The proxy codec 214 of the present disclosure also represents the decoding bit rate of the real codec 212. The proxy codec 214 can model the decoding bit rate of the real codec 212 by using, for example, the differentiable mathematical approximation techniques discussed above. As another example, the proxy codec 214 can model the decoding bit rate of the real codec 212 by using a machine learning model trained to predict the decoding bit rate of the real codec 212.

[0034] Therefore, in an embodiment of the present disclosure, a hybrid codec 210 for training a neural decoding system 100, 200 is provided. The hybrid codec 210 includes: a real codec 212 for encoding data preprocessed by the first neural network 116, and for decoding the encoded data for post-processing by the second neural network 146; and a proxy codec 214, which is a differentiable representation of the real codec 212 and is used to back-propagate a loss function, the proxy codec 214 representing the decoding bit rate of the real codec 212 and the decoding distortion of the real codec 212 (i.e., the difference between the preprocessed data and the decoded preprocessed data), for training at least the weights 114 of the first neural network 116.

[0035] Thus, a hybrid codec 210 is provided for use with the neural decoding system 100, 200, wherein the real codec 212 can be used in the forward pass in training and runtime inference, and the proxy codec 214 can be used only in the backward pass. Thus, using the hybrid codec 210 of the present invention can advantageously improve the reliability of the training of the neural network, because the loss function for training the neural network is based on the reconstruction of the streaming data from the decoding using the real codec 212, which can remove potential error sources.

[0036] Additionally, because differentiability is achieved in the backward pass due to the use of the proxy codec 214, the derivatives of the loss function can be backpropagated using the proxy codec 214, which can advantageously avoid any problems that may be caused by differences between the proxy codec 214 and the real codec 212, compared to the case where the proxy codec is also used in the forward pass in training and the real codec is only used during runtime inference.

[0037] In addition, in an embodiment of the present disclosure, a neural decoding system 100, 200 for streaming is provided. The neural decoding system 100, 200 includes: a preprocessing unit 110, configured to preprocess the streaming data 102 using a first neural network 116 trained to preprocess the streaming data to obtain preprocessed data; a hybrid codec 210 as described elsewhere herein; an encoding unit 120, configured to encode the preprocessed data using a real codec 212; a decoding unit 130, configured to decode the encoded preprocessed data using the real codec 212; and a post-processing unit 140, configured to post-process the decoded preprocessed data using a second neural network 146 trained to post-process the decoded preprocessed data to obtain reconstruction of the streaming data.

[0038] In some embodiments of the present disclosure, the neural decoding system 100, 200 may include a loss calculation unit 220, which is configured to calculate a loss function, which represents the decoding bit rate of the streaming data 102 and the decoding distortion between the streaming data 102 and the reconstruction of the streaming data 142; and a back-propagation unit 230, which is configured to further train the weights 114 of the first neural network 116 by back-propagating the loss function using a proxy codec 214 that is a differentiable representation of the real codec 212.

[0039] Back propagation unit 230 may use any suitable back propagation technique, such as gradient descent. Because proxy codec 214 is differentiable, derivatives of a loss function computed using the reconstruction of streaming data 142 may be back propagated using proxy codec 214.

[0040] In some embodiments of the present disclosure, back propagation unit 230 may be configured to further train weights 144 of second neural network 146 by back propagating the loss function via second neural network 146 before back propagating the loss function using proxy codec 214. Back propagation unit 230 may be configured to further train weights 144 of second neural network 146 by back propagating the loss function in a manner similar to further training of the first neural network as described elsewhere herein.

[0041] return Figure 1 In some embodiments of the present disclosure, the neural decoding system 100 , 200 may include a training repository unit 150 configured to store the streaming data 102 to further train the first neural network 116 and the second neural network 146 .

[0042] Optionally, in these embodiments of the present disclosure, training repository unit 150 may be configured to store reconstructions of streaming data 142 in association with streaming data 102 to further train first neural network 116 and second neural network 146 .

[0043] The training repository unit 150 may advantageously enable generation of larger training data sets, since the neural decoding system 100 , 200 may be trained using data from runtime inference (rather than only at the training stage) due to the use of the hybrid codec 210 .

[0044] Further optionally, in these embodiments of the present disclosure, the back propagation unit 230 may be configured to further train the weights 114, 144 when the utilization level of the neural decoding system 100, 200 is below a predetermined threshold. For example, if the neural decoding system 100, 200 is implemented in a video game streaming system, the back propagation unit 230 may further train the weights 114, 144 when the utilization of the video game streaming system is below a predetermined threshold, such as during off-peak hours. Therefore, the neural decoding system 100, 200 can advantageously perform further training by utilizing existing hardware without affecting the performance of the neural decoding system 100, 200.

[0045] Optionally, in embodiments of the present disclosure where the neural decoding system 100, 200 includes a training repository unit 150, the neural decoding system 100, 200 may include a filter unit 160 configured to filter the data 102, 142 to be stored in the training repository unit 150 using a third neural network trained to identify data that is most likely to improve the first neural network 116 and the second neural network 146 when further trained.

[0046] In some cases, the first neural network 116 may be trained to output a signal to the filter unit 160 in response to the first neural network 116 determining that the data 102 to be streamed may be outside the range of weights used by the first neural network 116. In these cases, the third neural network may use the signal as another input to evaluate whether the data 102, 142 should be stored in the training repository 150. Alternatively, the filter unit 160 may bypass the third neural network and store the data 102, 142 in the training repository 150 in response to receiving the signal from the first neural network 116.

[0047] Reference now Figure 3 In some embodiments of the present disclosure, a method 300 for streaming using neural decoding using a hybrid codec 210 is provided. The method includes the following steps: preprocessing 310 the streaming data using a first neural network trained to preprocess the streaming data to obtain preprocessed data; encoding 320 the preprocessed data using a real codec 212; decoding 330 the encoded preprocessed data using the real codec 212; and post-processing 340 the decoded preprocessed data using a second neural network trained to post-process the decoded preprocessed data to obtain a reconstruction of the streaming data.

[0048] Reference now Figure 4In some embodiments of the present disclosure, a method 400 for training a first neural network according to the method 300 for streaming is provided. The method 400 for training comprises the following steps: calculating 410 a loss function representing a decoding bit rate of streaming data encoded according to the method 300 for streaming and a decoding distortion between the streaming data and a reconstruction of the streaming data obtained according to the method 300 for streaming; and further training 420 the weights of the first neural network by back-propagating the loss function using the proxy codec 214.

[0049] Reference now Figure 5 In some embodiments of the present disclosure, a management device 500 for a neural decoding stream transmission system (such as the neural decoding stream transmission system 100 or 200) is provided. The management device 500 includes a receiving unit 510, a storage unit 520, an identification unit 530, and a transmission unit 540.

[0050] Before explaining the components of the management device 500 in more detail, it should be noted that the management device 500 of the present disclosure can be used with a neural decoding system using a hybrid codec 210 as described elsewhere herein, but is not limited to use with a neural decoding system using a hybrid codec 210 as described elsewhere herein.

[0051] The receiving unit 510 is configured to receive one or more tags indicating the type of streaming data to be streamed by the neural decoding system. For example, the tag may indicate the type of data to be streamed, such as audio data, video data, or video game data. Additionally or alternatively, the tag may indicate the content to be streamed. For example, in the case of audio data, the tag may indicate the genre of music, that the audio data is dialogue, or a specific artist of the music, a specific song or portion of a song, and / or any other suitable indication of the content of the audio data. As another example, in the case of video game data to be streamed, the tag may indicate a specific game, a specific portion of a video game, or any other suitable indication of the content of the video game data. In some cases, one or more tags may be received for each buffer of data to be streamed by the neural decoding system.

[0052] It should be noted that in some cases, the data to be streamed may be a combination of different types of data, each of which may be indicated by one or more tags. For example, video game data may be streamed together with audio data.

[0053] The storage unit 520 is configured to store a plurality of sets of weights for a neural network of a neural decoding system. The corresponding weight sets are associated with at least one tag indicating a type of streaming data.

[0054] The identification unit 530 is configured to identify a set of weights from the storage unit 520 according to the received label. For example, if the receiving unit 510 receives a set of labels "A", "B", "C", and "D" indicating the type of streaming data to be streamed by the neural decoding system, and the storage unit 520 has a set of weights stored in association with the labels "A", "B", "C", and "D", the identification unit 530 can identify the set of weights. It should be noted that for this purpose, only 'A', 'B', 'C', and 'D' are given as labels, and later are examples, and should not be interpreted as a limitation on the format or content of the labels disclosed herein.

[0055] However, an exact match is not required. For example, if an exact match does not exist, the identification unit 530 can identify a set of weights stored in the storage unit 520 that are associated with the most tags shared by the received tags. For example, referring to the tags received in the above example, if the storage unit 520 stores a first set of weights associated with tags "A" and "B", and a second set of weights associated with tags "B", "C", and "D", the identification unit 530 can identify the second set of weights because it is a set of weights associated with the most tags shared by the received tags.

[0056] In addition, some tags may be given a higher priority than other tags. For example, tag 'A' may have a higher priority than other tags. In this case, referring to the example in the previous paragraph, the identification unit 530 may identify a first weight set based on the received tags, even if the second tag set has more common tags with the received tags, because the first weight set (and the second weight set is not) is associated with the 'A' tag having a higher priority than the other tags.

[0057] In some cases, such as when there are only limited matches or even no matches, identification unit 530 may identify a set of default weights. There may be more than one set of default weights, and a particular default set may be selected based on one or more of the received tags in a manner similar to that described above.

[0058] The transmission unit 540 is configured to transmit first weight data representing a portion of the identified set of weights to the neural decoding system for a first neural network of the neural decoding system. The transmission unit 540 is also configured to transmit second weight data representing another portion of the identified set of weights to the neural decoding system for a second neural network of the neural decoding system.

[0059] The management device 500 of the present disclosure can advantageously enable more efficient streaming that can be customized for given streaming data while still utilizing efficient codecs that are typically implemented via specific hardware circuits because the customization is performed by the specific weights of the neural network sent by the management device 500.

[0060] In some cases, at least one of the set of weights stored by storage unit 520 may have been generated by training the first neural network and the second neural network independently of each other, as described elsewhere herein. Thus, a first portion of the identified set of weights represented by the first weight data and a second portion of the identified set of weights represented by the second weight data may have been generated by training the first neural network and the second neural network independently of each other.

[0061] In this case, at least one of the tags may indicate that the streaming data should be streamed using a pre-processing network and a post-processing network that have been trained independently of each other. Alternatively or additionally, when there is a limited match or even no match between the received tags and the tags associated with the weight sets stored by the storage unit 520 as described elsewhere herein, one of the default weight sets described elsewhere herein may be a weight set including a first portion of weights and a second portion of weights generated by training the first neural network and the second neural network independently of each other.

[0062] Alternatively or additionally, at least one of the set of weights stored by the storage unit 520 may be generated by training the first neural network and the second neural network together, as described elsewhere herein. Thus, a first portion of the identified set of weights represented by the first weight data and a second portion of the identified set of weights represented by the second weight data may have been generated by training the first neural network and the second neural network together.

[0063] In this case, at least one of the tags may indicate that the streaming data should be streamed using the pre-processing network and the post-processing network that have been trained together. Alternatively or additionally, when there is a limited match or even no match between the received tags and the tags associated with the weight sets stored by the storage unit 520 as described elsewhere herein, one of the default weight sets described elsewhere herein may be a weight set including a first portion of weights and a second portion of weights generated by training the first neural network and the second neural network together.

[0064] In some embodiments of the present disclosure, the receiving unit 510 may be configured to receive one or more device attributes representing one or more attributes of a neural decoding receiving device of the neural decoding system. In these embodiments, the identifying unit 530 may be configured to identify the set of weights from the storage unit based on the received label and the one or more received device attributes.

[0065] For example, some types of neural decoding receiving devices may have more capabilities than other types. In the case where the data to be streamed is video game data, a decoding receiving device with certain capabilities (which may be indicated by device attributes) may participate in the rendering of the video game, which may result in different data to be streamed than a receiving device that does not participate in the rendering. Therefore, it may be more optimal to use a different set of weights based on the attributes of the receiving device. Therefore, the management device 500 may also customize the weights to be sent based on the attributes of the receiving device.

[0066] As another example, if the receiving device has processing power below a predetermined threshold, it may be more efficient to use a smaller neural network for the receiving device.Thus, identification unit 540 may identify a lighter set of weights for at least the second neural network and optionally also for the first neural network.

[0067] In some embodiments of the present disclosure, the corresponding weight data may indicate the difference between the weights previously transmitted for the corresponding neural network and the weights in the corresponding portion of the identified set of weights. In these embodiments, the transmission unit 540 may therefore transmit changes to the weights of the neural network rather than transmitting an entirely new set of weights. This may advantageously reduce the required bandwidth, particularly in the case of frequent updates to the weights.

[0068] In some embodiments of the present disclosure, the neural decoding system to which the weight data is transmitted may optionally store one or more sets of weights. In this case, the corresponding weight data may simply indicate which combination of the stored weight sets to use. In some cases, one or more of the weight sets stored by the neural decoding system may be predefined. Therefore, the management device 500 may be aware of the weight sets stored by the neural decoding system.

[0069] Alternatively or additionally, the one or more sets of weights stored by the neural decoding system may be one or more sets of weights that have been previously used by the neural decoding system (eg, they may have been stored by the neural decoding system from a previous streaming session).

[0070] In this case, one or more of the tags received by the management device 500 may indicate a set of weights stored by the neural decoding system. The identification unit 530 may, but need not, use these tags when identifying a set of weights. Alternatively or additionally, these tags may be used to identify the format of the corresponding weight data after the identification unit 530 has identified a set of weights.

[0071] For example, if the tags indicate that the identified set of weights has been stored by the neural decoding system, then the indication of the identified set of weights can be identified as the format of the corresponding weight data in this case. As another example, if the tags indicate that the identified set of weights is not stored by the neural decoding system, a portion of the set of weights; a difference between weights previously transmitted for the corresponding neural network and the weights in the corresponding portion of the identified set of weights; or a difference between a set of weights stored by the neural decoding system (which can be indicated by at least one of the tags) and the weights in the corresponding portion of the identified set of weights can be identified as the format of the corresponding weight data.

[0072] In some embodiments of the present disclosure, the storage unit 520 may be configured to store a plurality of quantization tables for a codec. The respective quantization tables may be associated with at least one tag indicating a stream data type, as described elsewhere herein. In these cases, the identification unit 530 may be configured to identify the quantization table from the storage unit 520 according to the received tag. The transmission unit 540 may be configured to transmit at least a portion of the identified quantization table to the neural decoding system for the encoding unit of the neural decoding system. The transmission unit 540 may be configured to transmit at least another portion of the identified quantization table to the neural decoding system for the decoding unit of the neural decoding system.

[0073] Thus, in addition to customized neural network weights, the management device 500 can enable the encoder / decoder of the neural coding system to use customized quantization tables for a given type of data to be streamed. This can further improve decoding performance.

[0074] Reference now Figure 6 In some embodiments of the present disclosure, a neural decoding transmission device 600 for stream transmission is provided. The neural decoding transmission device 600 includes a recognition unit 610; a transmission unit 620; a receiving unit 630; a weight modification unit 640; a preprocessing unit 650; and an encoding unit 660.

[0075] The identification unit 610 is configured to identify one or more tags indicating the type of streaming data to be streamed by the neural decoding transmission device. For example, the streaming data may be associated with metadata indicating the type of the streaming data, and the identification unit 610 may identify one or more tags based on the associated metadata. As another example, the identification unit 610 may alternatively or additionally be configured to identify the type of streaming data based on one or more attributes of the streaming data, and to identify one or more tags based on the identified attributes. Alternatively or additionally, any other technique suitable for the identification unit 610 to identify one or more tags may be used.

[0076] The transmission unit 620 is configured to transmit one or more identified tags to a management device (such as the management device 190 or 500 ).

[0077] The receiving unit 630 is configured to receive first weight data representing a weight of the first neural network from the management device.

[0078] The weight modification unit 640 is configured to modify the weight of the first neural network according to the first weight data received by the receiving unit. As described elsewhere herein, the weight data may indicate a change in the weight of the first neural network, or may indicate a set of weights for the first neural network. Thus, the weight modification unit 640 may modify the weight of the first neural network by updating the weight of the first neural network according to the weight data indicating a change in the weight of the first neural network, or may set the weight of the first neural network to the weight indicated by the weight data according to the weight data indicating a set of weights for the first neural network.

[0079] The preprocessing unit 650 is configured to preprocess the streaming data using a first neural network to obtain preprocessed data. The encoding unit 560 is configured to encode the preprocessed data using a real codec according to the techniques described elsewhere herein.

[0080] The transmission unit 620 is configured to transmit the encoded pre-processed data stream to a neural decoding receiving device for streaming reception.

[0081] Reference now Figure 7 In some embodiments of the present disclosure, a neural decoding receiving device 700 for receiving streaming data is provided. The neural decoding receiving device 700 includes a receiving unit 710; a weight modification unit 720 configured to modify the weight of the second neural network according to the second weight data received by the receiving unit; a decoding unit 730; and a post-processing unit 740.

[0082] The receiving unit 710 is configured to receive second weight data representing weights of a second neural network from a management device, such as the management device 190 or 500. The receiving unit 710 is also configured to receive encoded preprocessed streaming data from a neural decoding transmission device for streaming, such as the neural decoding transmission device 600.

[0083] The weight modification unit 720 is configured to modify the weight of the second neural network according to the second weight data received by the receiving unit 710 .

[0084] The decoding unit 730 is configured to decode the encoded preprocessed stream data using a real codec to obtain decoded preprocessed stream data. The post-processing unit 740 is configured to post-process the decoded preprocessed stream data using a second neural network to obtain a reconstruction of the stream data, as described elsewhere herein.

[0085] In some embodiments of the present disclosure, the neural decoding receiving device 700 may further include a transmission unit 750 configured to send one or more device properties of the neural decoding receiving device 700 to a management device according to the techniques described elsewhere herein.

[0086] In some embodiments of the present disclosure, a neural decoding system for streaming transmission is provided, which includes a management device 500 , a neural decoding transmission device 600 , and a neural decoding reception device 700 .

[0087] Optionally, in any embodiment of the neural decoding system described herein, the neural decoding system may further include a segmentation unit 170; and a synthesis unit 180, which Figure 1 An example neural decoding system 100 is shown.

[0088] The segmentation unit 170 may be configured to segment the stream data 102 into a plurality of stream data content components according to the content in the stream data before the stream data is preprocessed by the preprocessing unit 110. For example, if the stream data 102 is video game data, the content of the stream data 102 may include a graphical user interface (GUI) overlaid on the main content. In this example, the segmentation unit may segment the stream data 102 into a GUI stream data content component and a main content stream data content component.

[0089] The corresponding stream data content component can be streamed by the neural decoding system independently of other stream data content components to obtain a corresponding reconstruction of the corresponding stream data content component. Thus, referring to the above example, the GUI stream data content component and the main content stream data content component can be streamed by the neural decoding system independently of each other. In addition, one or more stream data content components can be scaled from one resolution to another resolution prior to preprocessing.

[0090] In these embodiments of the present disclosure, the synthesis unit 180 may be configured to combine multiple reconstructions of the corresponding stream data content components for output by the neural decoding system. Referring again to the above example, the reconstruction of the GUI stream data content component and the reconstruction of the main content stream data content component may be combined by the synthesis unit 180 before being output by the neural decoding system.

[0091] Optionally, in any embodiments of the neural decoding system described herein, at least a portion of the streaming data may be in a format not supported by the real codec. In these cases, a first neural network may be trained to convert the portion of the streaming data during pre-processing into latent data 118 having a format supported by the real codec. Additionally, a second neural network may be trained to convert the decoded latent data included in the decoded pre-processed data into a reconstruction of the portion of the streaming data in a format not supported by the real codec during post-processing.

[0092] Further optionally, in the case where the neural decoding system includes a management device (such as management device 190 or 500), at least one of the one or more tags received by the management device may indicate a portion of the streaming data in a format not supported by the real codec. Therefore, the weights provided to the neural network may be further customized to enable the neural network to convert the portion of the streaming data to / from the potential data 118.

[0093] Reference now Figure 8 , a method 800 for managing a neural decoding system is provided. The method includes the following steps: receiving 810 one or more tags indicating the type of streaming data to be streamed by the neural decoding system; storing 820 a plurality of weight sets of a neural network of the neural decoding system, wherein a respective weight set is associated with at least one tag indicating the type of streaming data; identifying 830 a set of weights from a storage unit according to the received tags; for a first neural network of the neural decoding system, sending 840 first weight data representing a portion of the identified set of weights to the neural decoding system; and for a second neural network of the neural decoding system, sending 850 second weight data representing another portion of the identified set of weights to the neural decoding system.

[0094] In some embodiments of the present disclosure, there is provided a non-transitory machine-readable storage medium storing a hybrid codec 210. The hybrid codec 210 has been described elsewhere herein.

[0095] In some embodiments of the present disclosure, a computer program is provided, which includes computer executable instructions suitable for causing a computer system to perform any method described elsewhere herein.

[0096] It should be understood that the above-described methods may be performed on conventional hardware appropriately adapted to the application by software instructions or by the inclusion or replacement of dedicated hardware.

[0097] Thus, the required adaptation of existing parts of conventional equivalent devices may be implemented in the form of a computer program product comprising processor-implementable instructions stored on a non-transitory machine-readable medium (such as a floppy disk, optical disk, hard disk, solid-state disk, PROM, RAM, flash memory, or any combination of these or other storage media), or implemented in hardware as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array) or other configurable circuit suitable for adapting conventional equivalent devices. Separately, such a computer program may be transmitted via a data signal on a network such as an Ethernet, a wireless network, the Internet, or any combination of these or other networks.

[0098] The foregoing discussion discloses and describes only exemplary embodiments of the present invention. As will be appreciated by those skilled in the art, the present invention may be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, the disclosure of the present invention is intended to be illustrative, rather than limiting, the scope of the present invention and other claims. The present disclosure, including any readily discernible variations taught herein, partially defines the scope of the appended claim terms so that no inventive subject matter is dedicated to the public.

[0099] Examples of this technology are defined by the following numbered clauses:

[0100] 1. A hybrid codec for training a neural decoding system, the hybrid codec comprising:

[0101] a real codec for encoding data pre-processed by the first neural network and for decoding the encoded data for post-processing by the second neural network; and

[0102] A proxy codec that is a differentiable representation of the true codec and is used to back-propagate a loss function that represents a decoding bit rate of the true codec and a decoding distortion between the post-processed data and the original data for training at least the weights of the first neural network.

[0103] 2. A neural decoding system for streaming, the neural decoding system comprising:

[0104] a preprocessing unit configured to preprocess the streaming data using a first neural network trained to preprocess the streaming data to obtain preprocessed data;

[0105] The hybrid codec of clause 1;

[0106] an encoding unit configured to encode the preprocessed data using the real codec;

[0107] a decoding unit configured to decode the encoded pre-processed data using the real codec; and

[0108] A post-processing unit is configured to post-process the decoded pre-processed data using a second neural network trained to post-process the decoded pre-processed data to obtain a reconstruction of the stream data.

[0109] 3. The neural decoding system according to clause 2, comprising:

[0110] a loss calculation unit configured to calculate a loss function, the loss function representing a decoding bit rate of the stream data and a decoding distortion between the stream data and a reconstruction of the stream data; and

[0111] A back-propagation unit is configured to further train the weights of the first neural network by back-propagating the loss function using the proxy codec as a differentiable representation of the real codec.

[0112] 4. A neural decoding system according to clause 3, wherein the back-propagation unit is configured to further train the weights of the second neural network by back-propagating the loss function through the second neural network before back-propagating the loss function using the proxy codec.

[0113] 5. A neural decoding system according to any one of clauses 2 to 4, comprising:

[0114] A training repository unit is configured to store the stream data to further train the first neural network and the second neural network.

[0115] 6. A neural decoding system according to clause 5, wherein the training repository unit is configured to store a reconstruction of the streaming data in association with the streaming data to further train the first neural network and the second neural network.

[0116] 7. A neural decoding system according to any one of clauses 5 and 6 as dependent upon any one of clauses 3 and 4, wherein the back-propagation unit is configured to further train the weights when the utilization level of the neural decoding system is below a predetermined threshold.

[0117] 8. A neural decoding system according to any one of clauses 5-7, further comprising:

[0118] A filter unit configured to filter the data to be stored in the training repository unit using a third neural network trained to identify the data most likely to improve the first neural network and the second neural network upon further training.

[0119] 9. A method for streaming using neural coding using the hybrid codec of clause 1, the method comprising the steps of:

[0120] preprocessing the stream data using a first neural network trained to preprocess the stream data to obtain preprocessed data;

[0121] encoding the preprocessed data using the real codec;

[0122] decoding the encoded pre-processed data using the real codec; and

[0123] The decoded pre-processed data is post-processed using a second neural network trained to post-process the decoded pre-processed data to obtain a reconstruction of the stream data.

[0124] 10. A method for training a first neural network according to clause 9, the method comprising the steps of:

[0125] calculating a loss function representing a decoding bit rate of the stream data encoded according to the method of clause 9 and a decoding distortion between the stream data and a reconstruction of the stream data obtained according to the method of clause 9; and

[0126] The weights of the first neural network are further trained by back-propagating the loss function using the proxy codec.

[0127] 11. A management device for a neural decoding stream transmission system, the management device comprising:

[0128] a receiving unit configured to receive one or more tags indicating a type of streaming data to be streamed by the neural decoding system;

[0129] a storage unit configured to store a plurality of weight sets of a neural network of the neural decoding system, wherein a corresponding weight set is associated with at least one tag indicating a type of streaming data;

[0130] an identification unit configured to identify a set of weights from the storage unit according to the received label; and

[0131] A transmission unit is configured to transmit first weight data representing a portion of the identified set of weights to the neural decoding system for a first neural network of the neural decoding system, and to transmit second weight data representing another portion of the identified set of weights to the neural decoding system for a second neural network of the neural decoding system.

[0132] 12. A management device according to clause 11, wherein

[0133] The receiving unit is configured to receive one or more device properties representing one or more properties of a neural decoding receiving device of the neural decoding system; and

[0134] The identification unit is configured to identify a set of weights from the storage unit based on the received tags on the one or more received device attributes.

[0135] 13. A management device according to any of clauses 11 and 12, wherein the respective weight data indicates a difference between weights previously transmitted for the respective neural network and weights in the respective part of the identified set of weights.

[0136] 14. A management device according to any one of clauses 11 to 13, wherein

[0137] The storage unit is configured to store a plurality of quantization tables for a codec, wherein respective quantization tables are associated with at least one tag indicating a type of stream data;

[0138] The identification unit is configured to identify a quantization table from the storage unit according to the received label; and

[0139] The transmission unit is configured to transmit at least a portion of the identified quantization table to the neural decoding system for an encoding unit of the neural decoding system, and to transmit at least another portion of the identified quantization table to the neural decoding system for a decoding unit of the neural decoding system.

[0140] 15. A neural decoding transmission device for streaming, the neural decoding transmission device comprising:

[0141] an identification unit configured to identify one or more tags indicating a type of streaming data to be streamed by the neural decoding transmission device;

[0142] a transmission unit configured to transmit one or more identified tags to a management device;

[0143] a receiving unit configured to receive first weight data representing a weight of a first neural network from the management device;

[0144] a weight modification unit configured to modify the weight of the first neural network according to the first weight data received by the receiving unit;

[0145] a preprocessing unit configured to preprocess the stream data using the first neural network to obtain preprocessed data;

[0146] an encoding unit configured to encode the preprocessed data using a real codec; and

[0147] The transmission unit is configured to transmit the encoded pre-processed data stream to the neural decoding receiving device for streaming reception.

[0148] 16. A neural decoding receiving device for receiving streaming data, the neural decoding receiving device comprising:

[0149] a receiving unit configured to receive second weight data representing a weight of a second neural network from a management device;

[0150] a weight modification unit configured to modify the weight of the second neural network according to the second weight data received by the receiving unit;

[0151] The receiving unit is configured to receive the encoded pre-processed streaming data from a neural decoding transmission device for streaming transmission;

[0152] a decoding unit configured to decode the encoded preprocessed stream data using the real codec to obtain decoded preprocessed stream data; and

[0153] A post-processing unit is configured to post-process the decoded pre-processed stream data using the second neural network to obtain a reconstruction of the stream data.

[0154] 17. The neural decoding receiving device according to clause 16, further comprising:

[0155] A transmission unit is configured to send one or more device attributes of the neural decoding receiving device to the management device.

[0156] 18. A neural decoding system for streaming, comprising:

[0157] A management device according to any of clauses 11 to 14;

[0158] A neural decoding transmission device according to clause 15; and

[0159] A neural decoding receiving device according to any of clauses 16-17.

[0160] 19. A method for managing a neural decoding system, the method comprising the steps of:

[0161] receiving one or more tags indicating a type of streaming data to be streamed by the neural decoding system;

[0162] storing a plurality of sets of weights for a neural network of the neural decoding system, wherein a respective set of weights is associated with at least one label indicating a stream data type;

[0163] identifying a set of weights from the storage unit based on the received labels;

[0164] For a first neural network of the neural decoding system, sending first weight data representing a portion of the identified set of weights to the neural decoding system; and

[0165] For a second neural network of the neural decoding system, second weight data representing another portion of the identified set of weights is sent to the neural decoding system.

[0166] 20. A neural decoding system according to any one of clauses 2 to 8 and 18, comprising:

[0167] a segmentation unit configured to segment the stream data into a plurality of stream data content components according to the content in the stream data before the stream data is preprocessed by the preprocessing unit, wherein

[0168] streaming, by the neural decoding system, a corresponding stream data content component independently of other stream data content components to obtain a corresponding reconstruction of the corresponding stream data content component; and

[0169] A synthesis unit is configured to combine multiple reconstructions of corresponding stream data content components for output by the neural decoding system.

[0170] 21. A neural decoding system according to any one of clauses 2-8, 18 and 20, wherein

[0171] A portion of the stream data is in a format not supported by the real codec;

[0172] The first neural network is trained to convert the portion of the stream data into latent data in a format supported by the real codec during the pre-processing; and

[0173] The second neural network is trained to convert, during post-processing, decoded latent data comprised by the decoded pre-processed data into a reconstruction of the portion of the stream data in a format not supported by the real codec.

[0174] 22. A neural decoding system according to clause 21 when dependent on clause 18, wherein at least one of the one or more tags received by the management device indicates the portion of the stream data in a format not supported by the real codec.

[0175] 23. A non-transitory machine-readable storage medium storing the hybrid codec according to clause 1.

[0176] 24. Computer software which, when executed by a computer, causes the computer to perform a method according to any one of clauses 9, 10 and 19.

[0177] 25. A non-transitory machine-readable storage medium storing computer software according to clause 24.

Claims

1. A hybrid codec for training a neural decoding system, the hybrid codec comprising: a real codec for encoding data pre-processed by the first neural network and for decoding the encoded data for post-processing by the second neural network; as well as A proxy codec that is a differentiable representation of the true codec and is used to back-propagate a loss function that represents a decoding bit rate of the true codec and a decoding distortion between the post-processed data and the original data for training at least the weights of the first neural network.

2. A neural decoding system for streaming, the neural decoding system comprising: a preprocessing unit configured to preprocess the streaming data using a first neural network trained to preprocess the streaming data to obtain preprocessed data; The hybrid codec according to claim 1; an encoding unit configured to encode the preprocessed data using the real codec; a decoding unit configured to decode the encoded preprocessed data using the real codec; as well as A post-processing unit is configured to post-process the decoded pre-processed data using a second neural network trained to post-process the decoded pre-processed data to obtain a reconstruction of the stream data.

3. The neural decoding system according to claim 2, comprising: a loss calculation unit configured to calculate a loss function, the loss function representing a decoding bit rate of the stream data and a decoding distortion between the stream data and a reconstruction of the stream data; as well as A back-propagation unit is configured to further train the weights of the first neural network by back-propagating the loss function using the proxy codec as a differentiable representation of the real codec.

4. The neural decoding system according to claim 3, wherein: The back-propagation unit is configured to further train weights of the second neural network by back-propagating the loss function through the second neural network before back-propagating the loss function using the proxy codec.

5. The neural decoding system according to any one of claims 2 to 4, comprising: A training repository unit is configured to store the stream data to further train the first neural network and the second neural network.

6. The neural decoding system according to claim 5, wherein: The training repository unit is configured to store a reconstruction of the streaming data in association with the streaming data to further train the first neural network and the second neural network.

7. A neural decoding system according to any one of claims 5 and 6, when dependent on any one of claims 3 and 4, wherein: The back-propagation unit is configured to further train the weights when the utilization level of the neural decoding system is below a predetermined threshold.

8. The neural decoding system according to any one of claims 5-7, further comprising: A filter unit configured to filter the data to be stored in the training repository unit using a third neural network trained to identify the data most likely to improve the first neural network and the second neural network upon further training.

9. The neural decoding system according to any one of claims 2 to 8, comprising: a segmentation unit configured to segment the stream data into a plurality of stream data content components according to the content in the stream data before the stream data is preprocessed by the preprocessing unit, wherein streaming, by the neural decoding system, a corresponding stream data content component independently of other stream data content components to obtain a corresponding reconstruction of the corresponding stream data content component; and A synthesis unit is configured to combine multiple reconstructions of corresponding stream data content components for output by the neural decoding system.

10. The neural decoding system according to any one of claims 2 to 9, wherein: A portion of the stream data is in a format not supported by the real codec; The first neural network is trained to convert the portion of the stream data into latent data in a format supported by the real codec during the pre-processing; and The second neural network is trained to convert, during the post-processing, decoded latent data comprised by the decoded pre-processed data into a reconstruction of the portion of the stream data in a format not supported by the real codec.

11. A method for streaming using neural decoding using the hybrid codec according to claim 1, the method comprising the following steps: preprocessing the stream data using a first neural network trained to preprocess the stream data to obtain preprocessed data; encoding the preprocessed data using the real codec; decoding the encoded preprocessed data using the real codec; as well as The decoded pre-processed data is post-processed using a second neural network trained to post-process the decoded pre-processed data to obtain a reconstruction of the stream data.

12. A method for training a first neural network according to claim 11, the method comprising the following steps: calculating a loss function representing a decoding bit rate of the stream data decoded by the method according to claim 9 and a decoding distortion between the stream data and a reconstruction of the stream data obtained by the method according to claim 9; and The weights of the first neural network are further trained by back-propagating the loss function using the proxy codec.

13. A non-transitory machine-readable storage medium storing the hybrid codec according to claim 1.

14. Computer software which, when executed by a computer, causes the computer to perform the method according to any one of claims 11 to 12.

15. A non-transitory machine-readable storage medium storing computer software according to claim 14.

Images