Devices, systems, codecs and methods for neural coding

Abstract

A hybrid codec comprises a real codec and a proxy codec. The real codec may be any standard codec. The proxy codec is a differentiable representation of the real codec. A streaming method uses neural coding in which streaming data is pre-processed with a neural network, encoded and decoded using a real codec, then post-processed using a neural network to reconstruct the original streaming data. A method for training a neural network calculates a loss function representative of a streaming data coding bit-rate and distortion, and back-propagates this loss function using a proxy codec. A neural coding streaming manager stores neural network weights associated with tags indicating streaming data types. The manager identifies weights based on a received tag and transmits portions of these weights to two neural networks. Neural coding transmitters and receivers modify a neural network’s weights based on received weight data. The transmitter pre-processes streaming data using the neural network before encoding the pre-processed data with a real codec. The receiver decodes data with a real codec before post-processing the decoded data with a real codec. The received weight data may be identified based on a tag indicating a streaming content type.

Images