Method and data processing system for lossy image or video encoding, transmission and decoding

Abstract

A method for lossy image or video encoding, transmission, and decoding, the method comprising the steps of: with a first neural network, producing a latent representation of optical flow information using the first image and the second image, the optical flow information being indicative of a difference between the first image and the second image; with a second neural network, decoding the latent representation of optical flow information to produce an approximation of the optical flow information; with a third neural network, producing an output image using the optical flow information, wherein the output image is an approximation of the first image; wherein the first image and the second image comprise data arranged in a respective plurality of image channels; and wherein using the first image and the second image to produce a latent representation of optical flow information comprises using a subset of the plurality of channels.

Description

[0001] Method and data processing system for lossy image or videoencoding, transmission and decodingBACKGROUNDThis invention relates to a method and system for lossy image or video encoding, transmissionand decoding, a method, apparatus, computer program and computer readable storage mediumfor lossy image or video encoding and transmission, and a method, apparatus, computerprogram and computer readable storage medium for lossy image or video receipt and decoding.There is increasing demand from users of communications networks for images and videocontent. Demand is increasing not just for the number of images viewed, and for the playingtime of video; demand is also increasing for higher resolution content. This places increasingdemand on communications networks and increases their energy use because of the largeramount of data being transmitted.To reduce the impact of these issues, image and video content is compressed for transmissionacross the network. The compression of image and video content can be lossless or lossycompression. In lossless compression, the image or video is compressed such that all of theoriginal information in the content can be recovered on decompression. However, when usinglossless compression there is a limit to the reduction in data quantity that can be achieved. Inlossy compression, some information is lost from the image or video during the compressionprocess. Known compression techniques attempt to minimise the apparent loss of informationby the removal of information that results in changes to the decompressed image or video thatis not particularly noticeable to the human visual system. JPEG, JPEG2000, AVC, HEVC andAVI are examples of compression processes for image and / or video files.In general terms, known lossy image compression techniques use the spatial correlationsbetween pixels in images to remove redundant information during compression. For example,in an image of a blue sky, if a given pixel is blue, there is a high likelihood that the neighbouringpixels, and their neighbouring pixels, and so on, are also blue. There is accordingly no need toretain all the raw pixel data. Instead, we can retain only a subset of the pixels which take upfewer bits and infer the pixel values of the other pixels using information derived from spatialcorrelations.A similar approach is applied in known lossy video compression techniques. That is, spatialcorrelations between pixels allow the removal of redundant information during compression.However, in video compression, there is further information redundancy in the form of temporalcorrelations. For example, in a video of an aircraft flying across a blue-sky background, mostof the pixels of the blue sky do not change at all between frames of the video. The mostof the blue sky pixel data for the frame at position t = 0 in the video is identical to that atposition t = 10. Storing this identical, temporally correlated, information is inefficient. Instead,only the blue sky pixel data for a subset of the frames is stored and the rest are inferred frominformation derived from temporal correlations.In the realm of lossy video compression in particular, the removal of redundant temporallycorrelated information in a video sequence is known inter-frame redundancy.One technique using inter-frame redundancy that is widely used in standard video compressionalgorithms involves the categorization of video frames into three types: I-frames, P-frames, andB-frames. Each frame type carries distinct properties concerning their encoding and decodingprocess, playing different roles in achieving high compression ratios while maintainingacceptable visual quality.I-frames, or intra-coded frames, serve as the foundation of the video sequence. These framesare self-contained, each one encoding a complete image without reference to any other frame.In terms of compression, I-frames are least compressed among all frame types, thus carryingthe most data. However, their independence provides several benefits, including being thestarting point for decompression and enabling random access, crucial for functionalities likefast-forwarding or rewinding the video.P-frames, or predictive frames, utilize temporal redundancy in video sequences to achievegreater compression. Instead of encoding an entire image like an I-frame, a P-frame representsthe difference between itself and the closest preceding I- or P-frame. The process, known asmotion compensation, identifies and encodes only the changes that have occurred, therebysignificantly reducing the amount of data transmitted. Nonetheless, P-frames are dependent onprevious frames for decoding. Consequently, any error during the encoding or transmissionprocess may propagate to subsequent frames, impacting the overall video quality.B-frames, or bidirectionally predictive frames, represent the highest level of compression.Unlike P-frames, B-frames use both the preceding and following frames as references in theirencoding process. By predicting motion both forwards and backwards in time, B-framesencode only the differences that cannot be accurately anticipated from the previous and nextframes, leading to substantial data reduction. Although this bidirectional prediction makesB-frames more complex to generate and decode, it does not propagate decoding errors sincethey are not used as references for other frames. Artificial intelligence (AI) based compressiontechniques achieve compression and decompression of images and videos through the use oftrained neural networks in the compression and decompression process. Typically, duringtraining of the neutral networks, the difference between the original image and video and thecompressed and decompressed image and video is analyzed and the parameters of the neuralnetworks are modified to reduce this difference while minimizing the data required to transmitthe content. However, AI based compression methods may achieve poor compression resultsin terms of the appearance of the compressed image or video or the amount of informationrequired to be transmitted.An example of an AI based image compression process comprising a hyper-network is describedin Ballé, Johannes, et al. “Variational image compression with a scale hyperprior.” arXivpreprint arXiv:1802.01436 (2018), which is hereby incorporated by reference.An example of an AI based video compression approach is shown in Agustsson, E., Minnen, D.,Johnston, N., Balle, J., Hwang, S. J., and Toderici, G. (2020), Scale-space flow for end-to-endoptimized video compression. In Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition (pp. 8503-8512), which is hereby incorporated by reference.A further example of an AI based video compression approach is shown in Mentzer, F.,Agustsson, E., Ballé, J., Minnen, D., Johnston, N., and Toderici, G. (2022, November). Neuralvideo compression using gans for detail synthesis and propagation. In Computer Vision–ECCV2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, PartXXVI (pp. 562-578), which is hereby incorporated by reference.SUMMARYAccording to an aspect of the present disclosure, there is provided a method for lossy image orvideo encoding and transmission, and decoding, the method comprising the steps of:receiving a first image and a second image at a first computer system;with a first neural network, producing a latent representation of optical flow informationusing the first image and the second image, the optical flow information being indicative of adifference between the first image and the second image;transmitting the latent representation of optical flow information to a second computersystem; with a second neural network, decoding the latent representation of optical flowinformation to produce an approximation of the optical flow information; andwith a third neural network, producing an output image using the optical flow information,wherein the output image is an approximation of the first image;wherein the first image and the second image comprise data arranged in a respectiveplurality of image channels; andwherein using the first image and the second image to produce a latent representation ofoptical flow information comprises using a subset of the plurality of channels.Optionally, the subset comprises a single channel of the plurality of channels of each of thefirst image and the second image.Optionally, the plurality of channels of each of the first image and the second image comprisea luma channel and at least one chroma channel.Optionally, the subset comprises the luma channel.Optionally, the luma channel and the at least one chroma channel are defined in a YUV colourspace.Optionally, the at least one chroma channel has a different resolution to the luma channel.Optionally, the YUV colour space comprises a YUV 4:2:0 or YUV 4:2:2 colour space.Optionally, using the first image and the second image to produce a latent representation ofoptical flow information comprises:producing a representation of optical flow information at a plurality of resolutions andusing the representation of optical flow information at the plurality of resolutions to producesaid latent representation of optical flow information.Optionally, a representation of optical flow information at a first resolution of said plurality ofresolutions is based on a representation of optical flow information at a second resolution ofsaid plurality of resolutions.Optionally, using the first image and the second image to produce a latent representation ofoptical flow information comprises:using the representation of optical flow information at one of said plurality of resolutionsto warp a representation of the first image at a different resolution of said plurality of resolutions.Optionally, wherein a representation of optical flow information at a different one of saidplurality of resolutions is based on the warped first image and the second image at one of saidplurality of resolutions.Optionally, the representation of optical flow information is estimated using the subset of theplurality of channels, and wherein the warping is performed using the plurality of channels.According to an aspect, there is provided a method for lossy image or video encoding andtransmission, the method comprising the steps of:receiving a first image and a second image at a first computer system;with a first neural network, producing a latent representation of optical flow informationusing the first image and the second image, the optical flow information being indicative of adifference between the first image and the second image;wherein the first image and the second image comprise data arranged in a plurality ofimage channels; andwherein using the first image and the second image to produce a latent representation ofoptical flow information comprises using a subset of the plurality of channels.According to an aspect, there is provided a method for lossy image or video receipt anddecoding, the method comprising the steps of:receiving a latent representation of optical flow information at a second computer system,the optical flow information being indicative of a difference between a first image and a secondimage each comprising data arranged in a plurality of image channels, the latent representationof optical flow information being produced with first neural network using a subset of theplurality of channels;with a second neural network, decoding the latent representation of optical flowinformation to produce an approximation of the optical flow information; andwith a third neural network, producing an output image using the optical flow information,wherein the output image is an approximation of the first image.According to an aspect, there is provided data processing apparatus configured to perform anyof the above methods.According to an aspect, there is provided a computer program comprising instructions which,when the program is executed by a computer, cause the computer to carry out any of the abovemethods.According to an aspect, there is provided a computer-readable storage medium comprisinginstructions which, when executed by a computer, cause the computer to carry out any of theabove methods.According to an aspect of the present disclosure, there is provided a method for lossy image orvideo encoding and transmission, and decoding, the method comprising the steps of:receiving a first image and a second image at a first computer system;with a first neural network, producing a latent representation of optical flow informationusing the first image and the second image, the optical flow information being indicative of adifference between the first image and the second image;transmitting the latent representation of optical flow information to a second computersystem; with a second neural network, decoding the latent representation of optical flowinformation to produce an approximation of the optical flow information; andwith a third neural network, producing an output image using the optical flow information,wherein the output image is an approximation of the first image;wherein the first image and the second image comprise data arranged in respective lumachannels and chroma channels; andwherein using the first image and the second image to produce a latent representation ofoptical flow information comprises:producing a representation of the optical flow information based on the respectiveluma channels of the first image and the second image; anddownsampling the representation of the optical flow information.Optionally, said downsampling comprises downsamping the representation of the optical flowinformation to a resolution of the respective chroma channels of the first image and the secondimage.Optionally, the method comprises warping the data in the respective chroma channels usingthe downsampled representation of the optical flow information.Optionally, the method comprisies warping the data in the respective luma channels using thedownsampled representation of the optical flow information.Optionally, using the first image and the second image to produce a latent representation ofoptical flow information comprises:producing a representation of optical flow information from the respective luma channelsat a plurality of resolutions and using the representation of optical flow information at theplurality of resolutions to produce said latent representation of optical flow information.Optionally, a representation of optical flow information at a first resolution of said plurality ofresolutions is based on a representation of optical flow information at a second resolution ofsaid plurality of resolutions.Optionally, the representation of optical flow information at one or more of said plurality ofresolutions is based on said warped data.Optionally, the respective luma channels and chroma channels are defined in a YUV colourspace.Optionally, the respective chroma channels have a different resolution to the luma channel.Optionally, the YUV colour space comprises a YUV 4:2:0 or YUV 4:2:2 colour space.According to an aspect, there is provided a method for lossy image or video encoding andtransmission, the method comprising the steps of:receiving a first image and a second image at a first computer system;with a first neural network, producing a latent representation of optical flow informationusing the first image and the second image, the optical flow information being indicative of adifference between the first image and the second image;transmitting the latent representation of optical flow information to a second computersystem; wherein the first image and the second image comprise data arranged in respective lumachannels and chroma channels; andwherein using the first image and the second image to produce a latent representation ofoptical flow information comprises:producing a representation of the optical flow information based on the respectiveluma channels of the first image and the second image; anddownsampling the representation of the optical flow information.According to an aspect, there is provided a method for lossy image or video receipt anddecoding, the method comprising the steps of:receiving a latent representation of optical flow information to a second computer system,the optical flow information being indicative of a difference between a first image and a secondimage each comprising data arranged in respective luma channels and chroma channels, thelatent representation of optical flow information being produced with a first neural networkusing a downsampled representation of the optical flow information based on the respectiveluma channels of the first image and the second image;with a second neural network, decoding the latent representation of optical flowinformation to produce an approximation of the optical flow information; andwith a third neural network, producing an output image using the optical flow information,wherein the output image is an approximation of the first image.According to an aspect, there is provided data processing apparatus configured to perform anyof the above methods.According to an aspect, there is provided a computer program comprising instructions which,when the program is executed by a computer, cause the computer to carry out any of the abovemethods.According to an aspect, there is provided a computer-readable storage medium comprisinginstructions which, when executed by a computer, cause the computer to carry out any of theabove methods.According to an aspect, there is provided a method for lossy image or video encoding andtransmission, and decoding, the method comprising the steps of:receiving a first image and a second image at a first computer system, the first image andthe second image comprising data arranged in a respective plurality of image channels;transforming the data by distributing information of the image channels from a spatialdimension into a channel dimension;with a first neural network, producing a latent representation of optical flow informationusing the transformed data, the optical flow information being indicative of a differencebetween the first image and the second image;transmitting the latent representation of optical flow information to a second computersystem; with a second neural network, decoding the latent representation of optical flowinformation to produce an approximation of the optical flow information; andwith a third neural network, producing an output image using the optical flow information,wherein the output image is an approximation of the first image.Optionally, the information comprises pixel values on said plurality of image channels.Optionally, the plurality of image channels comprises a subset of channels in a first spatialresolution different to a spatial resolution of the other channels in the plurality of imagechannels.Optionally, the transformed data comprises a plurality of image channels each having a samespatial resolution.Optionally, said same spatial resolution is lower than the spatial resolution of said subset ofchannels.Optionally, said same spatial resolution is lower than the spatial resolution of said otherchannels of the plurality of channels,Optionally, the data comprises 3-channel YUV data and wherein said transforming comprisestransforming the 3-channel YUV data into 24-channel data.Optionally, the YUV data comprises one of 4:2:0 YUV data or 4:2:2 YUV data.Optionally, the transformation comprises performing a pixel unshuffle operation on the data.Optionally, the pixel unshuffle operation is defined by a first block size for a first image channelof the data, and defined by a second block size for a second image channel of the data .Optionally, the transformation comprises performing a convolution operation on the data.Optionally, the convolution operation is defined by a first stride for a first image channel of thedata, and defined by a second stride for a second image channel of the data.Optionally, the transformation comprises upsampling a subset of said plurality of imagechannels to produce a plurality of image channels each having a same spatial resolution.Optionally, one or more of the first, second or third neural networks is defined by a convolutionoperation, and wherein said transforming increases a receptive field of the convolutionoperation.According to an aspect of the present disclsoure, there is provided a method for lossy image orvideo encoding and transmission, the method comprising the steps of:receiving a first image and a second image at a first computer system, the first image andthe second image comprising data arranged in a respective plurality of image channels;transforming the data by distributing information of the image channels from a spatialdimension into a channel dimension;with a first neural network, producing a latent representation of optical flow informationusing the transformed data, the optical flow information being indicative of a differencebetween the first image and the second image;transmitting the latent representation of optical flow information to a second computersystem.According to an aspect of the present disclosure, there is provided a method for lossy image orvideo receipt and decoding, the method comprising the steps of:receiving a first image and a second image at a first computer system, the first image andthe second image comprising data arranged in a respective plurality of image channels;transforming the data by distributing information of the image channels from a spatialdimension into a channel dimension;with a first neural network, producing a latent representation of optical flow informationusing the transformed data, the optical flow information being indicative of a differencebetween the first image and the second image;receiving a latent representation of optical flow information at a second computer system,the ;with a second neural network, decoding the latent representation of optical flowinformation to produce an approximation of the optical flow information; andwith a third neural network, producing an output image using the optical flow information,wherein the output image is an approximation of the first image.According to an aspect of the present disclosure, there is provided a method for lossy image orvideo receipt and decoding, the method comprising the steps of:receiving a latent representation of optical flow information at a second computer system,the optical flow information being indicative of a difference between a first image and a secondimage each comprising data arranged in a plurality of image channels, the latent representationof optical flow information being produced with a first neural network and by transforming thedata by distributing information of the image channels from a spatial dimension into a channeldimension; with a second neural network, decoding the latent representation of optical flowinformation to produce an approximation of the optical flow information; andwith a third neural network, producing an output image using the optical flow information,wherein the output image is an approximation of the first image.According to an aspect, there is provided data processing apparatus configured to perform anyof the above methods.According to an aspect, there is provided a computer program comprising instructions which,when the program is executed by a computer, cause the computer to carry out any of the abovemethods.According to an aspect, there is provided a computer-readable storage medium comprisinginstructions which, when executed by a computer, cause the computer to carry out any of theabove methods.BRIEF DESCRIPTION OF THE DRAWINGSAspects of the invention will now be described by way of examples, with reference to thefollowing figures in which:Figure 1 illustrates an example of an image or video compression, transmission and decom-pression pipeline.Figure 2 illustrates a further example of an image or video compression, transmission anddecompression pipeline including a hyper-network.Figure 3 illustrates an example of a video compression, transmission and decompressionpipeline.Figure 4 illustrates an example of a video compression, transmission and decompressionsystem.Figure 5 illustrates an example of a video compression, transmission and decompressionsystem.Figure 6 illustrates an example of a flow module of a video compression, transmission anddecompression system.Figure 7a illustrates an example of a video compression, transmission and decompressionsystem.Figure 7b illustrates an example of a video compression, transmission and decompressionsystem.Figure 8 illustrates an example of a flow module of a video compression, transmission anddecompression system.Figure 9 illustrates an example of a flow module of a video compression, transmission anddecompression system.DETAILED DESCRIPTION OF THE DRAWINGSCompression processes may be applied to any form of information to reduce the amountof data, or file size, required to store that information. Image and video information is anexample of information that may be compressed. The file size required to store the information,particularly during a compression process when referring to the compressed file, may bereferred to as the rate. In general, compression can be lossless or lossy. In both forms ofcompression, the file size is reduced. However, in lossless compression, no information is lostwhen the information is compressed and subsequently decompressed. This means that theoriginal file storing the information is fully reconstructed during the decompression process.In contrast to this, in lossy compression information may be lost in the compression anddecompression process and the reconstructed file may differ from the original file. Image andvideo files containing image and video data are common targets for compression.In a compression process involving an image, the input image may be represented as ^^. Thedata representing the image may be stored in a tensor of dimensions ^^ × ^^ × ^^, where ^^represents the height of the image, ^^ represents the width of the image and ^^ represents thenumber of channels of the image. Each ^^ × ^^ data point of the image represents a pixel valueof the image at the corresponding location. Each channel ^^ of the image represents a differentcomponent of the image for each pixel which are combined when the image file is displayed bya device. For example, an image file may have 3 channels with the channels representing thered, green and blue component of the image respectively. In this case, the image informationis stored in the RGB colour space, which may also be referred to as a model or a format.Other examples of colour spaces or formats include the CMKY and the YCbCr colour models.However, the channels of an image file are not limited to storing colour information and otherinformation may be represented in the channels. As a video may be considered a series ofimages in sequence, any compression process that may be applied to an image may also beapplied to a video. Each image making up a video may be referred to as a frame of the video.The output image may differ from the input image and may be represented by ^^. The differencebetween the input image and the output image may be referred to as distortion or a differencein image quality. The distortion can be measured using any distortion function which receivesthe input image and the output image and provides an output which represents the differencebetween input image and the output image in a numerical way. An example of such a methodis using the mean square error (MSE) between the pixels of the input image and the outputimage, but there are many other ways of measuring distortion, as will be known to the personskilled in the art. The distortion function may comprise a trained neural network.Typically, the rate and distortion of a lossy compression process are related. An increase inthe rate may result in a decrease in the distortion, and a decrease in the rate may result in anincrease in the distortion. Changes to the distortion may affect the rate in a correspondingmanner. A relation between these quantities for a given compression technique may be definedby a rate-distortion equation.AI based compression processes may involve the use of neural networks. A neural network isan operation that can be performed on an input to produce an output. A neural network maybe made up of a plurality of layers. The first layer of the network receives the input. One ormore operations may be performed on the input by the layer to produce an output of the firstlayer. The output of the first layer is then passed to the next layer of the network which mayperform one or more operations in a similar way. The output of the final layer is the output ofthe neural network.Each layer of the neural network may be divided into nodes. Each node may receive at leastpart of the input from the previous layer and provide an output to one or more nodes in asubsequent layer. Each node of a layer may perform the one or more operations of the layer onat least part of the input to the layer. For example, a node may receive an input from one ormore nodes of the previous layer. The one or more operations may include a convolution, aweight, a bias and an activation function. Convolution operations are used in convolutionalneural networks. When a convolution operation is present, the convolution may be performedacross the entire input to a layer. Alternatively, the convolution may be performed on at leastpart of the input to the layer.Each of the one or more operations is defined by one or more parameters that are associatedwith each operation. For example, the weight operation may be defined by a weight matrixdefining the weight to be applied to each input from each node in the previous layer to eachnode in the present layer. In this example, each of the values in the weight matrix is a parameterof the neural network. The convolution may be defined by a convolution matrix, also knownas a kernel. In this example, one or more of the values in the convolution matrix may be aparameter of the neural network. The activation function may also be defined by values whichmay be parameters of the neural network. The parameters of the network may be varied duringtraining of the network.Other features of the neural network may be predetermined and therefore not varied duringtraining of the network. For example, the number of layers of the network, the number ofnodes of the network, the one or more operations performed in each layer and the connectionsbetween the layers may be predetermined and therefore fixed before the training process takesplace. These features that are predetermined may be referred to as the hyperparameters of thenetwork. These features are sometimes referred to as the architecture of the network.To train the neural network, a training set of inputs may be used for which the expected output,sometimes referred to as the ground truth, is known. The initial parameters of the neuralnetwork are randomized and the first training input is provided to the network. The output ofthe network is compared to the expected output, and based on a difference between the outputand the expected output the parameters of the network are varied such that the differencebetween the output of the network and the expected output is reduced. This process is thenrepeated for a plurality of training inputs to train the network. The difference between theoutput of the network and the expected output may be defined by a loss function. The result ofthe loss function may be calculated using the difference between the output of the networkand the expected output to determine the gradient of the loss function. Back-propagation ofthe gradient descent of the loss function may be used to update the parameters of the neuralnetwork using the gradients ^^^^ / ^^^^ of the loss function. A plurality of neural networks in asystem may be trained simultaneously through back-propagation of the gradient of the lossfunction to each network.In the context of image or video compression, this type of system, where simultaneous trainingwith back-propagation through each element or the whole network architecture may be referredto as end-to-end, learned image or video compression. Unlike in traditional compressionalgorithms that use primarily handcrafted, manually constructed steps, an end-to-end learnedsystem learns itself during training what combination of parameters best achieves the goal ofminimising the loss function. This approach is advantageous compared to systems that are notend-to-end learned because an end-to-end system has a greater flexibility to learn weights andparameters that might be counter-intuitive to someone handcrafting features.It will be appreciated that the term "training" or "learning" as used herein means the processof optimizing an artificial intelligence or machine learning model, based on a given set of data.This involves iteratively adjusting the parameters of the model to minimize the discrepancybetween the model’s predictions and the actual data, represented by the above-describedrate-distortion loss function.The training process may comprise multiple epochs. An epoch refers to one complete passof the entire training dataset through the machine learning algorithm. During an epoch, themodel’s parameters are updated in an effort to minimize the loss function. It is envisaged thatmultiple epochs may be used to train a model, with the exact number depending on variousfactors including the complexity of the model and the diversity of the training data.Within each epoch, the training data may be divided into smaller subsets known as batches.The size of a batch, referred to as the batch size, may influence the training process. A smallerbatch size can lead to more frequent updates to the model’s parameters, potentially leading tofaster convergence to the optimal solution, but at the cost of increased computational resources.Conversely, a larger batch size involves fewer updates, which can be more computationallyefficient but might converge slower or even fail to converge to the optimal solution.The learnable parameters are updated by a specified amount each time, determined by thelearning rate. The learning rate is a hyperparameter that decides how much the parametersare adjusted during the training process. A smaller learning rate implies smaller steps in theparameter space and a potentially more accurate solution, but it may require more epochs toreach that solution. On the other hand, a larger learning rate can expedite the training processbut may risk overshooting the optimal solution or causing the training process to diverge.The training described herein may involve use of a validation set, which is a portion of thedata not used in the initial training, which is used to evaluate the model’s performance and toprevent overfitting. Overfitting occurs when a model learns the training data too well, to thepoint that it fails to generalize to unseen data. Regularization techniques, such as dropout orL1 / L2 regularization, can also be used to mitigate overfitting.It will be appreciated that training a machine learning model is an iterative process thatmay comprise selection and tuning of various parameters and hyperparameters. As will beappreciated, the specific details, such as hyper parameters and so on, of the training processmay vary and it is envisaged that producing a trained model in this way may achieved in anumber of different ways with different epochs, batch sizes, learning rates, regularisations,and so on, the details of which are not essential to enabling the advantages and effects of thepresent disclosure, except where stated otherwise. The point at which an “untrained” neuralnetwork is considered be “trained” is envisaged to be case specific and depend on, for example,on a number of epochs, a plateauing of any further learning, or some other metric and is notconsidered to be essential in achieving the advantages described herein.More details of an end-to-end, learned compression process will now be described. It will beappreciated that in some cases, end-to-end, learned compression processes may be combinedwith one or more components that are handcrafted or trained separately.In the case of AI based image or video compression, the loss function may be defined by therate distortion equation. The rate distortion equation may be represented by ^^^^^^^^ = ^^ + ^^ ∗ ^^,where ^^ is the distortion function, ^^ is a weighting factor, and ^^ is the rate loss. ^^ may bereferred to as a lagrange multiplier. The langrange multiplier provides as weight for a particularterm of the loss function in relation to each other term and can be used to control which termsof the loss function are favoured when training the network.In the case of AI based image or video compression, a training set of input images maybe used. An example training set of input images is the KODAK image set (for exampleat www.cs.albany.edu / xypan / research / snr / Kodak.html). An example training set of inputimages is the IMAX image set. An example training set of input images is the Imagenetdataset (for example at www.image-net.org / download). An example training set of inputimages is the CLIC Training Dataset P (“professional”) and M (“mobile”) (for example athttp: / / challenge.compression.cc / tasks / ).An example of an AI based compression, transmission and decompression process 100 isshown in Figure 1. As a first step in the AI based compression process, an input image 5 isprovided. The input image 5 is provided to a trained neural network 110 characterized by afunction ^^^^ acting as an encoder. The encoder neural network 110 produces an output basedon the input image. This output is referred to as a latent representation of the input image 5. Ina second step, the latent representation is quantised in a quantisation process 140 characterisedby the operation ^^, resulting in a quantized latent. The quantisation process transforms thecontinuous latent representation into a discrete quantized latent. An example of a quantizationprocess is a rounding function.In a third step, the quantized latent is entropy encoded in an entropy encoding process 150 toproduce a bitstream 130. The entropy encoding process may be for example, range or arithmeticencoding. In a fourth step, the bitstream 130 may be transmitted across a communicationnetwork.In a fifth step, the bitstream is entropy decoded in an entropy decoding process 160. Thequantized latent is provided to another trained neural network 120 characterized by a function^^^^ acting as a decoder, which decodes the quantized latent. The trained neural network 120produces an output based on the quantized latent. The output may be the output image of theAI based compression process 100. The encoder-decoder system may be referred to as anautoencoder.Entropy encoding processes such as range or arithmetic encoding are typically able to losslesslycompress given input data up to close to the fundamental entropy limit of that data, as determinedby the total entropy of the distribution of that data. Accordingly, one way in which end-to-end,learned compression can minimise the rate loss term of the rate-distortion loss function andthereby increase compression effectiveness is to learn autoencoder parameter values thatproduce low entropy latent representation distributions. Producing latent representationsdistributed with as low an entropy as possible allows entropy encoding to compress the latentdistributions as close to or to the fundamental entropy limit for that distribution. The lowerthe entropy of the distribution, the more entropy encoding can losslessly compress it and thelower the amount of data in the corresponding bitstream. In some cases where the latentrepresentation is distributed according to a gaussian or Laplacian distribution, this learningmay comprise learning optimal location and scale parameters of the gaussian or Laplaciandistributions, in other cases, it allows the learning of more flexible latent representationdistributions which can further help to achieve the minimising of the rate-distortion lossfunction in ways that are not intuitive or possible to do with handcrafted features. Examples ofthese and other advantages are described in WO2021 / 220008A1, which is incorporated in itsentirety by reference.Something which is closely linked to the entropy encoding of the latent distribution and whichaccordingly also has an effect on the effectiveness of compression of end-to-end learnedapproaches is the quantisation step. During inference, a rounding function may be used toquantise a latent representation distribution into bins of given sizes, a rounding function isnot differentiable everywhere. Rather, a rounding function is effectively one or more stepfunctions whose gradient is either zero (at the top of the steps) or infinity (at the boundarybetween steps). Back propagating a gradient of a loss function through a rounding functionis challenging. Instead, during training, quantisation by rounding function is replaced byone or more other approaches. For example, the functions of a noise quantisation model aredifferentiable everywhere and accordingly do allow backpropagation of the gradient of theloss function through the quantisation parts of the end-to-end, learned system. Alternatively, astraight-through estimator (STE) quantisation model or one other quantisation models may beused. It is also envisaged that different quantisation models may be used for during evaluationof different term of the loss function. For example, noise quantisation may used to evaluate therate or entropy loss term of the rate-distortion loss function while STE quantisation may beused to evaluate the distortion term.In a similar manner to how learning parameters top produce certain distributions of the latentrepresentation facilitates achieving better rate loss term minimisation, end-to-end learning ofthe quantisation process achieves a similar effect. That is, learnable quantisation parametersprovide the architecture with a further degree of freedom to achieve the goal of minimising theloss function. For example, parameters corresponding to quantisation bin sizes may be learnedwhich is likely to result in an improved rate-distortion loss outcome compared to approachesusing hand-crafted quantisation bin sizes.Further, as the rate-distortion loss function constantly has to balance a rate loss term against adistortion loss term, it has been found that the more degrees of freedom the system has duringtraining, the better the architecture is at achieving optimal rate and distortion trade off.The system described above may be distributed across multiple locations and / or devices. Forexample, the encoder 110 may be located on a device such as a laptop computer, desktopcomputer, smart phone or server. The decoder 120 may be located on a separate device whichmay be referred to as a recipient device. The system used to encode, transmit and decode theinput image 5 to obtain the output image 6 may be referred to as a compression pipeline.The AI based compression process may further comprise a hyper-network 105 for thetransmission of meta-information that improves the compression process. The hyper-network105 comprises a trained neural network 115 acting as a hyper-encoder ^^ ℎ^^and a trained neuralnetwork 125 acting as a hyper-decoder ^^ℎ^^. An example of such a system is shown in Figure 2.Components of the system not further discussed may be assumed to be the same as discussedabove. The neural network 115 acting as a hyper-decoder receives the latent that is the output ofthe encoder 110. The hyper-encoder 115 produces an output based on the latent representationthat may be referred to as a hyper-latent representation. The hyper-latent is then quantizedin a quantization process 145 characterised by ^^ℎ to produce a quantized hyper-latent. Thequantization process 145 characterised by ^^ℎ may be the same as the quantisation process 140characterised by ^^ discussed above.In a similar manner as discussed above for the quantized latent, the quantized hyper-latent isthen entropy encoded in an entropy encoding process 155 to produce a bitstream 135. Thebitstream 135 may be entropy decoded in an entropy decoding process 165 to retrieve thequantized hyper-latent. The quantized hyper-latent is then used as an input to trained neuralnetwork 125 acting as a hyper-decoder. However, in contrast to the compression pipeline 100,the output of the hyper-decoder may not be an approximation of the input to the hyper-decoder115. Instead, the output of the hyper-decoder is used to provide parameters for use in theentropy encoding process 150 and entropy decoding process 160 in the main compressionprocess 100. For example, the output of the hyper-decoder 125 can include one or more ofthe mean, standard deviation, variance or any other parameter used to describe a probabilitymodel for the entropy encoding process 150 and entropy decoding process 160 of the latentrepresentation. In the example shown in Figure 2, only a single entropy decoding process 165and hyper-decoder 125 is shown for simplicity. However, in practice, as the decompressionprocess usually takes place on a separate device, duplicates of these processes will be presenton the device used for encoding to provide the parameters to be used in the entropy encodingprocess 150.Further transformations may be applied to at least one of the latent and the hyper-latent at anystage in the AI based compression process 100. For example, at least one of the latent and thehyper latent may be converted to a residual value before the entropy encoding process 150,155is performed. The residual value may be determined by subtracting the mean value of thedistribution of latents or hyper-latents from each latent or hyper latent. The residual valuesmay also be normalised.To perform training of the AI based compression process described above, a training set ofinput images may be used as described above. During the training process, the parameters ofboth the encoder 110 and the decoder 120 may be simultaneously updated in each trainingstep. If a hyper-network 105 is also present, the parameters of both the hyper-encoder 115and the hyper-decoder 125 may additionally be simultaneously updated in each training step.The training process may further include a generative adversarial network (GAN). Whenapplied to an AI based compression process, in addition to the compression pipeline describedabove, an additional neutral network acting as a discriminator is included in the system. Thediscriminator receives an input and outputs a score based on the input providing an indicationof whether the discriminator considers the input to be ground truth or fake. For example, theindicator may be a score, with a high score associated with a ground truth input and a lowscore associated with a fake input. For training of a discriminator, a loss function is used thatmaximizes the difference in the output indication between an input ground truth and input fake.When a GAN is incorporated into the training of the compression process, the output image 6may be provided to the discriminator. The output of the discriminator may then be used in theloss function of the compression process as a measure of the distortion of the compressionprocess. Alternatively, the discriminator may receive both the input image 5 and the outputimage 6 and the difference in output indication may then be used in the loss function of thecompression process as a measure of the distortion of the compression process. Training ofthe neural network acting as a discriminator and the other neutral networks in the compressionprocess may be performed simultaneously. During use of the trained compression pipelinefor the compression and transmission of images or video, the discriminator neural network isremoved from the system and the output of the compression pipeline is the output image 6.Incorporation of a GAN into the training process may cause the decoder 120 to performhallucination. Hallucination is the process of adding information in the output image 6 thatwas not present in the input image 5. In an example, hallucination may add fine detail tothe output image 6 that was not present in the input image 5 or received by the decoder 120.The hallucination performed may be based on information in the quantized latent received bydecoder 120.Details of a video compression process will now be described. As discussed above, a video ismade up of a series of images arranged in sequential order. AI based compression process100 described above may be applied multiple times to perform compression, transmissionand decompression of a video. For example, each frame of the video may be compressed,transmitted and decompressed individually. The received frames may then be grouped toobtain the original video.The frames in a video may be labelled based on the information from other frames that is usedto decode the frame in a video compression, transmission and decompression process. Asdescribed above, frames which are decoded using no information from other frames may bereferred to as I-frames. Frames which are decoded using information from past frames may bereferred to as P-frames. Frames which are decoded using information from past frames andfuture frames may be referred to as B-frames. Frames may not be encoded and / or decoded inthe order that they appear in the video. For example, a frame at a later time step in the videomay be decoded before a frame at an earlier time.The images represented by each frame of a video may be related. For example, a number offrames in a video may show the same scene. In this case, a number of different parts of thescene may be shown in more than one of the frames. For example, objects or people in a scenemay be shown in more than one of the frames. The background of the scene may also beshown in more than one of the frames. If an object or the perspective is in motion in the video,the position of the object or background in one frame may change relative to the position ofthe object or background in another frame. The transformation of a part of the image froma first position in a first frame to a second position in a second frame may be referred to asflow, warping or motion compensation. The flow may be represented by a vector. One or moreflows that represent the transformation of at least part of one frame to another frame may bereferred to as a flow map.An example AI based video compression, transmission, and decompression process 200 isshown in Figure 3. The process 200 shown in Figure 3 is divided into an I-frame part 201for decompressing I-frames, and a P-frame part 202 for decompressing P-frames. It will beunderstood that these divisions into different parts are arbitrary and the process 200 may bealso be considered as a single, end-to-end pipeline.As described above, I-frames do not rely on information from other frames so the I-frame part201 corresponds to the compression, transmission, and decompression process illustrated inFigures 1 or 2. The specific details will not be repeated here but, in summary, an input image^^0 is passed into an encoder neural network 203 producing a latent representation which isquantised and entropy encoded into a bitstream 204. The subscript 0 in ^^0 indicates the inputimage corresponds to a frame of a video stream at position t = 0. This may be the first frame ofan entire video stream or the first frame of a chunk of a video stream made up of, for example,an I-frame and a plurality of subsequent P-frames and / or B-frames. The bitstream 204 is thenentropy decoded and passed into a decoder neural network 205 to reproduce a reconstructedimage ^^0 which in this case is an I-frame. The decoding step may be performed both locallyat the same location as where the input image compression occurs as well as at the locationwhere the decompression occurs. This allows the reconstructed image ^^0 to be available forlater use by components of both the encoding and decoding sides of the pipeline.In contrast to I-frames, P-frames (and B-frames) do rely on information from other frames.Accordingly, the P-frame part 202 at the encoding side of the pipeline takes as input not onlythe input image ^^^^ that is to be compressed (corresponding to a frame of a video stream atposition t), but also one or more previously reconstructed images ^^^^−1 from an earlier framet-1. As described above, the previously reconstructed ^^^^−1 is available at both the encodeand decode side of the pipeline and can accordingly be used for various purposes at both theencode and decode sides.At the encode side, previously reconstructed images may be used for generating a flow mapscontaining information indicative of inter-frame movement of pixels between frames. In theexample of Figure 3, both the image being compressed ^^^^ and the previously reconstructedimage from an earlier frame ^^^^−1 are passed into a flow module part 206 of the pipeline. Theflow module part 206 comprises an autoencoder such as that of the autoencoder systems ofFigures 1 and 2 but where the encoder neural network 207 has been trained to produce alatent representation of a flow map from inputs ^^^^−1 and ^^^^ , which is indicative of inter-framemovement of pixels or pixel groups between ^^^^−1 and ^^^^ . The latent representation of the flowmap is quantised and entropy encoded to compress it and then transmitted as a bitstream 208.On the decode side, the bitstream is entropy decoded and passed to a decoder neural network209 to produce a reconstructed flow map ^^ .The reconstructed flow map ^^ is applied to the previously reconstructed image ^^^^−1 to generatea warped image ^^^^−1,^^. It is envisaged that any suitable warping technique may be used, forexample bi-linear or tri-linear warping, as is described in Agustsson, E., Minnen, D., Johnston,N., Balle, J., Hwang, S. J., and Toderici, G. (2020), Scale-space flow for end-to-end optimizedvideo compression. In Proceedings of the IEEE / CVF Conference on Computer Vision andPattern Recognition (pp. 8503-8512), which is hereby incorporated by reference. It is furtherenvisaged that a scale-space flow approach as described in the above paper may also optionallybe used. The warped image ^^^^−1,^^ is a prediction of how the previously reconstructed image^^^^−1 might have changed between frame positions t-1 and t, based on the output flow mapproduced by the flow module part 206 autoencoder system from the inputs of ^^^^ and ^^^^−1.As with the I-frame, the reconstructed flow map ^^ and corresponding warped image ^^^^−1,^^may be produced both on the encode side and the decode side of the pipeline so they areavailable for use by other components of the pipeline on both the encode and decode sides.In the example of Figure 3, both the image being compressed ^^^^ and the ^^^^−1,^^ are passedinto a residual module part 210 of the pipeline. The residual module part 210 comprises anautoencoder system such as that of the autoencoder systems of Figures 1 and 2 but where theencoder neural network 211 has been trained to produce a latent representation of a residualmap indicative of differences between the input mage ^^^^ and the warped image ^^^^−1,^^. Thelatent representation of the residual map is then quantised and entropy encoded into a bitstream212 and transmitted. The bitstream 212 is then entropy decoded and passed into a decoderneural network 213 which reconstructs a residual map ^^ from the decoded latent representation.Alternatively, a residual map may first be pre-calculated between ^^^^ and the ^^^^−1,^^ and thepre-calculated residual map may be passed into an autoencoder for compression only. Thishand-crafted residual map approach is computationally simpler, but reduces the degrees offreedom with which the architecture may learn weights and parameters to achieve its goalduring training of minimising the rate-distortion loss function.Finally, on the decode side, the residual map ^^ is applied (e.g. combined by addition, subtractionor a different operation) to the warped image to produce a reconstructed image ^^^^ which is areconstruction of image ^^^^ and accordingly corresponds to a P-frame at position t in a sequenceof frames of a video stream. It will be appreciated that the reconstructed image ^^^^ can then beused to process the next frame. That is, it can be used to compress, transmit and decompress^^^^+1, and so on until an entire video stream or chunk of a video stream has been processed.Alternatively, the residual autoencoder may be trained to reconstruct the frame ^^^^ directlyfrom the entropy decoded bitstream by removing the connection between ^^^^−1,^^ and the outputof the residual block 210, thereby eliminating any direct combination step with the warpedpreviously decoded image to speed up inference. In this case, the flow information is intuitivelyunderstood to be indirectly captured within the residual information, which the residual decoderis able to learn to use to directly reconstruct the output image ^^^^ .Alternatively, the residual autoencoder may be trained to reconstruct the frame ^^^^ directly fromthe entropy decoded bitstream in combination with some representation of flow injected intoone or more layers of the residual decoder. In this case, the flow information is intuitivelyunderstood to be indirectly captured within the injected information, which the residual decoderis able to learn to use while decoding the latent representation of flow information to directlyreconstruct the output image ^^^^ .Thus, for a block of video frames comprising an I-frame and ^^ subsequent P-frames, thebitstream may contain (i) a quantised, entropy encoded latent representation of the I-frameimage, and (ii) a quantised, entropy encoded latent representation of a flow map and residualmap of each P-frame image. For completeness, whilst not illustrated in Figure 3, any of theautoencoder systems of Figure 3 may comprise hyper and hyper-hyper networks such as thosedescribed in connection with Figure 2. Accordingly, the bitstream may also contain hyper andhyper-hyper parameters, their latent quantised, entropy encoded latent representations and soon, of those networks as applicable.Finally, the above approach may generally also be extended to B-frames, for example as isdescribed in Pourreza, R., and Cohen, T. (2021). Extending neural p-frame codecs for b-framecoding. In Proceedings of the IEEE / CVF International Conference on Computer Vision (pp.6680-6689).The above-described flow and residual based approach is highly effective at reducing theamount of data that is to be transmitted because, as long as at least one reconstructed frame(e.g. I-frame ^^^^−1) is available, the encode side only compresses and transmits a flow mapand a residual map (and any hyper or hyper-hyper parameter information, as applicable) toreconstruct a subsequent frame.Figure 4 shows an example of an AI image or video compression process such as that describedabove in connection with Figures 1-3 implemented in a video streaming system 400. Thesystem 400 comprises a first device 401 and a second device 402. The first and seconddevices 401, 402 may be user devices such as smartphones, tablets, AR / VR headsets or otherportable devices. In contrast to known systems which primarily perform inference on GPUssuch as Nvidia A100, Geforce 3090, Gefore 4090 GPU cards, the system 400 of Figure 4performs inference on a CPU of the first and second devices respectively. That is, computefor performing both encoding and decoding are performed by the respective CPUs of the firstand second devices 401, 402. This places very different power usage, memory and runtimeconstraints on the implementation of the above methods than when implementing AI-basedcompression methods on GPUs. In one example, the CPU of first and second devices 401, 402may comprise a Qualcomm Snapdragon CPU.The first device 401 comprises a media capture device 403, such as a camera, arranged tocapture a plurality of images, referred to hereafter as a video stream 404, of a scene 404. Thevideo stream 404 is passed to a pre-processing module 406 which splits the video stream intoblocks of frames, various frames of which will be designated as I-frames, P-frames, and / orB-frames. The blocks of frames are then compressed by an AI-compression module 407comprising the encode side of the AI-based video compression pipeline of Figure 3. Theoutput of the AI-compression module is accordingly a bitstream 408a which is transmittedfrom the first device 401, for example via a communications channel, for example over oneor more of a WiFi, 3G, 4G or 5G channel, which may comprise internet or cloud-based 409communications.The second device 402 receives the communicated bitstream 408b which is passed to anAI-decompression module 410 comprising the decode side of the AI-based video compressionpipeline of Figure 3. The output of the AI-decompression module 402 is the reconstructedI-frames, P-frames and / or B-frames which are passed to a post-processing module 411 wherethey can prepared, for example passed into a buffer, in preparation for streaming 412 to andrendering on a display device 413 of the second device 402.It is envisaged that the system 400 of Figure 4 may be used for live video streaming at 30fps ofa 1080p video stream, which means a cumulative latency of both the encode and decode sideis below substantially 50ms, for example substantially 30ms or less. Achieving this level ofruntime performance with only CPU compute on user devices presents challenges which arenot addressed by known methods and systems or in the wider AI-compression literature.For example, execution of different parts of the compression pipeline during inferencemay be optimized by adjusting the order in which operations are performed using one ormore known CPU scheduling methods. Efficient scheduling can allow for operations to beperformed in parallel, thereby reducing the total execution time. It is also envisaged thatefficient management of memory resources may be implemented, including optimising cachingmethods such as storing frequently-accessed data in faster memory locations, and memoryreuse, which minimizes memory allocation and deallocation operations.A number of concepts related to the AI compression processes and / or their implementationin a hardware system discussed above will now be described. Although each concept isdescribed separately, one or more of the concepts described below may be applied in an AIbased compression process as described above.

[0002] Concept 1: Single-channel flow estimationAs described above, two commonly used color spaces in digital image processing are RGB(Red, Green, Blue) and YUV. While RGB is widely used in computer graphics and digitaldisplays, YUV is frequently employed in video compression and transmission.In more detail, RGB is an additive color model that represents colors as a combination of red,green, and blue light. Each color channel is typically represented by an 8-bit value, rangingfrom 0 to 255. The combination of these three channels allows for the representation of a widerange of colors. In the RGB color space, a color is represented as a triplet (R, G, B), where R,G, and B are the intensity values of the red, green, and blue components, respectively. Forexample, (255, 0, 0) represents pure red, (0, 255, 0) represents pure green, and (0, 0, 255)represents pure blue. Black is represented as (0, 0, 0), while white is represented as (255, 255,255). YUV Color SpaceIn contrast, YUV separates the color information into two components: luminance (Y) andchrominance (U and V). The YUV color space takes advantage of the human visual system’ssensitivity to brightness and color information.Luminance (Y)The luminance (or Luma) component (Y) represents the brightness or intensity of a pixel. It iscalculated as a weighted sum of the RGB components:^^ = ^^1^^ + ^^2^^ + ^^3^^where ^^1, ^^2, ^^3 are weights assigned to each color channel based on the human eye’ssensitivity to different wavelengths of light. The green channel is given the highest weightbecause the human eye is most sensitive to green light. A non-limiting example set of weightsmay be:^^ = 0.299^^ + 0.587^^ + 0.114^^The chrominance components (U and V) represent the color information in the YUV colorspace. They are calculated by subtracting the luminance value from the blue and red colorchannels, respectively, for example:^^ = ^^ − ^^^^ = ^^ − ^^YUV can also be defined in a digitised representation referred to as BT.601. The BT.601standard, also known as CCIR 601, provides recommendations for standard-definition digitalvideo. It specifies the digitization of the YUV color space components using an 8-bitrepresentation, essential for converting analog video signals into a digital format for televisionbroadcasts, DVDs, and early streaming video formats. In digital systems, the Y, U, andV components are represented by 8-bit values, allowing for 256 levels of intensity rangingfrom 0 to 255. This quantization process converts the continuous range of YUV values intodiscrete levels suitable for digital storage and processing. The BT.601 standard defines theconversion formulas from RGB to YUV in a digital context, taking into account digital systemcharacteristics and the need for efficient color information encoding.In one illustrative example, the digital representation of the luminance (Y) component fromRGB values is given by the formula:^^ = 16 + (65.481 · ^^ + 128.553 · ^^ + 24.966 · ^^)The chrominance components may then calculated with the following formulas:^^ = 128 + (−37.797 · ^^ − 74.203 · ^^ + 112.0 · ^^)^^ = 128 + (112.0 · ^^ − 93.786 · ^^ − 18.214 · ^^)In these formulas, R, G and B are the digital values of the red, green, and blue components,normalized to the range of 0 to 1 (e.g., a value of 255 in an 8-bit system is represented as1.0). The constants include offsets (16 for ^^ and 128 for ^^ and ^^) to center the chrominancecomponents, and scaling factors to adjust the amplitude of the signals within the 8-bit range.More generally, in YUV color spaces, the U component represents the difference between theblue channel and the luminance, while the V component represents the difference between thered channel and the luminance.By separating the luminance and chrominance information, YUV allows for more effectivecompression techniques because the human visual system is more sensitive to changes inbrightness than changes in color, so the luminance component can be compressed with lessloss of perceived quality compared to the chrominance components.YUV color space also allows for subsampling of the chrominance components to further reducethe amount of data required for video transmission. Subsampling involves reducing the spatialresolution of the U and V components while preserving the full resolution of the luminancecomponent.Known subsampling schemes include:4:4:4 - No subsampling. The U and V components have the same resolution as the luminancecomponent.4:2:2 - The U and V components are subsampled horizontally by a factor of 2.4:2:0 - The U and V components are subsampled both horizontally and vertically by a factorof 2.Subsampling reduces the amount of color information without significantly impacting theperceived visual quality, as the human eye is less sensitive to color details than brightnessdetails.Consider next the illustrative flow-residual compression pipeline shown in Figure 5, whichshows flow encoder / decoder networks and residual encoder / decoder networks of an AI basedcompression pipeline. This may be, for example, similar to the AI based compression pipelineof the type shown in Figure 3.The flow encoder neural network takes a current image ^^^^ and a previous image ^^^^−1, encodesthese into a flow latent representation which is optionally quantised and entropy encoded andtransmitted as a bit stream. On the decode side, the bitstream is received, entropy decoded intothe flow latent representation that the flow decoder neural network uses as input to producea representation of flow ^^ , e.g. a flow map. The flow map ^^ is applied to a previouslydecoded image ^^^^−1 to generate a warped version of that previously decoded image ^^^^−1,^^ . Thewarped version of the previously decoded image ^^^^−1,^^ is then fed into the residual encodernetwork, together with the current image ^^^^ to produce a residual latent representation which isoptionally quantised and entropy encoded and transmitted as a bit stream. On the decode side,the bitstream is received and entropy decoded back into the residual latent representation ^^ andused by the residual decoder neural network, in combination with information from the warpedversion of the previously decoded image ^^^^−1,^^ , to produce the reconstructed image ^^^^ . In thecase of Figure 5, the information associated with the warped previously decoded image ^^^^−1,^^is optionally first processed by module ^^ , referred to herein after as a composition adapter, forexample to downsample and / or pad it before it is fed into the residual decoder together withthe entropy decoded residual latent representation to produce the final reconstructed image^^^^ . This process may then be repeated for ^^^^+1 and so on to encode, transmit, and decode asequence of frames.An example implementation of the flow encoder neural network 207 is shown in Figure 6.Figure 6 illustrates an example of a flow module, in this case a network 600, configured toestimate information indicative of a difference between an image ^^^^−1 and an image ^^^^ , e.g.flow information. Figure 6 is provided as an example of how such flow information may becalculated between two input or output images.The network 600 comprises a set of layers 601a, 601b respectively for an image ^^^^−1 and animage ^^^^ from respective times or positions in a sequence ^^ − 1 and ^^ of a sequence of imageframes. The set of layers 601a, 601b may define one or more convolution operations and / ornonlinear activations in the layers to sequentially downsample the input images to producea pyramid of feature maps for different levels of coarseness or spatial resolution. This maycomprise performing ℎ / 2 ^^ / 2 downsampling in a first layer, ℎ / 4 ^^ / 4 downsampling in asecond layer ℎ / 8 ^^ / 8 downsampling in a third layer, ℎ / 16 ^^ / 16 downsampling in a fourthlayer, ℎ / 32 ^^ / 32 downsampling in a fifth layer, ℎ / 64 ^^ / 64 downsampling in a sixth layer,and so on. It will of course be appreciated that these downsampling operations and levels ofcoarseness or spatial resolution of a pyramid feature map are exemplary only and others levelsare also envisaged.With the downsampling operations performed and the corresponding pyramid of feature mapsgenerated, a first cost volume 602 is calculated at the most course level between the pixelsof the first image ^^^^−1 and the corresponding pixels of in the second image ^^^^ . Cost volumesdefine the matchmaking cost of matching the pixels in the initial image with the pixels in thelater image. That is, the closeness of each pixel, or a subset of all pixels, in the initial imageto one or more pixels in the later image is determined with a measure of closeness such as avector or dot product, a cosine similarity, a mean absolute difference, or some other measureof closeness. This metric may be calculated against all pixels in the later image, or only forpixels in a predetermined search radius such as a 1-10 pixel radius (preferably a 1, 2, 3, or 4pixel radius), or some other radius as described in connection with concept 4 below, aroundthe pixel coordinate corresponding to the pixel against which the measure of closeness is beingcalculated.Finally, a first flow 603 can be estimated from the first cost volume 602. This may be achievedusing, for example a flow extractor network which may comprise a convolutional neuralnetwork comprising a plurality of layers trained to output a tensor defining a flow map fromthe input cost volumes. Other methods of calculating flow information from cost volumes willalso be known to the skilled person.The same process is then repeated for the other levels of coarseness to calculate a second costvolume 604 and second flow 605, and so on for the cost volumes and flows associated witheach of the levels of coarseness have been calculated, up to the final cost volume 606 and flow607.The weights and / or biases of any activation layers in network 600 (e.g. optionally in thedownsampling convolution layers and / or in a flow extractor network that produces flow mapsfrom the cost volumes) are trainable parameters and can accordingly be updated during trainingeither alone, or in an end to end manner with the rest of the compression pipeline. The trainablenature of these parameters provides the network 600 with flexibility to produce feature maps ateach level of spatial resolution (i.e. pyramid feature maps) and / or at the flow outputs that areforced into a distribution that best allows the network to meet its training objective (e.g. bettercompression, better reconstruction accuracy, more accurate reconstruction of flow, and so on).For example, it allows the network 600 to produce feature maps that, when cost volumes and / orflow are calculated therefrom, produce cost volumes or flows that are distributed roughlymatching the latent representation distribution that would previously have been expected to beoutput by a dedicated flow encoder module. This effectively allows a dedicated flow encoderto be omitted entirely from the flow compression part of the pipeline, as is shown in theillustrative example of Figure 7a, descibed later.Optionally, for each level of coarseness or resolution, the flow of the previous level or levels ofcoarseness or resolution may be used to warp 608, 609, the feature maps before the cost volumeis calculated. This has the effect of artificially reducing the amount of relative movementbetween the pixels of the t and t - 1 images or feature maps when calculating the cost volumes,reducing flow errors for high movement details. The inventors have realized that removingwarping entirely or in some levels of coarseness or resolution can substantially reduce runtimeof flow calculation while maintaining good levels of flow accuracy.As the warping process uses inputs from different levels of coarseness or spatial resolution, theflow estimation output may be upsampled 610, 611 first to match the coarseness resolution ofthe feature map to which the flow is being applied in the warping process.The outputs of the flow module may accordingly be one or more cost volumes or somerepresentation thereof, and / or one or more flows or some representation thereof).Note that whilst the term cost volume has been used above, the information indicative ofdifferences between the respective inputs need not be a strict cost volume in the mathematicalsense, but may be any representation of this information. For example a compressivelycalculated cost volume, applying for example the principles of compressive sensing to estimatean approximate cost volume by sampling only a small number of pixel differences comparedto performing a complete pixel-wise cost volume calculation. This compressively calculatedcost volume approach may be applied to all embodiments described herein.As described above, it will be appreciated that running a flow-based compression pipeline oncetraining has been completed relies on estimation of flow, whether by handcrafted algorithmor through some trained network. The output estimated flow itself may be compressed andtransmitted in a bitstream, which is typically done by a dedicated flow encoder networkthat encodes the flow information into a latent representation distributed according to adistribution that can be efficiently entropy encoded. Irrespective of the flow estimationapproach taken, dedicated flow encoders increase run time and partly contribute to preventinglearned compression codecs from running in real time or near real time.Consider for example the pipeline of Figure 3. At the encode side, a previously reconstructedimage and a new, to-be-encoded image may be used for generating a flow maps containinginformation indicative of inter-frame movement of pixels between frames. In the example ofFigure 3, both the current image being compressed ^^^^ and the previously reconstructed imagefrom an earlier frame ^^^^−1 are passed into a flow module 207 that typically has two parts: aflow estimation part, and a dedicated encoder part that encodes the estimated flows into alatent representation that can be efficiently entropy encoded. The dedicated flow encoder iseffectively a specialized network dedicated to producing latent representations of flow that aredistributed as close to an optimally entropy encodable distribution as possible. In Figure 3,these two parts are shown as a single component.The flow estimation part may comprise, for example, the flow module 600 of Figure 6. That is,the flow module 600 produces its cost volumes and flows, then passes one or more of theseinto the second part: the dedicated flow encoder network that is trained to produce the latentrepresentation of flow that can be efficiently entropy encoded before being sent in a bitstreamto the decoder. This approach is slow because we first have to calculate the cost volumes andflows before we can encode them into a latent representation, which itself is a slow process. Wethen do the entropy encoding of the latent representation of the cost volume(s) and / or flow(s)into a bitstream which is finally transmitted. This multi-step approach increases run time.Instead, in order reduce compute and runtime overhead, the present disclosure also envisagesomitting the dedicated flow encoder. For example, the outputs of the flow module 600 output(s)are directly entropy encoded and fed into the bitstream without first encoding them into alatent representation. Given that the process of encoding flow module output(s) into a latentrepresentation with a flow compression encoder is computationally expensive, removing thiscomponent entirely from the flow compression module 207 results in a significant decrease inruntime, thereby contributing to the goal of being able to run the pipeline in inference in realtime or near real time.This is, in part, made possible by virtue of the trainable nature of the flow module 600,whereby the weights and biases of one or more of the convolution and / or activation layers ofthe flow module 600 result in cost volumes or flows that are already distributed according to adistribution that corresponds roughly to that which a dedicated flow encoder may produce.This is illustrated in more detail in Figure 7a.Figure 7a illustratively shows two flow compression modules 700a and 700b which may be usedas flow module 207 in Figure 3. The same reference numerals are used for like-components.In the first module 600a, two input images ^^^^ and ^^^^−1 are passed 701a, 701b are input into aflow module such as flow module 600 of Figure 6, which produces pyramid feature maps 702of different levels of coarseness, and corresponding cost volumes and / or flows 703. The finalflow estimation is then passed to a dedicated flow compression encoder 704 which encodesit to produce a latent representation of the final flow estimation. The output may thus be alatent representation of one or more of a H W, H / 2 W / 2, H / 4 W / 4, H / 8 W / 8, H / 16 W16. H / 32W / 32, H / 64 W / 64, or some other resolution, cost volumes and / or flows. This is finally entropyencoded into a bitstream 705, and transmitted. The bitstream 705 is entropy decoded andthe decoded bitstream is passed to a decoder 606 which reconstructs the cost volumes and / orflows which may be used in a flow-based approach as described above in the general conceptssection.However, as described above, the approach of the flow compression module 700a with adedicated flow compression encoder 704 is slow and computationally expensive.In contrast, the present inventors have realized that omitting the dedicated flow compressionencoder 704 entirely and instead directly entropy encoding one or more outputs of the flowmodule 600 into the bitstream 705 without first passing it through the dedicated flow encoder704 results in a substantial speed up at run time. Counter-intuitively, this removal of thededicated flow compression encoder 704 does not appreciably appear to effect the performanceof the rest of the compression pipeline both in terms of distortion and bit rate. This modifiedapproach is illustrated with flow compression module 700b in Figure 7a where it is apparentthat the flow encoder 704 of flow compression module 700b has effectively been chopped out.Thus, instead of encoding the cost volumes and / or flows into a latent representation beforethe entropy encoding step, the one or more of a H W, H / 2 W / 2, H / 4 W / 4, H / 8 W / 8, H / 16W16. H / 32 W / 32, H / 64 W / 64, or some other resolution, cost volumes and / or flows are simplyentropy encoded directly and sent out in the bitstream.This approach is based on the insight that flow modules such as flow module 600 have trainableparameters and accordingly have a great deal of flexibility in terms of the distribution ofoutputs they can be trained to produce. The inventors have found that when the dedicated flowcompression encoder 704 is included, the flow module 600 has no need to produce outputs ina distribution that can be efficiently entropy encoded because the neural networks that make upthe compression pipeline as a whole are simply able to rely on the dedicated flow encoder 704to minimize any contribution to bitrate that the flow information has in the bitstream. However,when the dedicated flow compression encoder 704 is removed, networks of the compressionpipeline are no longer able to rely on the dedicated flow encoder 704. In its place, the inventorshave found that the flow module 600 learns during training to compensate by outputting costvolumes and / or flows that are similarly distributed as those that would be output by a dedicatedflow compression encoder 704. This can be understood as the flow module 600 (that is, thetrainable networks within it such as the flow extractor network and / or activation layers in theconvolution layers) effectively being forced to mimic the dedicated flow compression encoder704 during training when the loss is being minimized because the system can no longer rely onthe (now removed) dedicated flow compression encoder 704 to perform that task.It is envisaged that the training of the flow module 600 may either be performed in an end-to-endmanner together with the rest of the compression pipeline or alternatively in a student-teacherapproach where the network of the flow module 600 is the student component, and a knownoptical flow model and pre-trained flow compression network is the teacher component.Additionally or alternatively, the training of the flow module 600 may be performed separatelyusing data on which the groundtruth flow is known. For example, by using 3D animation videodata whose groundtruth flow is known a priori through the animation program used to generateit, or using auto flow genreation methods.This counter-intuitive removal of the dedicated flow compression encoder 704 from the flowcompression module to force the other components to effectively take on the tasks previouslyperformed by the dedicated flow compression encoder 704 contributes significantly to aspeed up at run time speed on the encoding side of the compression pipeline. This approachaccordingly makes a substantial stride forward towards the goal of achieving real time or nearreal time performance during inference.Figure 7b illustrates the introduction of the flow compression modules 700b into a flow-residualcompression pipeline, such as that of Figure 3, by illustratively showing the encoding anddecoding of p-frames of an image stream, whereby a groundtruth image and its previouslyencoded and decoded reconstruction at t-1 are available. This corresponds generally to theflow-residual compression pipeline shown in Figure 3 and accordingly uses the same referencenumbers for corresponding features. However in this case, the output of the flow module 702is natively in the distribution of a latent representation and can be immediately transmitted tothe decoder, without needing a standalone dedicated flow compression encoder. In addition, itis envisaged that the warped reconstructed, previously decoded image from t-1 may be passeddirectly to the residual decoder, in addition to whatever is output by the residual encoder 211.This provides the residual decoder with the additional context of the warped reconstructed,previously decoded image.Returning now to the YUV colour space, the presently described pyramidal estimation offlow in AI-based compression pipelines provides some unexpected synergies when workingin YUV colour space. Traditionally, AI-based compression research is performed on RGBdata in RGB space. In RGB space, as the input tensors (with R, G and B channels) propagatethrough the networks, the channels in the tensor are all treated equally. For example, duringflow estimation, such as described above in connection with Figures 6 to 7b, the pyramid layersand associated operations are performed treatimg the R, G and B channels of the input tensorsequally. This is because, in RGB space, all channels contain an approximately equal amountof movement information in a frame sequence.The present inventors have realised that, when moving out of RGB space and into YUVspace, the flow information is largely contained in the luma channel. Accordingly, flow canbe estimated using the flow module operating only on the luma channel. Cutting the numberof channels that get fed through the pyramid layers from three channels to single channelresults in a substantial speed up with very little loss in overall performance in terms of imagereconstruction accuracy (e.g. measured by distortion such as an MSE score) and compressionperformance (e.g. measured in in bits per pixel).Accordingly the present concept 1 is directed to replacing a multi-channel input tensor to theflow modules shown in Figures 6 to 7b with a single-channel tensor. Whilst this is envisagedto be the luma channel when working with YUV data, this concept may also be generalised toother types of data where information in one of the channels contains enough information tosufficiently estimate flow, as well as the generation of custom data types comprising a pluralityof channels where one of the channels is optimised for enabling an AI-based flow module suchas that shown in Figures 6 to 7b to determine flow. For completeness, it is noted that concept 1may also be combined with the other concepts described herein.Figure 8 illustratively shows a modified flow module 800 corresponding to that of Figure 6,like numbered reference are used for like components. Additionally, a pre-processing module801 is introduced before the flow module 800. This may form part of the flow module 800itself or may form part of some other component of the pipeline, for example as a componentof a data intake module (not shown) or other pre-processing modules.The pre-processing module is configured to select the luma channel from the a YUV input tofeed into the flow module 800, after which the flow module 800 operates as described above inrelation to Figure 6.The pseudocode provided below illustratively shows an exemplary operation of the pre-processing module, configured to select a luma channel from an input in YUV format:

[0003] Algorithm 1 Select Luma Channel from YUV Tensorprocedure SelectLumaChannel(^^^^^^^^^^^^^^^^^^^^^^)^^^^^^^^ℎ^^ ← HeightOf(^^^^^^^^^^^^^^^^^^^^^^)^^^^^^^^ℎ ←WidthOf(^^^^^^^^^^^^^^^^^^^^^^) ^^^^^^^^^^ℎ^^^^^^^^^^ ← CreateMatrix(^^^^^^^^ℎ^^, ^^^^^^^^ℎ)for ^^ ← 1 to ^^^^^^^^ℎ^^ dofor ^^ ← 1 to ^^^^^^^^ℎ do^^^^^^^^^^ℎ^^^^^^^^^^ [^^] [ ^^] ← ^^^^^^^^^^^^^^^^^^^^^^ [1] [^^] [ ^^]end forend forReturn: ^^^^^^^^^^ℎ^^^^^^^^^^end procedureIt will be apprecaited that the above example is illustrative only and other methods ofimplementing a luma channel selection method will be known by the skilled person.Concept 2: Downsampled YUV warpingOne issue that arises in the implementation of concept 1 described above is the facilitatingof warping operations (for example as shown in Figure 6) performed during flow estimationwhen working in YUV space.In traditional compression, warping is used to exploit temporal redundancy between consecutiveframes. The goal of warping is to estimate and compensate for the motion of objects or regionswithin a video sequence, allowing for more efficient compression by reducing the amount ofinformation that is to be encoded. A frame is typically divided into smaller blocks, such asmacroblocks or coding units, which are then processed independently. Warping is performedon these blocks to find the best matching block in a reference frame, usually a preceding orsucceeding frame, and to generate a motion vector that describes the displacement between thecurrent block and its best match.The warping process can be described mathematically using a motion model. One commonlyused motion model is the affine motion model, which assumes that the motion of a block canbe represented by a linear transformation. The affine motion model is defined by a 2x3 matrix:^^ ^^^^11^^12^^^^^^^^^^ where ^^11, ^^12, ^^21, ^^22 represent the and shear parameters, and ^^^^ , ^^^^ representthe translation parameters.Given a pixel (^^, ^^) in the current block, its corresponding location (^^′, ^^′) in the referenceframe can be computed using the affine transformation:^^ ^′^^^ ^ ^^^^ ^^ ^^^^^1^^^ ^^^^^^^^^^1 12^ ^^^^^^^^^^^ The motion estimation process involves searching for the best matching block in the referenceframe that minimizes a certain distortion metric, such as the sum of absolute differences (SAD)or the sum of squared differences (SSD). By way of example, the SAD metric can be expressedas: ∑^^−^1 ^∑^−^1^^^^^^ (^^, ^^) = (^^, ^^) − ^^(^^ + ^^, ^^ + ^^) |where ^^ (^^, ^^) ^^, ^^ + ^^) represents thepixel values of the candidate block in the reference frame shifted by (^^, ^^), and ^^ is the blocksize.The motion estimation search can be performed using various algorithms, such as full search,three-step search, or diamond search, to find the motion vector (^^^^, ^^^^) that minimizes thedistortion metric:(^^^^, ^^^^) = arg(m^^i,^n^)^^^^^^ (^^, ^^)Once the motion vector is determined, the current block can be reconstructed by copying thepixels from the reference frame at the displaced location indicated by the motion vector. Thisprocess is known as motion compensation.However, the affine motion model has limitations in representing complex motion patterns,such as non-rigid or deformable objects. More advanced motion models, such as the projectivemotion model or the elastic motion model, can be employed in traditional compression as analternative. These models introduce additional parameters to capture more complex motionpatterns at the cost of increased computational complexity.The projective motion model, for example, is defined by a 3x3 homography matrix:^^^^^′^^ ^^ℎ11ℎ12ℎ13^^ ^^^^^^^ ^^ ^^ ^ ^ ^^ ^ ^ ^^ ^^ ^^ ^^^^^^ ^^ ^^1^^ where ^^′ is a scaling factor, and the final coordinates are obtained by dividing ^^′ and ^^′ by ^^′.Elastic motion models, such as the free-form deformation (FFD) model, allow for even moreflexibility in representing complex motion. FFD models define a deformation grid over theimage and use spline interpolation to compute the displacement of each pixel based on thegrid points.The choice of motion model depends on the characteristics of the video content and thedesired trade-off between compression efficiency and computational complexity. In practice,video compression standards, such as H.264 / AVC or H.265 / HEVC, often use hierarchicalmotion estimation and compensation techniques, where the video frames are decomposed intoa pyramid of resolutions, and motion estimation is performed at each level to capture bothlarge-scale and small-scale motion.In contrast, in AI-based compression, warping has a more indirect role: facilitating theestimation of a more accurate representation of flow at different resolutions using trainedneural network pyramid layers, as shown in e.g. Figure 6, which in turn can facilitate the moreaccurate reconstruction by a fully neural network based residual encoder / decoder module, asshown in Figure 6b.The present inventors have realised that there are some improvements that can be made toAI-based flow estimation when working in YUV 4:2:2 and / or 4:2:0 space. For example,when the inputs into the flow encoder of Figure 6 are luma (Y) channels in full resolution,and the associated UV channels are in a non-full resolution (using for example the 4:2:0, or4:2:2 subsampling scheme), the output flow representation that is fed into the warping step isin full-resolution. Downstream, the final output that is eventually derived from the warpedframe is to be assembled from the luma and chroma information. Accordingly, the chromainformation still has to be reintroduced and recombined with the luma information in some way.That is, if the chroma information is missing entirely from the outputs of the flow estimationsteps then it would have to be sent separately in the bitstream so that the decode side gets thechroma information one way or another. Instead of sending chroma information separately, itis combined again with the luma information during warping.Accordingly, the luma (Y) channel information from which flow is being estimated isrecombined with the chroma (UV) channel information before warping is performed on thecombined YUV information. The present inventors have realised that the overall effect onimage reconstruction accuracy and compression performance that the warping step provides inthe architecture of Figure 6 is approximately the same irrespective of whether the warpingstep is performed in the full-resolution of the luma (Y) channel or in the half-resolution of thechroma (UV) channels. However performing warping in the half-resolution of the chroma(UV) channels is substantially faster computationally. Accordingly, the full-resolution luma(Y) channel information and any flow information derived therefrom can be downsampled tomatch the lower resolution of the (UV) channels before the warping step is performed in thislower resolution. The resulting warped, combined YUV information is then used to estimatethe flow for the next resolution layer of the feature pyramid.The same approach may be applied to each resolution layer of the feature pyramids of thearchitecture of Figure 6 to provide an overall speed up of the flow encoder module of 1-10milliseconds across a wide variety of different hardware devices, thereby contributing to thegoal of real time encoding and decoding speeds.The above described approach can also be generalised beyond YUV to any input data withmultiple channels where one or more of the channels is in a different resolution to the rest ofthe channels in the input data.Figure 9 illustratively shows an example modified flow module 900 of the present disclosuredemonstrating a non-limiting implementation example of concept 2. As with Figure 8, theinput ^^^^−1 and ^^^^ are pre-processed to select respectively the luma and chroma channels. Thismay be performed with separate pre-processing modules 901a, 901b, or with a single moduleor with some other component of a data intake module of the pipeline, as will be appreciatedby the skilled person. The luma channel is fed into the feature pyramid layers as in Figure6, but the chroma channels are instead fed directly into the warping steps 608, 609. For thedownsampling pyramid layer where the native chroma channel resolution matches the lumachannel resolution, no further downsampling of the chroma channel is performed. For thedownsampling pyramid layer where the native chroma channel resolution does not match theluma channel resolution, a downsampling step (not shown) may be performed on the chromachannels so that the warping is performed using matching luma and chroma channel resolutions.The operation of the pre-processing modules 901a and 901b to select the respective luma andchroma channels may correspond to that described in relation to concept 1 above.It will further be appreciated that concepts 1 and 2 may also be combined alone or togetherwith concept 3 described below.Concept 3: Channel dimension stuffingIn traditional AI-based compression, operating in RGB space with equal resolution R, G and Bchannels, the implementation of the convolution operations of any of the neural networks isstraightforward and requires no special consideration as the shape of the tensor is defined bythree channels of equal dimensions. The same consideration applies to 4:4:4 YUV input datawhere each of the Y, U and V channels are in the same resolution so the tensor comprises threechannels of equal dimensions. However, the situation is different when considering 4:2:2 or4:2:0 YUV data because the chroma channels (UV) are of different dimensions to the the lumachannel (Y). The shape of the tensor is accordingly more complicated and convolutions of theneural networks of the AI-based compression pipeline cannot be applied in a straightforwardway to a non-uniformly shaped tensor.The present concept 3 is directed to solving this problem.One approach is to upsample the two chroma (UV) channels to match the resolution of theluma (Y) channel to produce a uniformly shaped input tensor (effectively pre-processing YUV4:2:0 or YUV 4:2:0 into YUV 4:4:4 data) before performing any convolutions on the, nowuniformly shaped, tensor.This approach is illustrated in more detail in the following pseudocode:Algorithm 2 Upsample YUV 4:2:0 Tensor to YUV 4:4:4 Tensorfunction upsample_yuv420_to_yuv444_tensor(yuv420_tensor)⊲ Assuming yuv420_tensor has shape [height, width, 3]⊲ where the last dimension represents Y, U, and V channelsheight, width, _ ← shape(yuv420_tensor)⊲ Extract the Y, U, and V channels from the yuv420_tensory_channel ← yuv420_tensor[:, :, 0]u_channel ← yuv420_tensor[::2, ::2, 1]v_channel ← yuv420_tensor[::2, ::2, 2]⊲ Perform upsampling for U and V channelsupsampled_u_channel ← upsample(u_channel, scale_factor=2)upsampled_v_channel ← upsample(v_channel, scale_factor=2)⊲ Stack the Y channel with the upsampled U and V channelsyuv444_tensor ← stack([y_channel, upsampled_u_channel, upsampled_v_channel], axis=-1)Return: yuv444_tensorend function

[0004] That is, the method may comprise receiving an input tensor in YUV 4:2:0 format, this maybe, for example, the format in the input data has been captured natively by an image capturedevice, whereby the input tensor has a height dimension and width dimension representing x-ycoordinates in an image comprising a plurality of pixels, and whereby the input tensor hasthree channels made up of a luma channel (Y) and two chroma channels (UV).The shape of the tensor is extracted and upsampling is performed using a scale factor of 2 onthe U and V channels. The Y channel is left alone. The Y channel is then stacked with the nowupsampled U and V channels and the resulting tensor is returned. The returned tensor now hasa uniform shape where all the channels have the same resolution and accordingly convolutionscan be performed on the tensor in the usual way.The upsampling may comprise one or more of: nearest neighbour interpolation, bilinearinterpolation, bicubic interpolation, a transposed convolution (deconvolution) or any otherupsampling technique known to the skilled person.One problem with this approach however is that it can be slow. This is because (i) upsamplingtwo whole channels requires additional computation time and (ii) the rest of the pipeline thenoperates in a higher resolution which means there are more computations overall (e.g. inthe flow estimation, the warping, the residual estimation, and so on). This first approach isaccordingly referred to as a naive approach as focuses on simplicity and does not take intoaccount any synergies that YUV 4:2:0 may have with an AI-based compression pipeline.However this approach is computationally slow and substantially increases run time as theentire compression pipeline would be running in the higher YUV 4:4:4 space.Another naive approach is to achieve matching tensor dimensions by introducing two or moreadditional convolution operations to an input layer of the pipeline that operate with one stridefor the Y channel and a different stride for the UV channels. For example, performing aconvolution on the Y channel with a stride of 4 and a convolution on the UV channels with astride of 2 to produce Y, U and V channels of equal dimensions that can then be summed toproduce the overall uniformly shaped YUV tensor to feed into the rest of the neural networksin the usual way.This approach is illustrated in the following pseudocode:

[0005] Algorithm 3 Perform Convolutions on YUV 4:2:0 Tensorfunction convolve_yuv420_tensor(yuv420_tensor)⊲ Assuming yuv420_tensor has shape [height, width, 3]⊲ where the last dimension represents Y, U, and V channels⊲ and the U and V channels have half the resolution of the Y channelheight, width, _ ← shape(yuv420_tensor)⊲ Define the convolution kernels for Y and UV channelsy_kernel← get_convolution_kernel(stride=4) uv_kernel← get_convolution_kernel(stride=2) ⊲Extract the Y, U, and V channels from the yuv420_tensory_channel ← yuv420_tensor[:, :, 0]u_channel ← yuv420_tensor[::2, ::2, 1]v_channel ← yuv420_tensor[::2, ::2, 2]⊲ Perform convolution on the Y channel with stride 4convolved_y_channel ← convolve(y_channel, y_kernel, stride=4)⊲ Perform convolution on the U and V channels with stride 2convolved_u_channel ← convolve(u_channel, uv_kernel, stride=2)convolved_v_channel ← convolve(v_channel, uv_kernel, stride=2)⊲ sum the convolved Y, U, and V channelsconvolved_tensor ← sum([convolved_y_channel, convolved_u_channel, convolved_v_channel], axis=-1)Return: convolved_tensorend functionThat is, the method may comprise receiving an input tensor in YUV 4:2:0 format, this maybe, for example, the format in the input data has been captured natively by an image capturedevice, whereby the input tensor has a height dimension and width dimension representing x-ycoordinates in an image comprising a plurality of pixels, and whereby the input tensor hasthree channels made up of a luma channel (Y) and two chroma channels (UV). The shape ofthe tensor is identified and the respective convolution kernels for the Y channel and for theUV channels are identified. The Y, U and V channels are identified, the stride 4 convolutionis performed on the Y channel, and the stride 2 convolution is performed on the U and Vchannels. The resulting convolved channels now have matching dimensions so can be summedto produce an overall uniformly shaped tensor which can be fed into the neural networks of theAI-compression pipeline in the usual way.This approach is an improvement over the first naive approach because the resulting tensorafter the convolution does not have upsampled U and V channels so the runtime of the overallpipeline is faster.However, the present inventors have realised that a third approach exists that is substantiallyfaster than either of the above naive approaches, and synergistically also results in improvedaccuracy in terms of image reconstruction and performance in terms of compression rates.Thus providing a way to handle the non-uniform shape of the YUV 4:2:0 data in a way thatactually improves overall performance of the compression pipeline.This third approach is to emulate a single convolution operation on the non-uniform input byperforming two different space to depth pixel unshuffle operation on the non-uniform inputtensor. Specifically, a space to depth pixel unshuffle operation with a block size of 4 is appliedto the luma channel, and a space to depth pixel unshuffle operation with a block size of 2 isapplied to each of the chroma channels. These are then stacked to produce the final tensorcomprising 24 channels. That is, the space to depth pixel unshuffle operation with a block sizeof 4 takes the luma channel and applies a 4x4 block to the pixels of the luma channel whichare distributed in the channel dimension to produce 16 channels. Similarly, the space to depthpixel unshuffle operation with a block size of 2 takes the U channel and applies a 2x2 blockto the pixels of the U channel and distributes these in the channel dimension to produce 4channels, and the same applies to the V channel which produces 4 more channels to give anoverall 24 channels.This approach results in two advantages over the above-described naive approaches.Firstly, the present inventors have realised that, on many hardware devices, it is faster toperform operations on data that has a smaller spatial resolution and larger channel dimensionthan it is to perform the same operations on larger spatial resolutions but smaller channeldimensions. Accordingly, even though the input and outputs of the overall pipeline are thesame, converting the input YUV data into a smaller spatial resolution format with more channeldimensions results in a significant speed up of the overall pipeline of the order of milliseconds.Secondly, the space to depth pixel unshuffle operations have the synergistic effect of increasingthe receptive field of any subsequent convolution operations of the neural networks asthey perform their respective convolutions on the input tensor (for example, in the flowencoder / decoder and / or residual encoder / decoder). That is, in a given convolution window,the output can only be influenced by whatever information is in the input. If a convolutionwindow is only a single ^^^^^^ grid of pixels of a single channel, the receptive field is the single^^^^^^ grid of pixels of that channel. If we add additional channels to that convolution window,the output can now be influenced by the additional channel information. If we distribute pixelsthat originated from outside the ^^^^^^ grid into one or more additional / new channels that areincluded in the convolution window, then we are allowing the output to be influenced by thesenew pixels from outside the ^^^^^^ grid, thus indirectly increasing the receptive field of theconvolution without needing to increase the spatial dimensions of the ^^^^^^ grid.In turn, the increased receptive field allows the neural networks of the pipeline to harnessspatial correlations between pixels better to thereby more efficiently learn what information isredundant and can be compressed away, thereby achieving improved compression rate andimproved image reconstruction accuracy for a given bit rate.Thus, in the presently described example, distributing the spatial dimension information intothe channel dimensions using the applicable space to depth pixel unshuffle operations on therespective Y and UV channels solves the problem of applying convolutions throughout thepipeline to the non-uniformly shaped YUV 4:2:0 input tensor while, at the same time, theresulting 24-channel tensor achieves improved compression rates and image reconstructionaccuracy in a way that is computationally efficient and achieves runtime speed ups of the orderof milliseconds.The above approach is illustrated in more detail in the pseudocode below:Algorithm 4 Pixel Unshuffle YUV 4:2:0 Tensorfunction pixel_unshuffle_yuv420_tensor(yuv420_tensor)⊲ Assuming yuv420_tensor has shape [height, width, 3]⊲ where the last dimension represents Y, U, and V channels⊲ and the U and V channels have half the resolution of the Y channelheight, width, _ ← shape(yuv420_tensor)⊲ Extract the Y, U, and V channels from the yuv420_tensory_channel ← yuv420_tensor[:, :, 0]u_channel ← yuv420_tensor[::2, ::2, 1]v_channel ← yuv420_tensor[::2, ::2, 2]⊲ Apply pixel unshuffle to the Y channel with block size 4unshuffled_y_tensor ← pixel_unshuffle(y_channel, block_size=4)⊲ Apply pixel unshuffle to the U channel with block size 2unshuffled_u_tensor ← pixel_unshuffle(u_channel, block_size=2)⊲ Apply pixel unshuffle to the V channel with block size 2unshuffled_v_tensor ← pixel_unshuffle(v_channel, block_size=2)⊲ Stack the unshuffled Y, U, and V tensors along the channel dimensionoutput_tensor ← stack([unshuffled_y_tensor, unshuffled_u_tensor, unshuffled_v_tensor], axis=-1)Return: output_tensorend functionThat is, the method may comprise receiving an input tensor in YUV 4:2:0 format, this maybe, for example, the format in the input data has been captured natively by an image capturedevice, whereby the input tensor has a height dimension and width dimension representing x-ycoordinates in an image comprising a plurality of pixels, and whereby the input tensor hasthree channels made up of a luma channel (Y) and two chroma channels (UV). The shape ofthe tensor is identified and the Y, U and V channels extracted. A pixel unshuffle operationwith block size 4 is applied to the Y channel, a pixel unshuffle operation with block size 2 isapplies to the U channel, and a pixel unshuffle operation with block size 2 is applied to the Vchannel. The resulting 16 channel Y tensor, 4 channel U tensor and 4 channel V tensor arestacked to produce a 24 channel tensor of uniform shape which can be fed into the flow and / orresidual modules of the AI-based compression pipeline as before.It will be appreciated that the above described block sizes and dimensions for the reshapingoperations (e.g. a 24 channel tensor of uniform shape and so on) are illustrative only, and otherdimensions and uniform tensor shapes are also envisaged.Further, pixel unshuffle is only one reshaping operation that may be used. The above approachmay be generalised to any tensor reshaping operations that achieve the same effect. Thus theabove algorithm may be written in general pseudocode form as:

[0006] Algorithm 5 Generalized Reshape and Permutefunction reshape_and_permute_tensor(input_tensor, spatial_dims, block_sizes)input_shape ← shape(input_tensor) num_spatial_dims ← length(spatial_dims) output_shape ← [] for ^^ ← 1 to num_spatial_dims dodim_index← spatial_dims[i] block_size ← block_sizes[i] output_shape.append(input_shape[dim_index] / / block_size)output_shape.append(block_size) end forfor ^^ ← 1 to length(input_shape) doif ^^ not in spatial_dims thenoutput_shape.append(input_shape[i]) end ifend forreshaped_tensor ← reshape(input_tensor, output_shape)permute_dims ← [i for i in range(1, num_spatial_dims+1)] +[i+num_spatial_dims for i in range(1, num_spatial_dims+1)] +[i for i in range(1, length(output_shape)+1) if i not in permute_dims]permuted_tensor ← permute(reshaped_tensor, permute_dims)final_shape ← [output_shape[i] for i in range(num_spatial_dims)] +[prod(block_sizes) * input_shape[-1]]output_tensor ← reshape(permuted_tensor, final_shape)Return: output_tensorend functionThat is, in general form, the following input information is provided: an input tensor of anydimensions, with shape ^^1, ^^2, ..., ^^^^, channels, where ^^1, ^^2, ..., ^^^^ are the spatial dimensionsand channels is the channel dimension, a variable ^^^^^^^^^^^^^^^^^^^^^^ which is a a list specifying theindices of the spatial dimensions to be reshaped, and the variable ^^^^^^^^^^^^^^^^^^^^ which is a listspecifying the block size for each spatial dimension.The shape of the input tensor is determined and the number of spatial dimensions to bereshaped is calculated. An empty list, ^^^^^^^^^^^^^^ℎ^^^^^^, is intialised to store the shape of theintermediate reshaped tensor. We iterate over the spatial dimensions specified in ^^^^^^^^^^^^^^^^^^^^^^and update the ^^^^^^^^^^^^^^ℎ^^^^^^ list by dividing each spatial dimension by its corresponding blocksize and appending the block size as a new dimension. The remaining dimensions (non-spatialdimensions) from the input tensor are appended to the ^^^^^^^^^^^^^^ℎ^^^^^^ list and we reshape theinput tensor according to the ^^^^^^^^^^^^^^ℎ^^^^^^ to obtain the ^^^^^^ℎ^^^^^^^^^^^^^^^^^^^^. We create a list^^^^^^^^^^^^^^^^^^^^^^ to specify the permutation order for the dimensions of the ^^^^^^ℎ^^^^^^^^^^^^^^^^^^^^and iterate over the spatial dimensions and append the corresponding dimension indicesto ^^^^^^^^^^^^^^^^^^^^^^, followed by the indices of the newly added block size dimensions. Theremaining dimension indices (non-spatial dimensions) are appended to ^^^^^^^^^^^^^^^^^^^^^^ andwe permute the dimensions of the ^^^^^^ℎ^^^^^^^^^^^^^^^^^^^^ according to ^^^^^^^^^^^^^^^^^^^^^^ to obtain the^^^^^^^^^^^^^^^^^^^^^^^^^^^^. The ^^ ^^^^^^^^^^ℎ^^^^^^ is calculated by appending the spatial dimensions from^^^^^^^^^^^^^^ℎ^^^^^^ and the product of the block sizes multiplied by the number of channels and the^^^^^^^^^^^^^^^^^^^^^^^^^^^^ is reshaped according to the ^^ ^^^^^^^^^^ℎ^^^^^^ to obtain the ^^^^^^^^^^^^^^^^^^^^^^^^ . The^^^^^^^^^^^^^^^^^^^^^^^^ is then returned.The above generalised approach accordingly provides a framework for distributing pixelsfrom the spatial dimension into channel dimensions to increase the receptive window of thesubsequent convolution operations of the AI-based compression pipeline in a way that iscompute efficient.The subject matter and the functional operations described in this specification can beimplemented in digital electronic circuitry, in tangibly-embodied computer software orfirmware, in computer hardware, including the structures disclosed in this specification andtheir structural equivalents, or in combinations of one or more of them. The subject matterdescribed in this specification can be implemented as one or more computer programs, i.e.,one or more modules of computer program instructions encoded on a tangible non transitoryprogram carrier for execution by, or to control the operation of, data processing apparatus.Alternatively or in addition, the program instructions can be encoded on an artificially generatedpropagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, thatis generated to encode information for transmission to suitable receiver apparatus for executionby a data processing apparatus. The computer storage medium can be a machine-readablestorage device, a machine-readable storage substrate, a random or serial access memory device,or a combination of one or more of them. The computer storage medium is not, however, apropagated signal.The term “data processing apparatus” encompasses all kinds of apparatus, devices, andmachines for processing data, including by way of example a programmable processor, acomputer, or multiple processors or computers. The apparatus can include special purposelogic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specificintegrated circuit). The apparatus can also include, in addition to hardware, code that createsan execution environment for the computer program in question, e.g., code that constitutesprocessor firmware, a protocol stack, a database management system, an operating system, ora combination of one or more of them.A computer program (which may also be referred to or described as a program, software, asoftware application, a module, a software module, a script, or code) can be written in anyform of programming language, including compiled or interpreted languages, or declarative orprocedural languages, and it can be deployed in any form, including as a stand alone program oras a module, component, subroutine, or other unit suitable for use in a computing environment.A computer program may, but need not, correspond to a file in a file system. A program can bestored in a portion of a file that holds other programs or data, e.g., one or more scripts storedin a markup language document, in a single file dedicated to the program in question, or inmultiple coordinated files, e.g., files that store one or more modules, sub programs, or portionsof code. A computer program can be deployed to be executed on one computer or on multiplecomputers that are located at one site or distributed across multiple sites and interconnected bya communication network.The processes and logic flows described in this specification can be performed by one or moreprogrammable computers executing one or more computer programs to perform functionsby operating on input data and generating output. The processes and logic flows can also beperformed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g.,an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).Computers suitable for the execution of a computer program include, by way of example,can be based on general or special purpose microprocessors or both, or any other kind ofcentral processing unit. Generally, a central processing unit will receive instructions and datafrom a read only memory or a random access memory or both. The essential elements ofa computer are a central processing unit for performing or executing instructions and oneor more memory devices for storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer data to, or both, one or moremass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks.However, a computer need not have such devices. Moreover, a computer can be embedded inanother device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio orvideo player, a VR headset, a game console, a Global Positioning System (GPS) receiver, aserver, a mobile phones, a tablet computer, a notebook computer, a music player, an e-bookreader, a laptop or desktop computer, a PDAs, a smart phone, or other stationary or portabledevices, that includes one or more processors and computer readable media, or a portablestorage device, e.g., a universal serial bus (USB) flash drive, to name just a few.Computer readable media suitable for storing computer program instructions and data includeall forms of non-volatile memory, media and memory devices, including by way of examplesemiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magneticdisks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM andDVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in,special purpose logic circuitry.The subject matter described in this specification can be implemented in a computing systemthat includes a back end component, e.g., as a data server, or that includes a middlewarecomponent, e.g., an application server, or that includes a front end component, e.g., a clientcomputer having a graphical user interface or a Web browser through which a user can interactwith an implementation of the subject matter described in this specification, or any combinationof one or more such back end, middleware, or front end components. The components of thesystem can be interconnected by any form or medium of digital data communication, e.g., acommunication network. Examples of communication networks include a local area network(“LAN”) and a wide area network (“WAN”), e.g., the Internet.The computing system can include clients and servers. A client and server are generally remotefrom each other and typically interact through a communication network. The relationship ofclient and server arises by virtue of computer programs running on the respective computersand having a client-server relationship to each other.While this specification contains many specific implementation details, these should beconstrued as descriptions of features that may be specific to particular examples of particularinventions. Certain features that are described in this specification in the context of separateexamples can also be implemented in combination in a single example. Conversely, variousfeatures that are described in the context of a single example can also be implemented inmultiple examples separately or in any suitable subcombination.Similarly, while operations are depicted in the drawings in a particular order, this should notbe understood as requiring that such operations be performed in the particular order shownor in sequential order, or that all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processing may be advantageous.Moreover, the separation of various system modules and components in the examples describedabove should not be understood as requiring such separation in all examples, and it should beunderstood that the described program components and systems can generally be integratedtogether in a single software product or packaged into multiple software products.

Claims

CLAIMS1. A method for lossy image or video encoding and transmission, and decoding, the methodcomprising the steps of:receiving a first image and a second image at a first computer system;with a first neural network, producing a latent representation of optical flow informationusing the first image and the second image, the optical flow information being indicative of adifference between the first image and the second image;transmitting the latent representation of optical flow information to a second computersystem; with a second neural network, decoding the latent representation of optical flowinformation to produce an approximation of the optical flow information;with a third neural network, producing an output image using the optical flow information,wherein the output image is an approximation of the first image;wherein the first image and the second image comprise data arranged in a respectiveplurality of image channels; andwherein using the first image and the second image to produce a latent representation ofoptical flow information comprises using a subset of the plurality of channels.

2. The method of claim 1, wherein the subset comprises a single channel of the plurality ofchannels of each of the first image and the second image.

3. The method of claim 1 or 2, wherein the plurality of channels of each of the first image andthe second image comprise a luma channel and at least one chroma channel.

4. The method of claim 3, wherein the subset comprises the luma channel.

5. The method of claim 4, wherein the luma channel and the at least one chroma channel aredefined in a YUV colour space.

6. The method of claim 5, wherein the at least one chroma channel has a different resolution tothe luma channel.

7. The method of claim 6, wherein the YUV colour space comprises a YUV 4:2:0 or YUV4:2:2 colour space.

8. The method of any of claims 1 to 7, wherein using the first image and the second image toproduce a latent representation of optical flow information comprises:producing a representation of optical flow information at a plurality of resolutions andusing the representation of optical flow information at the plurality of resolutions to producesaid latent representation of optical flow information.

9. The method of claim 8, wherein a representation of optical flow information at a firstresolution of said plurality of resolutions is based on a representation of optical flow informationat a second resolution of said plurality of resolutions.

10. The method of any of claims 8 to 9, wherein using the first image and the second image toproduce a latent representation of optical flow information comprises:using the representation of optical flow information at one of said plurality of resolutionsto warp a representation of the first image at a different resolution of said plurality of resolutions.

11. The method of claim 10, wherein a representation of optical flow information at a differentone of said plurality of resolutions is based on the warped first image and the second image atone of said plurality of resolutions.

12. The method of any of claims 10 to 11, wherein the representation of optical flow informationis estimated using the subset of the plurality of channels, and wherein the warping is performedusing the plurality of channels.

13. A method for lossy image or video encoding and transmission, the method comprising thesteps of:receiving a first image and a second image at a first computer system;with a first neural network, producing a latent representation of optical flow informationusing the first image and the second image, the optical flow information being indicative of adifference between the first image and the second image;wherein the first image and the second image comprise data arranged in a plurality ofimage channels; andwherein using the first image and the second image to produce a latent representation ofoptical flow information comprises using a subset of the plurality of channels.

14. A method for lossy image or video receipt and decoding, the method comprising the stepsof: receiving a latent representation of optical flow information at a second computer system,the optical flow information being indicative of a difference between a first image and a secondimage each comprising data arranged in a plurality of image channels, the latent representationof optical flow information being produced with first neural network using a subset of theplurality of channels;with a second neural network, decoding the latent representation of optical flowinformation to produce an approximation of the optical flow information; andwith a third neural network, producing an output image using the optical flow information,wherein the output image is an approximation of the first image.

15. A data processing apparatus configured to perform the method of any of claims 1 to 14.

16. A computer program comprising instructions which, when the program is executed by acomputer, cause the computer to carry out the method of any of claims 1 to 14.

17. A computer-readable storage medium comprising instructions which, when executed by acomputer, cause the computer to carry out the method of any of claims 1 to 14.

18. A method for lossy image or video encoding and transmission, and decoding, the methodcomprising the steps of:receiving a first image and a second image at a first computer system;with a first neural network, producing a latent representation of optical flow informationusing the first image and the second image, the optical flow information being indicative of adifference between the first image and the second image;transmitting the latent representation of optical flow information to a second computersystem; with a second neural network, decoding the latent representation of optical flowinformation to produce an approximation of the optical flow information; andwith a third neural network, producing an output image using the optical flow information,wherein the output image is an approximation of the first image;wherein the first image and the second image comprise data arranged in respective lumachannels and chroma channels; andwherein using the first image and the second image to produce a latent representation ofoptical flow information comprises:producing a representation of the optical flow information based on the respectiveluma channels of the first image and the second image; anddownsampling the representation of the optical flow information.

19. The method of claim 18, wherein said downsampling comprises downsamping therepresentation of the optical flow information to a resolution of the respective chroma channelsof the first image and the second image.

20. The method of claim 18 or 19, comprising warping the data in the respective chromachannels using the downsampled representation of the optical flow information.

21. The method of any of claims 18 to 20, comprising warping the data in the respective lumachannels using the downsampled representation of the optical flow information.

22. The method of any of claims 18 to 21, wherein using the first image and the second imageto produce a latent representation of optical flow information comprises:producing a representation of optical flow information from the respective luma channelsat a plurality of resolutions and using the representation of optical flow information at theplurality of resolutions to produce said latent representation of optical flow information.

23. The method of claim 22, wherein a representation of optical flow information at afirst resolution of said plurality of resolutions is based on a representation of optical flowinformation at a second resolution of said plurality of resolutions.

24. The method of claim 23, when dependent on claim 20 or 21 wherein the representation ofoptical flow information at one or more of said plurality of resolutions is based on said warpeddata.

25. The method of any of claims 18 to 24, wherein the respective luma channels and chromachannels are defined in a YUV colour space.

26. The method of any of claim 25, wherein the respective chroma channels have a differentresolution to the luma channel.

27. The method of claim 26, wherein the YUV colour space comprises a YUV 4:2:0 or YUV4:2:2 colour space.

28. A method for lossy image or video encoding and transmission, the method comprising thesteps of:receiving a first image and a second image at a first computer system;with a first neural network, producing a latent representation of optical flow informationusing the first image and the second image, the optical flow information being indicative of adifference between the first image and the second image;transmitting the latent representation of optical flow information to a second computersystem; wherein the first image and the second image comprise data arranged in respective lumachannels and chroma channels; andwherein using the first image and the second image to produce a latent representation ofoptical flow information comprises:producing a representation of the optical flow information based on the respectiveluma channels of the first image and the second image; anddownsampling the representation of the optical flow information.

29. A method for lossy image or video receipt and decoding, the method comprising the stepsof: receiving a latent representation of optical flow information to a second computer system,the optical flow information being indicative of a difference between a first image and a secondimage each comprising data arranged in respective luma channels and chroma channels, thelatent representation of optical flow information being produced with a first neural networkusing a downsampled representation of the optical flow information based on the respectiveluma channels of the first image and the second image;with a second neural network, decoding the latent representation of optical flowinformation to produce an approximation of the optical flow information; andwith a third neural network, producing an output image using the optical flow information,wherein the output image is an approximation of the first image.

30. A data processing apparatus configured to perform the method of any of claims 18 to 29.

31. A computer program comprising instructions which, when the program is executed by acomputer, cause the computer to carry out the method of any of claims 18 to 29.

32. A computer-readable storage medium comprising instructions which, when executed by acomputer, cause the computer to carry out the method of any of claims 18 to 29.

33. A method for lossy image or video encoding and transmission, and decoding, the methodcomprising the steps of:receiving a first image and a second image at a first computer system, the first image andthe second image comprising data arranged in a respective plurality of image channels;transforming the data by distributing information of the image channels from a spatialdimension into a channel dimension;with a first neural network, producing a latent representation of optical flow informationusing the transformed data, the optical flow information being indicative of a differencebetween the first image and the second image;transmitting the latent representation of optical flow information to a second computersystem; with a second neural network, decoding the latent representation of optical flowinformation to produce an approximation of the optical flow information; andwith a third neural network, producing an output image using the optical flow information,wherein the output image is an approximation of the first image.

34. The method of claim 33, wherein the information comprises pixel values of said pluralityof image channels.

35. The method of claim 33 or 34, wherein the plurality of image channels comprises a subsetof channels in a first spatial resolution different to a spatial resolution of the other channels inthe plurality of image channels.

36. The method of any of claims 33 to 35, wherein the transformed data comprises a pluralityof image channels each having a same spatial resolution.

37. The method of claim 36, wherein said same spatial resolution is lower than the spatialresolution of said subset of channels.

38. The method of claim 36 or 37, wherein said same spatial resolution is lower than thespatial resolution of said other channels of the plurality of channels,39. The method of any of claims 33 to 38, wherein the data comprises 3-channel YUV data andwherein said transforming comprises transforming the 3-channel YUV data into 24-channeldata.

40. The method of claim 39, wherein the YUV data comprises one of 4:2:0 YUV data or 4:2:2YUV data.

41. The method of any of claims 33 to 40, wherein said transforming comprises performing apixel unshuffle operation on the data.

42. The method of claim 41, wherein the pixel unshuffle operation is defined by a first blocksize for a first image channel of the data, and defined by a second block size for a second imagechannel of the data .

43. The method of any of claims 33 to 40, wherein said transforming comprises performing aconvolution operation on the data.

44. The method of claim 43, wherein the convolution operation is defined by a first stride for afirst image channel of the data, and defined by a second stride for a second image channel ofthe data.

45. The method of any of claims 33 to 40, wherein said transforming comprises upsampling asubset of said plurality of image channels to produce a plurality of image channels each havinga same spatial resolution.

46. The method of any of claims 33 to 45, wherein one or more of the first, second or thirdneural networks is defined by a convolution operation, and wherein said transforming increasesa receptive field of the convolution operation.

47. A method for lossy image or video encoding and transmission, the method comprising thesteps of:receiving a first image and a second image at a first computer system, the first image andthe second image comprising data arranged in a respective plurality of image channels;transforming the data by distributing information of the image channels from a spatialdimension into a channel dimension;with a first neural network, producing a latent representation of optical flow informationusing the transformed data, the optical flow information being indicative of a differencebetween the first image and the second image;transmitting the latent representation of optical flow information to a second computersystem.

48. A method for lossy image or video receipt and decoding, the method comprising the stepsof: receiving a first image and a second image at a first computer system, the first image andthe second image comprising data arranged in a respective plurality of image channels;transforming the data by distributing information of the image channels from a spatialdimension into a channel dimension;with a first neural network, producing a latent representation of optical flow informationusing the transformed data, the optical flow information being indicative of a differencebetween the first image and the second image;receiving a latent representation of optical flow information at a second computer system,the ;with a second neural network, decoding the latent representation of optical flowinformation to produce an approximation of the optical flow information; andwith a third neural network, producing an output image using the optical flow information,wherein the output image is an approximation of the first image.

49. A method for lossy image or video receipt and decoding, the method comprising the stepsof: receiving a latent representation of optical flow information at a second computer system,the optical flow information being indicative of a difference between a first image and a secondimage each comprising data arranged in a plurality of image channels, the latent representationof optical flow information being produced with a first neural network and by transforming thedata by distributing information of the image channels from a spatial dimension into a channeldimension; with a second neural network, decoding the latent representation of optical flowinformation to produce an approximation of the optical flow information; andwith a third neural network, producing an output image using the optical flow information,wherein the output image is an approximation of the first image.

50. A data processing apparatus configured to perform the method of any of claims 33 to 49.

51. A computer program comprising instructions which, when the program is executed by acomputer, cause the computer to carry out the method of any of claims 33 to 49.

52. A computer-readable storage medium comprising instructions which, when executed by acomputer, cause the computer to carry out the method of any of claims 33 to 49.

Images