Method and data processing system for lossy image or video encoding, transmission and decoding

Abstract

A method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; performing consecutive downsampling operations on the input image to produce a downsampled input image and adding padding to the input image between one or more of the consecutive downsampling operations; encoding the downsampled input image using a first trained neural network to produce a latent representation; transmitting the latent representation to a second computer system; and decoding the latent representation using a second trained neural network to produce a downsampled output image; performing consecutive upsampling operations on the downsampled output image to produce an output image and removing padding between one or more of the consecutive upsampling operations, wherein the output image is an approximation of the input image.

Description

[0001] Method and data processing system for lossy image or videoencoding, transmission and decodingBACKGROUNDThis invention relates to a method and system for lossy image or video encoding, transmissionand decoding, a method, apparatus, computer program and computer readable storage mediumfor lossy image or video encoding and transmission, and a method, apparatus, computerprogram and computer readable storage medium for lossy image or video receipt and decoding.There is increasing demand from users of communications networks for images and videocontent. Demand is increasing not just for the number of images viewed, and for the playingtime of video; demand is also increasing for higher resolution content. This places increasingdemand on communications networks and increases their energy use because of the largeramount of data being transmitted.To reduce the impact of these issues, image and video content is compressed for transmissionacross the network. The compression of image and video content can be lossless or lossycompression. In lossless compression, the image or video is compressed such that all of theoriginal information in the content can be recovered on decompression. However, when usinglossless compression there is a limit to the reduction in data quantity that can be achieved. Inlossy compression, some information is lost from the image or video during the compressionprocess. Known compression techniques attempt to minimise the apparent loss of informationby the removal of information that results in changes to the decompressed image or video thatis not particularly noticeable to the human visual system. JPEG, JPEG2000, AVC, HEVC andAVI are examples of compression processes for image and / or video files.In general terms, known lossy image compression techniques use the spatial correlationsbetween pixels in images to remove redundant information during compression. For example,in an image of a blue sky, if a given pixel is blue, there is a high likelihood that the neighbouringpixels, and their neighbouring pixels, and so on, are also blue. There is accordingly no need toretain all the raw pixel data. Instead, we can retain only a subset of the pixels which take upfewer bits and infer the pixel values of the other pixels using information derived from spatialcorrelations.A similar approach is applied in known lossy video compression techniques. That is, spatialcorrelations between pixels allow the removal of redundant information during compression.However, in video compression, there is further information redundancy in the form of temporalcorrelations. For example, in a video of an aircraft flying across a blue-sky background, mostof the pixels of the blue sky do not change at all between frames of the video. The mostof the blue sky pixel data for the frame at position t = 0 in the video is identical to that atposition t = 10. Storing this identical, temporally correlated, information is inefficient. Instead,only the blue sky pixel data for a subset of the frames is stored and the rest are inferred frominformation derived from temporal correlations.In the realm of lossy video compression in particular, the removal of redundant temporallycorrelated information in a video sequence is known inter-frame redundancy.One technique using inter-frame redundancy that is widely used in standard video compressionalgorithms involves the categorization of video frames into three types: I-frames, P-frames, andB-frames. Each frame type carries distinct properties concerning their encoding and decodingprocess, playing different roles in achieving high compression ratios while maintainingacceptable visual quality.I-frames, or intra-coded frames, serve as the foundation of the video sequence. These framesare self-contained, each one encoding a complete image without reference to any other frame.In terms of compression, I-frames are least compressed among all frame types, thus carryingthe most data. However, their independence provides several benefits, including being thestarting point for decompression and enabling random access, crucial for functionalities likefast-forwarding or rewinding the video.P-frames, or predictive frames, utilize temporal redundancy in video sequences to achievegreater compression. Instead of encoding an entire image like an I-frame, a P-frame representsthe difference between itself and the closest preceding I- or P-frame. The process, known asmotion compensation, identifies and encodes only the changes that have occurred, therebysignificantly reducing the amount of data transmitted. Nonetheless, P-frames are dependent onprevious frames for decoding. Consequently, any error during the encoding or transmissionprocess may propagate to subsequent frames, impacting the overall video quality.B-frames, or bidirectionally predictive frames, represent the highest level of compression.Unlike P-frames, B-frames use both the preceding and following frames as references in theirencoding process. By predicting motion both forwards and backwards in time, B-framesencode only the differences that cannot be accurately anticipated from the previous and nextframes, leading to substantial data reduction. Although this bidirectional prediction makesB-frames more complex to generate and decode, it does not propagate decoding errors sincethey are not used as references for other frames. Artificial intelligence (AI) based compressiontechniques achieve compression and decompression of images and videos through the use oftrained neural networks in the compression and decompression process. Typically, duringtraining of the neutral networks, the difference between the original image and video and thecompressed and decompressed image and video is analyzed and the parameters of the neuralnetworks are modified to reduce this difference while minimizing the data required to transmitthe content. However, AI based compression methods may achieve poor compression resultsin terms of the appearance of the compressed image or video or the amount of informationrequired to be transmitted.An example of an AI based image compression process comprising a hyper-network is describedin Ballé, Johannes, et al. “Variational image compression with a scale hyperprior.” arXivpreprint arXiv:1802.01436 (2018), which is hereby incorporated by reference.An example of an AI based video compression approach is shown in Agustsson, E., Minnen, D.,Johnston, N., Balle, J., Hwang, S. J., and Toderici, G. (2020), Scale-space flow for end-to-endoptimized video compression. In Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition (pp. 8503-8512), which is hereby incorporated by reference.A further example of an AI based video compression approach is shown in Mentzer, F.,Agustsson, E., Ballé, J., Minnen, D., Johnston, N., and Toderici, G. (2022, November). Neuralvideo compression using gans for detail synthesis and propagation. In Computer Vision–ECCV2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, PartXXVI (pp. 562-578), which is hereby incorporated by reference.SUMMARYAccording to an aspect, there is provided a method for lossy image or video encoding,transmission and decoding, the method comprising the steps of:receiving an input image at a first computer system;performing consecutive downsampling operations on the input image to produce adownsampled input image and adding padding to the input image between one or more of theconsecutive downsampling operations;encoding the downsampled input image using a first trained neural network to produce alatent representation;transmitting the latent representation to a second computer system; anddecoding the latent representation using a second trained neural network to produce adownsampled output image;performing consecutive upsampling operations on the downsampled output image toproduce an output image and removing padding between one or more of the consecutiveupsampling operations, wherein the output image is an approximation of the input image.Optionally, the method comprises:adding padding to the input image between some but not all of the consecutive down-sampling operations; andremoving padding from the downsampled output image between some but not all of theconsecutive upsampling operations.Optionally, the method comprises adding padding to the input image before a downsam-pling operation where a dimension being downsampled is indivisible by a predetermineddownsampling amount.Optionally, the method comprises adding padding to the input image before a downsamplingoperation until a dimension being downsampled is divisible by a predetermined downsamplingamount.Optionally, the method comprises comprising:identifying a height and / or width dimension of the input image;calculating a sequence of height and / or width dimensions of the input image afterperforming each of the downsampling operations;generating an add padding schedule, the add padding schedule specifying between whichconsecutive downsampling operations the step of adding padding is to be performed; andusing the add padding schedule to perform said adding padding.Optionally, the method comprises:identifying a height and / or width dimension of the downsampled output image;calculating a sequence of height and / or width dimensions of the downsampled outputimage after performing each of the upsampling operations;generating a remove padding schedule, the remove padding schedule specifying betweenwhich consecutive downsampling operations the step of removing padding is to be performed;and using the remove padding schedule to perform said removing padding.Optionally, the remove padding schedule corresponds to the add padding schedule in reverse.Optionally, the input image has a height dimension of 1080 pixels, wherein the each downsam-pling operation halves the height dimension, and wherein said adding padding is performed inthe height dimension when the input image height dimension is 135 pixels.Optionally, the input image has a height dimension of 720 pixels, wherein the each downsam-pling operation halves the height dimension, and wherein said adding padding is performed inthe height dimension when the input image height dimension is 45 pixels.Optionally, at least one of the consecutive downsampling operations comprises a convolutionoperation performed with one or more convolution layers of the first neural network.Optionally, at least one of the consecutive upsampling operations comprises a convolutionoperation performed with one or more convolution layers of the second neural network.According to an aspect, there is provided a method of training one or more neural networks,the one or more neural networks being for use in lossy image or video encoding, transmissionand decoding, the method comprising the steps of:receiving a first input image at a first computer system;performing consecutive downsampling operations on the input image to produce adownsampled input image and adding padding to the input image between one or more of theconsecutive downsampling operations;encoding the downsampled input image using a first trained neural network to produce alatent representation;decoding the latent representation using a second trained neural network to produce adownsampled output image;performing consecutive upsampling operations on the downsampled output image toproduce an output image and removing padding between one or more of the consecutiveupsampling operations, wherein the output image is an approximation of the input image.evaluating a function based on a difference between the input image and the outputimage; updating the parameters of the first neural network and the second neural network basedon the evaluated function; andrepeating the above steps using a first set of input images to produce a first trained neuralnetwork and a second trained neural network.Optionally, the method comprises:adding padding to the input image between some but not all of the consecutive down-sampling operations; andremoving padding from the downsampled output image between some but not all of theconsecutive upsampling operations.Optionally, the method comprises adding padding to the input image before a downsam-pling operation where a dimension being downsampled is indivisible by a predetermineddownsampling amount.Optionally, the method comprises adding padding to the input image before a downsamplingoperation until a dimension being downsampled is divisible by a predetermined downsamplingamount.Optionally, the downsampling comprises performing one or more convolution operation usingone or more convolution layers of the first neural network, and wherein updating the parametersof the first neural network comprises updating weights of the one or more convolution layersto produce a padding-aware first neural network.Optionally, the upsampling comprises performing one or more convolution operations usingone or more convolution layers of the second neural network, and wherein updating theparameters of the second neural network comprises updating the weights of the one or moreconvolution layers to produce a padding-aware second neural network.According to an aspect, there is provided a data processing system configured to perform anyof the above methods.According to an aspect, there is provided a data processing apparatus configured to performany of the above methods.According to an aspect, there is provided a computer program comprising instructions which,when the program is executed by a computer, cause the computer to carry out any of the abovemethods.According to an aspect, there is provided a computer-readable storage medium comprisinginstructions which, when executed by a computer, cause the computer to carry out any of theabove methods.According to an aspect, there is provided a method for lossy image or video encoding andtransmission, and decoding, the method comprising the steps of:receiving a first image and a second image at a first computer system;with a first neural network, producing a latent representation of optical flow informationusing the first image and the second image, the optical flow information being indicative of adifference between the first image and the second image;quantising the latent representation using information indicative of a dependence of oneor more elements of the latent representation on one or more other elements of the latentrepresentation to produce a quantised normalised latent representation;transmitting the quantised normalised latent representation to a second computer system;reconstructing the latent representation from the quantised normalised latent representa-tion using the information indicative of the dependence of one or more elements of the latentrepresentation on one or more other elements of the latent representation;with a second neural network, decoding the latent representation of optical flow informa-tion to produce an approximation of the optical flow information; andwith a third neural network, producing an output image using the approximation ofoptical flow information, wherein the output image is an approximation of the first image.Optionally, said reconstructing comprises solving a system of linear equations based on theinformation indicative of the dependence of one or more elements of the latent representationon one or more other elements of the latent representation.Optionally, the method comprises:with a fourth neural network, producing a hyper latent representation of the optical flowinformation using the latent representation;transmitting the hyper latent representation to the second computer system; andwith a fifth neural network, producing said information indicative of the dependence ofone or more elements of the latent representation on one or more other elements of the latentrepresentation using the hyper latent representation.optionally, the values of the latent representation are distributed according to a distributiondefined by one or more parameters, and wherein the method comprises:with the fifth neural network, producing the one or more parameters using the hyperlatent representation; andentropy encoding the quantised normalised latent representation into a bitstream usingthe one or more parameters, wherein said transmitting comprises transmitting the bitstream.Optionally, the one or more parameters comprise one or more scale parameters and / or one ormore location parameters of the distribution.Optionally, said quantising comprises performing location shifted quantisation on the latentrepresentation using the one or more parameters and the information indicative of a dependenceof one or more elements of the latent representation on one or more other elements of thelatent representation.Optionally, the method comprises entropy encoding the hyper latent representation into thebitstream.Optionally, the method comprises:entropy decoding a first part of the bitstream to reconstruct the hyper latent representation;with the fifth neural network, decoding the hyper latent representation to produce the oneor more parameters and the information indicative of the dependence of one or more elementsof the latent representation on one or more other elements of the latent representation;entropy decoding a second part of the bitstream to reconstruct the quantised normalisedlatent representation using the one or more parameters; andperforming said reconstructing of the latent representation using the quantised normalisedlatent representation, the one or more parameters, and the information indicative of thedependence of one or more elements of the latent representation on one or more other elementsof the latent representationOptionally, the method comprises:warping a representation of the second image with the approximation of optical flowinformation; andwith a sixth neural network, producing a latent representation of an image residualindicative of a difference between the first image and the second image;wherein producing the output image using the optical flow information comprisesdecoding the latent representation of the image residual with the third neural network.Optionally, the method comprises combining the warped representation of the second imagewith the latent representation of the image residual and decoding the combined warpedrepresentation and latent representation of the image residual with the third neural network.Optionally, the information indicative of the dependence of one or more elements of the latentrepresentation on one or more other elements of the latent representation is defined as a tensorcomprising tensor elements that, when applied to the elements of the latent representationmodify said elements to be distributed according to an emulated autoregressive probabilitymass function.Optionally, producing said information indicative of the dependence of one or more elementsof the latent representation on one or more other elements of the latent representation usingthe hyper latent representation comprises predicting the tensor elements with the fifth neuralnetwork.According to a further aspect, there is provided a method for lossy image or video encodingand transmission,the method comprising the steps of:receiving a first image and a second image at a first computer system;with a first neural network, producing a latent representation of optical flow informationusing the first image and the second image, the optical flow information being indicative of adifference between the first image and the second image;quantising the latent representation using information indicative of a dependence of oneor more elements of the latent representation on one or more other elements of the latentrepresentation to produce a quantised normalised latent representation;transmitting the quantised normalised latent representation to a second computer system.According to a further aspect, there is provided a method for lossy image or video receipt anddecoding, the method comprising the steps of:receiving at a second computer system a quantised normalised latent representation, thequantised normalised latent representation produced from a latent representation of optical flowinformation indicative of a difference between first and second images and being produced usinginformation indicative of a dependence of one or more elements of the latent representation onone or more other elements of the latent representation;reconstructing the latent representation from the quantised normalised latent representa-tion using the information indicative of the dependence of one or more elements of the latentrepresentation on one or more other elements of the latent representation;with a second neural network, decoding the latent representation of optical flow informa-tion to produce an approximation of the optical flow information; andwith a third neural network, producing an output image using the approximation ofoptical flow information, wherein the output image is an approximation of the first image.According to a further aspect, there is provided a data processing apparatus configured toperform any of the above methods.According to a further aspect, there is provided a computer program comprising instructionswhich, when the program is executed by a computer, cause the computer to carry out any ofthe above methods.According to a further aspect, there is provided a computer-readable storage medium comprisinginstructions which, when executed by a computer, cause the computer to carry out any of theabove methods.According to an aspect of the present disclosure, there is provided a method of training one ormore neural networks, the one or more neural networks being for use in lossy image or videoencoding, transmission and decoding, the method comprising the steps of:receiving a first input image at a first computer system;encoding the first input image using a first neural network to produce a latent representa-tion; decoding the latent representation using a second neural network to produce a first outputimage, wherein the first output image is an approximation of the first input image;evaluating a function based on a difference between the first output image and the firstinput image;updating the parameters of the first neural network and the second neural network basedon the evaluated function;repeating the above steps using a first set of input images to produce a first trained neuralnetwork and a second trained neural network; andadding noise to outputs of one or more of the layers of the one or more neural networks duringsaid training.Optionally, the method comprises adding noise to an output of one or more layers of the firstneural network and / or the second neural network.Optionally, the method comprises adding noise to an output of one or more activation layers ofthe first neural network and / or the second neural network.Optionally, the method comprises:encoding the latent representation using a third neural network to produce a first hyperlatent representation;decoding the first hyper latent representation using a fourth neural network to produceinformation associated with a distribution of the latent representation; andwith the information associated with the distribution of the latent representation, entropyencoding and entropy decoding the latent representation.Optionally, the method comprises adding noise to an output of one or more layers of the thirdneural network and / or the fourth neural network.Optionally, the method comprises adding noise to an output of one or more activation layers ofthe third neural network and / or the fourth neural network.Optionally, the output of one or more activation layers of the fourth neural network comprisesa standard deviation parameter associated with the distribution of the latent representation anda mean parameter associated with the distribution of the latent representation, and wherein thenoise is added to the mean parameter not the standard deviation parameter.Optionally, the method comprises:receiving a second input image at a first computer system;with a fifth neural network, producing a latent representation of optical flow informationusing the first input image and the second input image, the optical flow information beingindicative of a difference between the first input image and the second input image;with a sixth neural network, decoding the latent representation of optical flow informationto produce an approximation of the optical flow information; andusing the approximation of the optical flow information to produce a second outputimage, wherein the second output image is an approximation of the second input image.Optionally, the method comprises adding noise to an output of one or more layers of the fifthand / or the sixth neural network.Optionally, the method comprises adding noise to an output of one or more activation layers ofthe fifth and / or the sixth neural network.Optionally, the method comprises:encoding the latent representation of optical flow information using a seventh neuralnetwork to produce a second hyper latent representation;decoding the second hyper latent representation using an eighth neural network toproduce information associated with a distribution of the optical flow information; andwith the information associated with the distribution of the optical flow information,entropy encoding and entropy decoding the latent representation of optical flow information.Optionally, the method comprises adding noise to an output of one or more layers of theseventh neural network and / or the eighth neural network.Optionally, the method comprises adding noise to an output of one or more activation layers ofthe seventh neural network and / or the eighth neural network.Optionally, the output of one or more activation layers of the eighth neural network comprisesa standard deviation parameter associated with the distribution of the optical flow informationand a mean parameter associated with the distribution of the optical flow information, andwherein the noise is added to the mean parameter not the standard deviation parameter.Optionally, the method comprises:receiving a second input image at a first computer system;with a ninth neural network, producing a latent representation of residual informationusing the first input image and the second input image, the residual information being indicativeof a difference between the first input image and the second input image;with an tenth neural network, decoding the latent representation of residual informationto produce a second output image, wherein the second output image is an approximation of thesecond input image.Optionally, the method comprises adding noise to an output of one or more layers of the ninthand / or the tenth neural network.Optionally, the method comprises adding noise to an output of one or more activation layers ofthe ninth and / or the tenth neural network.Optionally, the method comprises:encoding the latent representation of residual information using an eleventh neuralnetwork to produce a third hyper latent representation;decoding the third hyper latent representation using a twelfth neural network to produceinformation associated with a distribution of the latent representation of residual information;and with the information associated with the distribution of the latent representation ofresidual information, entropy encoding and entropy decoding the latent representation ofresidual information.Optionally, the method comprises adding noise to an output of one or more layers of theeleventh neural network and / or the twelfth neural network.Optionally the method comprises adding noise to an output of one or more activation layers ofthe eleventh neural network and / or the twelfth neural network.Optionally, the output of one or more activation layers of the twelfth neural network comprisesa standard deviation parameter associated with the distribution of the latent representationof residual information and a mean parameter associated with the distribution of the latentrepresentation of residual information, and wherein the noise is added to the mean parameternot the standard deviation parameter.Optionally, the noise is defined by a noise function parameterised by one or more variables,wherein the method comprises updating the one or more variables based on the evaluatedfunction, and wherein repeating the above steps comprises repeating the updating of the one ormore variables to produce a learned noise function during said training.Optionally, the noise function is parameterised by one or more standard deviation parametersand / or one or more mean parameters, and wherein repeating the above steps comprisesrepeating the updating of the one or more standard deviation parameters and / or the one ormore mean parameters.Optionally, the noise added to an output of one or more layers of one or more of the first, second,third, fourth, fifth, sixth, seventh, eighth, ninth, tenth, eleventh and / or twelfth neural networksis defined by a corresponding noise function of a plurality of noise functions parameterisedby respective one or more variables, wherein the method comprises updating the respectiveone or more variables based on the evaluated function, and wherein repeating the above stepscomprises repeating the updating of the respective one or more variables to produce a pluralityof learned noise functions.Optionally, at least one noise function of the plurality of noise functions is parameterised byone or more predetermined, fixed variables.According to a further aspect, there is provided a method for lossy image or video encoding,transmission and decoding, the method comprising the steps of:receiving an input image at a first computer system;encoding the input image using a first trained neural network to produce a latentrepresentation; transmitting the latent representation to a second computer system; anddecoding the latent representation using a second trained neural network to produce anoutput image, wherein the output image is an approximation of the input image;wherein the first trained neural network and / or the second trained neural network aretrained according to any of the above methods.According to a further aspect, there is provided a data processing system configured to performany of the above methods.According to a further aspect, there is provided a data processing apparatus configured toperform any of the above methods..According to a further aspect, there is provided a computer program comprising instructionswhich, when the program is executed by a computer, cause the computer to carry out any ofthe above methods.According to a further aspect, there is provided a computer-readable storage medium comprisinginstructions which, when executed by a computer, cause the computer to carry out any of theabove methods.BRIEF DESCRIPTION OF THE DRAWINGSAspects of the invention will now be described by way of examples, with reference to thefollowing figures in which:Figure 1 illustrates an example of an image or video compression, transmission and decom-pression pipeline.Figure 2 illustrates a further example of an image or video compression, transmission anddecompression pipeline including a hyper-network.Figure 3 illustrates an example of a video compression, transmission and decompressionpipeline.Figure 4 illustrates an example of a video compression, transmission and decompressionsystem.Figure 5 illustrates a sequence of consecutive downsampling operations of a video compression,transmission and decompression pipeline.Figure 6 illustrates a sequence of consecutive downsampling operations of a video compression,transmission and decompression pipeline.Figure 7 illustrates a sequence of consecutive upsampling operations an example of a videocompression, transmission and decompression pipeline.Figure 8 illustrates an example of a video compression, transmission and decompressionpipeline.Figure 9 illustrates an example of a video compression, transmission and decompressionpipeline.Figure 10 illustrates an example of an encode side a video compression, transmission anddecompression pipeline.Figure 11 illustrates an example of a decode side of a video compression, transmission anddecompression pipeline.Figure 12 illustrates an example of a video compression, transmission and decompressionpipeline.DETAILED DESCRIPTION OF THE DRAWINGSCompression processes may be applied to any form of information to reduce the amountof data, or file size, required to store that information. Image and video information is anexample of information that may be compressed. The file size required to store the information,particularly during a compression process when referring to the compressed file, may bereferred to as the rate. In general, compression can be lossless or lossy. In both forms ofcompression, the file size is reduced. However, in lossless compression, no information is lostwhen the information is compressed and subsequently decompressed. This means that theoriginal file storing the information is fully reconstructed during the decompression process.In contrast to this, in lossy compression information may be lost in the compression anddecompression process and the reconstructed file may differ from the original file. Image andvideo files containing image and video data are common targets for compression.In a compression process involving an image, the input image may be represented as ^^. Thedata representing the image may be stored in a tensor of dimensions ^^ × ^^ × ^^, where ^^represents the height of the image, ^^ represents the width of the image and ^^ represents thenumber of channels of the image. Each ^^ × ^^ data point of the image represents a pixel valueof the image at the corresponding location. Each channel ^^ of the image represents a differentcomponent of the image for each pixel which are combined when the image file is displayed bya device. For example, an image file may have 3 channels with the channels representing thered, green and blue component of the image respectively. In this case, the image informationis stored in the RGB colour space, which may also be referred to as a model or a format.Other examples of colour spaces or formats include the CMKY and the YCbCr colour models.However, the channels of an image file are not limited to storing colour information and otherinformation may be represented in the channels. As a video may be considered a series ofimages in sequence, any compression process that may be applied to an image may also beapplied to a video. Each image making up a video may be referred to as a frame of the video.The output image may differ from the input image and may be represented by ^^. The differencebetween the input image and the output image may be referred to as distortion or a differencein image quality. The distortion can be measured using any distortion function which receivesthe input image and the output image and provides an output which represents the differencebetween input image and the output image in a numerical way. An example of such a methodis using the mean square error (MSE) between the pixels of the input image and the outputimage, but there are many other ways of measuring distortion, as will be known to the personskilled in the art. The distortion function may comprise a trained neural network.Typically, the rate and distortion of a lossy compression process are related. An increase inthe rate may result in a decrease in the distortion, and a decrease in the rate may result in anincrease in the distortion. Changes to the distortion may affect the rate in a correspondingmanner. A relation between these quantities for a given compression technique may be definedby a rate-distortion equation.AI based compression processes may involve the use of neural networks. A neural network isan operation that can be performed on an input to produce an output. A neural network maybe made up of a plurality of layers. The first layer of the network receives the input. One ormore operations may be performed on the input by the layer to produce an output of the firstlayer. The output of the first layer is then passed to the next layer of the network which mayperform one or more operations in a similar way. The output of the final layer is the output ofthe neural network.Each layer of the neural network may be divided into nodes. Each node may receive at leastpart of the input from the previous layer and provide an output to one or more nodes in asubsequent layer. Each node of a layer may perform the one or more operations of the layer onat least part of the input to the layer. For example, a node may receive an input from one ormore nodes of the previous layer. The one or more operations may include a convolution, aweight, a bias and an activation function. Convolution operations are used in convolutionalneural networks. When a convolution operation is present, the convolution may be performedacross the entire input to a layer. Alternatively, the convolution may be performed on at leastpart of the input to the layer.Each of the one or more operations is defined by one or more parameters that are associatedwith each operation. For example, the weight operation may be defined by a weight matrixdefining the weight to be applied to each input from each node in the previous layer to eachnode in the present layer. In this example, each of the values in the weight matrix is a parameterof the neural network. The convolution may be defined by a convolution matrix, also knownas a kernel. In this example, one or more of the values in the convolution matrix may be aparameter of the neural network. The activation function may also be defined by values whichmay be parameters of the neural network. The parameters of the network may be varied duringtraining of the network.Other features of the neural network may be predetermined and therefore not varied duringtraining of the network. For example, the number of layers of the network, the number ofnodes of the network, the one or more operations performed in each layer and the connectionsbetween the layers may be predetermined and therefore fixed before the training process takesplace. These features that are predetermined may be referred to as the hyperparameters of thenetwork. These features are sometimes referred to as the architecture of the network.To train the neural network, a training set of inputs may be used for which the expected output,sometimes referred to as the ground truth, is known. The initial parameters of the neuralnetwork are randomized and the first training input is provided to the network. The output ofthe network is compared to the expected output, and based on a difference between the outputand the expected output the parameters of the network are varied such that the differencebetween the output of the network and the expected output is reduced. This process is thenrepeated for a plurality of training inputs to train the network. The difference between theoutput of the network and the expected output may be defined by a loss function. The result ofthe loss function may be calculated using the difference between the output of the networkand the expected output to determine the gradient of the loss function. Back-propagation ofthe gradient descent of the loss function may be used to update the parameters of the neuralnetwork using the gradients ^^^^ / ^^^^ of the loss function. A plurality of neural networks in asystem may be trained simultaneously through back-propagation of the gradient of the lossfunction to each network.In the context of image or video compression, this type of system, where simultaneous trainingwith back-propagation through each element or the whole network architecture may be referredto as end-to-end, learned image or video compression. Unlike in traditional compressionalgorithms that use primarily handcrafted, manually constructed steps, an end-to-end learnedsystem learns itself during training what combination of parameters best achieves the goal ofminimising the loss function. This approach is advantageous compared to systems that are notend-to-end learned because an end-to-end system has a greater flexibility to learn weights andparameters that might be counter-intuitive to someone handcrafting features.It will be appreciated that the term "training" or "learning" as used herein means the processof optimizing an artificial intelligence or machine learning model, based on a given set of data.This involves iteratively adjusting the parameters of the model to minimize the discrepancybetween the model’s predictions and the actual data, represented by the above-describedrate-distortion loss function.The training process may comprise multiple epochs. An epoch refers to one complete passof the entire training dataset through the machine learning algorithm. During an epoch, themodel’s parameters are updated in an effort to minimize the loss function. It is envisaged thatmultiple epochs may be used to train a model, with the exact number depending on variousfactors including the complexity of the model and the diversity of the training data.Within each epoch, the training data may be divided into smaller subsets known as batches.The size of a batch, referred to as the batch size, may influence the training process. A smallerbatch size can lead to more frequent updates to the model’s parameters, potentially leading tofaster convergence to the optimal solution, but at the cost of increased computational resources.Conversely, a larger batch size involves fewer updates, which can be more computationallyefficient but might converge slower or even fail to converge to the optimal solution.The learnable parameters are updated by a specified amount each time, determined by thelearning rate. The learning rate is a hyperparameter that decides how much the parametersare adjusted during the training process. A smaller learning rate implies smaller steps in theparameter space and a potentially more accurate solution, but it may require more epochs toreach that solution. On the other hand, a larger learning rate can expedite the training processbut may risk overshooting the optimal solution or causing the training process to diverge.The training described herein may involve use of a validation set, which is a portion of thedata not used in the initial training, which is used to evaluate the model’s performance and toprevent overfitting. Overfitting occurs when a model learns the training data too well, to thepoint that it fails to generalize to unseen data. Regularization techniques, such as dropout orL1 / L2 regularization, can also be used to mitigate overfitting.It will be appreciated that training a machine learning model is an iterative process thatmay comprise selection and tuning of various parameters and hyperparameters. As will beappreciated, the specific details, such as hyper parameters and so on, of the training processmay vary and it is envisaged that producing a trained model in this way may achieved in anumber of different ways with different epochs, batch sizes, learning rates, regularisations,and so on, the details of which are not essential to enabling the advantages and effects of thepresent disclosure, except where stated otherwise. The point at which an “untrained” neuralnetwork is considered be “trained” is envisaged to be case specific and depend on, for example,on a number of epochs, a plateauing of any further learning, or some other metric and is notconsidered to be essential in achieving the advantages described herein.More details of an end-to-end, learned compression process will now be described. It will beappreciated that in some cases, end-to-end, learned compression processes may be combinedwith one or more components that are handcrafted or trained separately.In the case of AI based image or video compression, the loss function may be defined by therate distortion equation. The rate distortion equation may be represented by ^^^^^^^^ = ^^ + ^^ ∗ ^^,where ^^ is the distortion function, ^^ is a weighting factor, and ^^ is the rate loss. ^^ may bereferred to as a lagrange multiplier. The langrange multiplier provides as weight for a particularterm of the loss function in relation to each other term and can be used to control which termsof the loss function are favoured when training the network.In the case of AI based image or video compression, a training set of input images maybe used. An example training set of input images is the KODAK image set (for exampleat www.cs.albany.edu / xypan / research / snr / Kodak.html). An example training set of inputimages is the IMAX image set. An example training set of input images is the Imagenetdataset (for example at www.image-net.org / download). An example training set of inputimages is the CLIC Training Dataset P (“professional”) and M (“mobile”) (for example athttp: / / challenge.compression.cc / tasks / ).An example of an AI based compression, transmission and decompression process 100 isshown in Figure 1. As a first step in the AI based compression process, an input image 5 isprovided. The input image 5 is provided to a trained neural network 110 characterized by afunction ^^^^ acting as an encoder. The encoder neural network 110 produces an output basedon the input image. This output is referred to as a latent representation of the input image 5. Ina second step, the latent representation is quantised in a quantisation process 140 characterisedby the operation ^^, resulting in a quantized latent. The quantisation process transforms thecontinuous latent representation into a discrete quantized latent. An example of a quantizationprocess is a rounding function.In a third step, the quantized latent is entropy encoded in an entropy encoding process 150 toproduce a bitstream 130. The entropy encoding process may be for example, range or arithmeticencoding. In a fourth step, the bitstream 130 may be transmitted across a communicationnetwork.In a fifth step, the bitstream is entropy decoded in an entropy decoding process 160. Thequantized latent is provided to another trained neural network 120 characterized by a function^^^^ acting as a decoder, which decodes the quantized latent. The trained neural network 120produces an output based on the quantized latent. The output may be the output image of theAI based compression process 100. The encoder-decoder system may be referred to as anautoencoder.Entropy encoding processes such as range or arithmetic encoding are typically able to losslesslycompress given input data up to close to the fundamental entropy limit of that data, as determinedby the total entropy of the distribution of that data. Accordingly, one way in which end-to-end,learned compression can minimise the rate loss term of the rate-distortion loss function andthereby increase compression effectiveness is to learn autoencoder parameter values thatproduce low entropy latent representation distributions. Producing latent representationsdistributed with as low an entropy as possible allows entropy encoding to compress the latentdistributions as close to or to the fundamental entropy limit for that distribution. The lowerthe entropy of the distribution, the more entropy encoding can losslessly compress it and thelower the amount of data in the corresponding bitstream. In some cases where the latentrepresentation is distributed according to a gaussian or Laplacian distribution, this learningmay comprise learning optimal location and scale parameters of the gaussian or Laplaciandistributions, in other cases, it allows the learning of more flexible latent representationdistributions which can further help to achieve the minimising of the rate-distortion lossfunction in ways that are not intuitive or possible to do with handcrafted features. Examples ofthese and other advantages are described in WO2021 / 220008A1, which is incorporated in itsentirety by reference.Something which is closely linked to the entropy encoding of the latent distribution and whichaccordingly also has an effect on the effectiveness of compression of end-to-end learnedapproaches is the quantisation step. During inference, a rounding function may be used toquantise a latent representation distribution into bins of given sizes, a rounding function isnot differentiable everywhere. Rather, a rounding function is effectively one or more stepfunctions whose gradient is either zero (at the top of the steps) or infinity (at the boundarybetween steps). Back propagating a gradient of a loss function through a rounding functionis challenging. Instead, during training, quantisation by rounding function is replaced byone or more other approaches. For example, the functions of a noise quantisation model aredifferentiable everywhere and accordingly do allow backpropagation of the gradient of theloss function through the quantisation parts of the end-to-end, learned system. Alternatively, astraight-through estimator (STE) quantisation model or one other quantisation models may beused. It is also envisaged that different quantisation models may be used for during evaluationof different term of the loss function. For example, noise quantisation may used to evaluate therate or entropy loss term of the rate-distortion loss function while STE quantisation may beused to evaluate the distortion term.In a similar manner to how learning parameters top produce certain distributions of the latentrepresentation facilitates achieving better rate loss term minimisation, end-to-end learning ofthe quantisation process achieves a similar effect. That is, learnable quantisation parametersprovide the architecture with a further degree of freedom to achieve the goal of minimising theloss function. For example, parameters corresponding to quantisation bin sizes may be learnedwhich is likely to result in an improved rate-distortion loss outcome compared to approachesusing hand-crafted quantisation bin sizes.Further, as the rate-distortion loss function constantly has to balance a rate loss term against adistortion loss term, it has been found that the more degrees of freedom the system has duringtraining, the better the architecture is at achieving optimal rate and distortion trade off.The system described above may be distributed across multiple locations and / or devices. Forexample, the encoder 110 may be located on a device such as a laptop computer, desktopcomputer, smart phone or server. The decoder 120 may be located on a separate device whichmay be referred to as a recipient device. The system used to encode, transmit and decode theinput image 5 to obtain the output image 6 may be referred to as a compression pipeline.The AI based compression process may further comprise a hyper-network 105 for thetransmission of meta-information that improves the compression process. The hyper-network105 comprises a trained neural network 115 acting as a hyper-encoder ^^ ℎ^^and a trained neuralnetwork 125 acting as a hyper-decoder ^^ℎ^^. An example of such a system is shown in Figure 2.Components of the system not further discussed may be assumed to be the same as discussedabove. The neural network 115 acting as a hyper-decoder receives the latent that is the output ofthe encoder 110. The hyper-encoder 115 produces an output based on the latent representationthat may be referred to as a hyper-latent representation. The hyper-latent is then quantizedin a quantization process 145 characterised by ^^ℎ to produce a quantized hyper-latent. Thequantization process 145 characterised by ^^ℎ may be the same as the quantisation process 140characterised by ^^ discussed above.In a similar manner as discussed above for the quantized latent, the quantized hyper-latent isthen entropy encoded in an entropy encoding process 155 to produce a bitstream 135. Thebitstream 135 may be entropy decoded in an entropy decoding process 165 to retrieve thequantized hyper-latent. The quantized hyper-latent is then used as an input to trained neuralnetwork 125 acting as a hyper-decoder. However, in contrast to the compression pipeline 100,the output of the hyper-decoder may not be an approximation of the input to the hyper-decoder115. Instead, the output of the hyper-decoder is used to provide parameters for use in theentropy encoding process 150 and entropy decoding process 160 in the main compressionprocess 100. For example, the output of the hyper-decoder 125 can include one or more ofthe mean, standard deviation, variance or any other parameter used to describe a probabilitymodel for the entropy encoding process 150 and entropy decoding process 160 of the latentrepresentation. In the example shown in Figure 2, only a single entropy decoding process 165and hyper-decoder 125 is shown for simplicity. However, in practice, as the decompressionprocess usually takes place on a separate device, duplicates of these processes will be presenton the device used for encoding to provide the parameters to be used in the entropy encodingprocess 150.Further transformations may be applied to at least one of the latent and the hyper-latent at anystage in the AI based compression process 100. For example, at least one of the latent and thehyper latent may be converted to a residual value before the entropy encoding process 150,155is performed. The residual value may be determined by subtracting the mean value of thedistribution of latents or hyper-latents from each latent or hyper latent. The residual valuesmay also be normalised.The concept of a hyper-encoder and hyper-decoder that form a hyper-network may be extendedfurther to a hyper-hyper-encoder and hyper-hyper-decoder, and so on, to effectively nestnetworks on top of each other, each being used to transmit information associated with thedistribution of the latent from the layer below it that is being entropy encoded and decoded.To perform training of the AI based compression process described above, a training set ofinput images may be used as described above. During the training process, the parameters ofboth the encoder 110 and the decoder 120 may be simultaneously updated in each trainingstep. If a hyper-network 105 is also present, the parameters of both the hyper-encoder 115 andthe hyper-decoder 125 may additionally be simultaneously updated in each training step.The training process may further include a generative adversarial network (GAN). Whenapplied to an AI based compression process, in addition to the compression pipeline describedabove, an additional neutral network acting as a discriminator is included in the system. Thediscriminator receives an input and outputs a score based on the input providing an indicationof whether the discriminator considers the input to be ground truth or fake. For example, theindicator may be a score, with a high score associated with a ground truth input and a lowscore associated with a fake input. For training of a discriminator, a loss function is used thatmaximizes the difference in the output indication between an input ground truth and input fake.When a GAN is incorporated into the training of the compression process, the output image 6may be provided to the discriminator. The output of the discriminator may then be used in theloss function of the compression process as a measure of the distortion of the compressionprocess. Alternatively, the discriminator may receive both the input image 5 and the outputimage 6 and the difference in output indication may then be used in the loss function of thecompression process as a measure of the distortion of the compression process. Training ofthe neural network acting as a discriminator and the other neutral networks in the compressionprocess may be performed simultaneously. During use of the trained compression pipelinefor the compression and transmission of images or video, the discriminator neural network isremoved from the system and the output of the compression pipeline is the output image 6.Incorporation of a GAN into the training process may cause the decoder 120 to performhallucination. Hallucination is the process of adding information in the output image 6 thatwas not present in the input image 5. In an example, hallucination may add fine detail tothe output image 6 that was not present in the input image 5 or received by the decoder 120.The hallucination performed may be based on information in the quantized latent received bydecoder 120.Details of a video compression process will now be described. As discussed above, a video ismade up of a series of images arranged in sequential order. AI based compression process100 described above may be applied multiple times to perform compression, transmissionand decompression of a video. For example, each frame of the video may be compressed,transmitted and decompressed individually. The received frames may then be grouped toobtain the original video.The frames in a video may be labelled based on the information from other frames that is usedto decode the frame in a video compression, transmission and decompression process. Asdescribed above, frames which are decoded using no information from other frames may bereferred to as I-frames. Frames which are decoded using information from past frames may bereferred to as P-frames. Frames which are decoded using information from past frames andfuture frames may be referred to as B-frames. Frames may not be encoded and / or decoded inthe order that they appear in the video. For example, a frame at a later time step in the videomay be decoded before a frame at an earlier time.The images represented by each frame of a video may be related. For example, a number offrames in a video may show the same scene. In this case, a number of different parts of thescene may be shown in more than one of the frames. For example, objects or people in a scenemay be shown in more than one of the frames. The background of the scene may also beshown in more than one of the frames. If an object or the perspective is in motion in the video,the position of the object or background in one frame may change relative to the position ofthe object or background in another frame. The transformation of a part of the image froma first position in a first frame to a second position in a second frame may be referred to asflow, warping or motion compensation. The flow may be represented by a vector. One or moreflows that represent the transformation of at least part of one frame to another frame may bereferred to as a flow map.An example AI based video compression, transmission, and decompression process 200 isshown in Figure 3. The process 200 shown in Figure 3 is divided into an I-frame part 201for decompressing I-frames, and a P-frame part 202 for decompressing P-frames. It will beunderstood that these divisions into different parts are arbitrary and the process 200 may bealso be considered as a single, end-to-end pipeline.As described above, I-frames do not rely on information from other frames so the I-frame part201 corresponds to the compression, transmission, and decompression process illustrated inFigures 1 or 2. The specific details will not be repeated here but, in summary, an input image^^0 is passed into an encoder neural network 203 producing a latent representation which isquantised and entropy encoded into a bitstream 204. The subscript 0 in ^^0 indicates the inputimage corresponds to a frame of a video stream at position t = 0. This may be the first frame ofan entire video stream or the first frame of a chunk of a video stream made up of, for example,an I-frame and a plurality of subsequent P-frames and / or B-frames. The bitstream 204 is thenentropy decoded and passed into a decoder neural network 205 to reproduce a reconstructedimage ^^0 which in this case is an I-frame. The decoding step may be performed both locallyat the same location as where the input image compression occurs as well as at the locationwhere the decompression occurs. This allows the reconstructed image ^^0 to be available forlater use by components of both the encoding and decoding sides of the pipeline.In contrast to I-frames, P-frames (and B-frames) do rely on information from other frames.Accordingly, the P-frame part 202 at the encoding side of the pipeline takes as input not onlythe input image ^^^^ that is to be compressed (corresponding to a frame of a video stream atposition t), but also one or more previously reconstructed images ^^^^−1 from an earlier framet-1. As described above, the previously reconstructed ^^^^−1 is available at both the encodeand decode side of the pipeline and can accordingly be used for various purposes at both theencode and decode sides.At the encode side, previously reconstructed images may be used for generating a flow mapscontaining information indicative of inter-frame movement of pixels between frames. In theexample of Figure 3, both the image being compressed ^^^^ and the previously reconstructedimage from an earlier frame ^^^^−1 are passed into a flow module part 206 of the pipeline. Theflow module part 206 comprises an autoencoder such as that of the autoencoder systems ofFigures 1 and 2 but where the encoder neural network 207 has been trained to produce alatent representation of a flow map from inputs ^^^^−1 and ^^^^ , which is indicative of inter-framemovement of pixels or pixel groups between ^^^^−1 and ^^^^ . The latent representation of the flowmap is quantised and entropy encoded to compress it and then transmitted as a bitstream 208.On the decode side, the bitstream is entropy decoded and passed to a decoder neural network209 to produce a reconstructed flow map ^^ .The reconstructed flow map ^^ is applied to the previously reconstructed image ^^^^−1 to generatea warped image ^^^^−1,^^. It is envisaged that any suitable warping technique may be used, forexample bi-linear or tri-linear warping, as is described in Agustsson, E., Minnen, D., Johnston,N., Balle, J., Hwang, S. J., and Toderici, G. (2020), Scale-space flow for end-to-end optimizedvideo compression. In Proceedings of the IEEE / CVF Conference on Computer Vision andPattern Recognition (pp. 8503-8512), which is hereby incorporated by reference. It is furtherenvisaged that a scale-space flow approach as described in the above paper may also optionallybe used. The warped image ^^^^−1,^^ is a prediction of how the previously reconstructed image^^^^−1 might have changed between frame positions t-1 and t, based on the output flow mapproduced by the flow module part 206 autoencoder system from the inputs of ^^^^ and ^^^^−1.As with the I-frame, the reconstructed flow map ^^ and corresponding warped image ^^^^−1,^^may be produced both on the encode side and the decode side of the pipeline so they areavailable for use by other components of the pipeline on both the encode and decode sides.In the example of Figure 3, both the image being compressed ^^^^ and the ^^^^−1,^^ are passedinto a residual module part 210 of the pipeline. The residual module part 210 comprises anautoencoder system such as that of the autoencoder systems of Figures 1 and 2 but where theencoder neural network 211 has been trained to produce a latent representation of a residualmap indicative of differences between the input mage ^^^^ and the warped image ^^^^−1,^^. Thelatent representation of the residual map is then quantised and entropy encoded into a bitstream212 and transmitted. The bitstream 212 is then entropy decoded and passed into a decoderneural network 213 which reconstructs a residual map ^^ from the decoded latent representation.Alternatively, a residual map may first be pre-calculated between ^^^^ and the ^^^^−1,^^ and thepre-calculated residual map may be passed into an autoencoder for compression only. Thishand-crafted residual map approach is computationally simpler, but reduces the degrees offreedom with which the architecture may learn weights and parameters to achieve its goalduring training of minimising the rate-distortion loss function.Finally, on the decode side, the residual map ^^ is applied (e.g. combined by addition, subtractionor a different operation) to the warped image to produce a reconstructed image ^^^^ which is areconstruction of image ^^^^ and accordingly corresponds to a P-frame at position t in a sequenceof frames of a video stream. It will be appreciated that the reconstructed image ^^^^ can then beused to process the next frame. That is, it can be used to compress, transmit and decompress^^^^+1, and so on until an entire video stream or chunk of a video stream has been processed.Thus, for a block of video frames comprising an I-frame and ^^ subsequent P-frames, thebitstream may contain (i) a quantised, entropy encoded latent representation of the I-frameimage, and (ii) a quantised, entropy encoded latent representation of a flow map and residualmap of each P-frame image. For completeness, whilst not illustrated in Figure 3, any of theautoencoder systems of Figure 3 may comprise hyper and hyper-hyper networks such as thosedescribed in connection with Figure 2. Accordingly, the bitstream may also contain hyper andhyper-hyper parameters, their latent quantised, entropy encoded latent representations and soon, of those networks as applicable.Finally, the above approach may generally also be extended to B-frames, for example as isdescribed in Pourreza, R., and Cohen, T. (2021). Extending neural p-frame codecs for b-framecoding. In Proceedings of the IEEE / CVF International Conference on Computer Vision (pp.6680-6689).The above-described flow and residual based approach is highly effective at reducing theamount of data that needs to be transmitted because, as long as at least one reconstructed frame(e.g. I-frame ^^^^−1) is available, the encode side only needs to compress and transmit a flowmap and a residual map (and any hyper or hyper-hyper parameter information, as applicable)to reconstruct a subsequent frame.Figure 4 shows an example of an AI image or video compression process such as that describedabove in connection with Figures 1-3 implemented in a video streaming system 400. Thesystem 400 comprises a first device 401 and a second device 402. The first and seconddevices 401, 402 may be user devices such as smartphones, tablets, AR / VR headsets or otherportable devices. In contrast to known systems which primarily perform inference on GPUssuch as Nvidia A100, Geforce 3090, Gefore 4090 GPU cards, the system 400 of Figure 4performs inference on a CPU of the first and second devices respectively. That is, computefor performing both encoding and decoding are performed by the respective CPUs of the firstand second devices 401, 402. This places very different power usage, memory and runtimeconstraints on the implementation of the above methods than when implementing AI-basedcompression methods on GPUs. In one example, the CPU of first and second devices 401, 402may comprise a Qualcomm Snapdragon CPU.The first device 401 comprises a media capture device 403, such as a camera, arranged tocapture a plurality of images, referred to hereafter as a video stream 404, of a scene 404. Thevideo stream 404 is passed to a pre-processing module 406 which splits the video stream intoblocks of frames, various frames of which will be designated as I-frames, P-frames, and / orB-frames. The blocks of frames are then compressed by an AI-compression module 407comprising the encode side of the AI-based video compression pipeline of Figure 3. Theoutput of the AI-compression module is accordingly a bitstream 408a which is transmittedfrom the first device 401, for example via a communications channel, for example over oneor more of a WiFi, 3G, 4G or 5G channel, which may comprise internet or cloud-based 409communications.The second device 402 receives the communicated bitstream 408b which is passed to anAI-decompression module 410 comprising the decode side of the AI-based video compressionpipeline of Figure 3. The output of the AI-decompression module 402 is the reconstructedI-frames, P-frames and / or B-frames which are passed to a post-processing module 411 wherethey can prepared, for example passed into a buffer, in preparation for streaming 412 to andrendering on a display device 413 of the second device 402.It is envisaged that the system 400 of Figure 4 may be used for live video streaming at 30fps ofa 1080p video stream, which means a cumulative latency of both the encode and decode sideis below substantially 50ms, for example substantially 30ms or less. Achieving this level ofruntime performance with only CPU compute on user devices presents challenges which arenot addressed by known methods and systems or in the wider AI-compression literature.For example, execution of different parts of the compression pipeline during inferencemay be optimized by adjusting the order in which operations are performed using one ormore known CPU scheduling methods. Efficient scheduling can allow for operations to beperformed in parallel, thereby reducing the total execution time. It is also envisaged thatefficient management of memory resources may be implemented, including optimising cachingmethods such as storing frequently-accessed data in faster memory locations, and memoryreuse, which minimizes memory allocation and deallocation operations.A number of concepts related to the AI compression processes and / or their implementationin a hardware system discussed above will now be described. Although each concept isdescribed separately, one or more of the concepts described below may be applied in an AIbased compression process as described above.Concept 1: Dynamic PaddingOne way to reduce the compute and memory overhead when performing AI-based compressionis to downsample input images, and / or representations thereof in the flow and / or residualcomponents of the pipeline, and running the pipeline on the downsampled input images, beforeupsampling the output of the pipeline back into the original input resolution. For example thecompute and memory overhead for a 1080p image is significantly higher than that of a 360pimage. However, downsampling (and corresponding upsampling) operations themselves cancontribute significantly to the compute and memory overhead. This is problematic when agoal of AI-based compression is to be able to encode and decode in real time or nearly realtime, at acceptable power usage levels.Downsampling and upsampling may be performed using convolution operations or pixel shuffleoperations. For example, upsampling with convolution (also known as a transposed convolution,or deconvolution) work by distributing input values over a larger spatial area, introducing zerosbetween input units which are then filled in a learnable manner via convolution operations.The weights in these convolutional filters are trainable, meaning they can be optimized duringthe training process to best perform the task at hand, such as reconstructing higher-resolutiondetails from lower-resolution inputs.Similarly, downsampling is typically achieved through strided convolutions or with pixelunshuffle operations, where the stride of the convolution is greater than one. This effectivelyskips over inputs according to the stride value, reducing the spatial dimensions of the outputcompared to the input. Like standard convolutions, the weights in strided convolutions arealso trainable. These weights learn to capture the most significant features of the input dataover their receptive fields, effectively compressing information while maintaining importantfeatures.When downsampling, the one or more resolutions of the input image (e.g. height and / or widthand / or some other dimension such as channel) is reduced by division by some amount suchas an integer. For example, if the integer is 2, the dimension being downsampled is dividedby 2 and the dimension is halved or reduced by 2x. Other integers, or indeed non-integervalues are also possible. For clarity we refer herein to downsampling by 2x multiples butnote that the disclosure applies equally to all downsampling amounts. In the illustrative casewhere the input image is natively divisible by 2 (e.g. a 2x downsample), no additional paddingor other dimension modifying pre-processing operations will be needed to make it divisibleby the integer. These kinds of dimension modifying pre-processing operations contributesignificantly to compute and memory overhead.Note also that the downsampling operation to reduce the dimension can also be performed anumber of times consecutively. In the halving downsampling example, this may accordinglydownsample one or more dimensions of the input image by 2x, 4x, 8x, 16x, and so on asdesired. However, in an "in the wild" setting, it is often difficult to control what the inputimage dimensions are. Accordingly a padding operation (or other dimension modifyingpre-processing operation) is performed on the input image before ingestion into the AI-basedcompression to ensure that the input image can be downsampled by the desired amount thedimension. The padding operation comprises adding a line of empty (e.g. black or white orsome other colour) pixels to the dimension being padded, such as the height and / or width.However, the inventors have realised that padding is a significant cause of failure modes andartefacts in the neural networks of an AI-based compression pipeline. Unlike block-based,deterministic traditional compression which can typically handle blocks of padding very easilywithout introducing artefacts, the same is not true of AI-based compression pipelines. InAI-based compression pipelines, the neural networks of the pipeline learn from what theyare given during trianing. This means that if some padding operation is used during imageingestion to allow the dimension halving downsampling operations to be performed efficiently,the neural networks will learn to expect the one or more lines of padding pixels. Thus, even ifan input image does not have the lines of padding as the dimension being downsampled isperfectly divisible by 2, the neural networks of the pipeline may nevertheless hallucinate theseinto the output reconstructed image.A further problem is that padding increases the number of pixels to be processed with absolutelyno compression performance benefit other than to facilitate convolutional downsampling. Pixelnumber increases are particularly detrimental in the flow module of the pipeline where motionis estimated by comparing pixels of two images and so even small increases in pixels beingchecked for motion can result in significant performance hits. For example, to ensure a 1080p(height) image can be downsampled by 128x by consecutive halving downsampling operations,the height dimension is padded by 72 lines of empty pixels, giving an input image of 1152p.This is corresponds to an approximately 7% increase in the number of pixels to be processedand a corresponding performance in terms of a slower runtime by approximately 10%. By wayof example, if we halve 1152 in consecutive downsampling operations we get the followingsequence of image dimensions, with the brackets denoting the amount of downsampling ateach stage and approximate number of lines at that stage that are padding:1152 (72 padding lines)576 (2x) (36 padding lines)288 (4x) (18 padding lines)144 (8x) (9 padding lines)72 (16x) (5 padding lines)36 (32x) (3 padding lines)18 (64x) (2 padding lines)9 (128x)Thus, while the addition of padding lines of pixels facilitates the use dimension halvingconvolution operations (or pixel shuffle operations or other operations) to get to 128xdownsampling, the increase in the total number of pixels being processed in the additional,cumulative 145 lines of padding across the sequence of increasingly downsampled images isdetrimental to performance.Whilst one approach to solving the hallucination problem may be to include images that requirepadding and images that don’t require padding in the training data, this does not solve theproblem that adding padding results in a significant performance hit due to the additionalpixels of the padding lines. Further, simply modifying the training data does not give theneural networks an architectural mechanism to learn when or why padding has been added andthus there is no strong training signal for the network to learn when or why to include emptypadding pixels at the image reconstruction stage, even if the training data is suitably varied.An improved method of handling input images of any dimension with an AI-based compressionpipeline is accordingly desired.The present inventors have realised the above problems can be solved by introducing aninput-dimension-based padding schedule into the consecutive downsampling operations thatonly pads the image input to that operation if the dimension being padded is not divisible bythe downsampling amount e.g. 2 in this example but also any other amount as being used forthe downsampling e.g. 3, 4, 5 or some other amount, and then only pads it just enough to getto a dimension that is divisible by the downsampling amount.This has two effects that are synergistic.Firstly, the total number of pixels being processed is reduced to only those steps where paddingis strictly necessary as shown in the example of a 1080p downsampling sequence below:1080540 (2x)270 (4x)135 (8x) (1 padding line) ← add 1 padding line here to get 13668 (16x) (1 padding line)34 (32x) (1 padding line)17 (64x) (2 padding lines) ← add 1 padding line here to get 189 (128x) (2 padding lines)In the above example using the approach of the present application, we are able to downsampleby 128x using, cumulatively 7 lines of padding in total across the whole sequence of increasinglydownsampled images. This is around 5% of the cumulative 145 lines of padding added whenusing the naive approach described above. The resulting performance increase is accordinglysignificant.Secondly, the input-dimension-based padding schedule makes the network "padding aware"because it forces the networks to see the same input image being padded at some levels ofdownsampling while not padded at other other levels of downsampling. As the weights ofthe convolution layers performing the downsampling are trainable, this means the networkhas a mechanism by which to learn when and why padding is added sometimes but not atother times to the same input image - something which is not possible when the padding isadded naively to all images before downsampling, or when attempting to achieve this effectthrough training data only. Thus, not only does this approach result in a runtime performanceboost due to fewer pixels being processed, but the networks become padding aware whichin turn results in fewer artefacts and hallucinations in the reconstructed images for a givenbitrate / compression rate, even when padding has been added.Figure 5 illustrates a sequence 500 of consecutive downsampling operations 501, 502, 503with naive padding lines 504 applied to an input image 505 that has a starting width dimension^^0 and a starting height dimension ^^0. The number of pixel lines added in both the heightand width dimensions are enough to ensure that the dimension being downsampled acrossthe entire sequence is always divisible by 2 so that the dimensions being downsampled canbe halved at every downsample. Accordingly, each downsampling operation 501, 502, 503halves the respective width and height dimension it is being applied to. Thus, after the firstdownsamlping operation 501 is performed, the resulting image 506 has width dimension ^^1and height dimension ^^ ^^1 where ^^1 = 0^^02 and ^^1 = 2 . The padding pixels 504 propagatethrough the downsampling operation 501 and thus remain in the downsampled image 506in downampled form 507. After the second downsampling operation 502 is performed, theresulting image 508 has width dimension ^^2 and height dimension ^^ ^^2 where ^^2 = 12 and^^ ^^ 2 =12 . As before, original padding lines 504 remain in the downsampled image 508 indownsampled form 509. After the third downsampling operation 503 is performed, theresulting image 510 has width dimension ^^ ^^3 and height dimension ^^3 where ^^3 = 22 and^^ ^^ 3 =22 . As before, the original padding lines 504 remain in the downsampled image 510 inform 511. Whilst not shown, it is envisaged that any number of consecutivedownsampling steps may be provided and in any dimension e.g. both height and width orheight alone or width alone, or some other dimension, as applicable. As described above, thisnaive padding approach detrimentally increases the compute and resource overhead due tothe addition of the padding lines, while at the same time causes an increase in the numberof artefacts present in the reconstructed after the downsampled image is passed through theAI-based compression pipeline and reconstrcuted on the decode side.Figure 6 illustrates a sequence 600 of consecutive downsampling operations similar to thatshown in Figure 5, except now the dynamic padding approach of the present disclosure isapplied. That is, rather than naively add padding lines to the input image to ensure divisibilityby the amount being downsampled for every downsample of the sequence, padding is onlyadded where the number of pixels of the dimension being downsampled at that step is notdivisble by that amount, and the number of pixels or pixel lines added at that step is just enoughto make the number divisible by that amount. For example, in the context of a downsamplingoperation that halves the dimension, the number of pixels in that dimension must be divisibleby 2. Padding is accordingly only added where that is not the case, and only in a quantity thatis necessary to get the dimension back to being divisible by 2. In practice, for the halvingexample, this means that the dynamic padding approach only adds a single line of padding forthe small number of instances where there are an odd number of pixels in the dimension beingdownsampled.As is shown in Figure 6, a sequence 600 of consecutive downsampling operations 601, 602,603 are applied to an input image 605. Before each downsampling operation is applied, it isdetermined whether or not the dimension(s) being downsampled is divisible by the amountbeing downsampled by, e.g. 2. In the example of Figure 6, we assume that the original imagecan be divided by 2 so we don’t add any padding before applying downsample operation601. This produces downsanpled image 606. We again check whether the dimension(s)being downsampled are divisible by 2, and they are, so we can proceed to apply downsampleoperation 602 to produce downsampled image 608. In this example, let’s assume that thedimensions being downsampled, e.g. height and width, are not divisible by 2 in image 608, forexample because there are an odd number of pixels in those dimensions. Applying the dynamicpadding approach, we add a single line of padding 609 to each dimension to make it divisbbleby 2 again, and proceed to perform downsample operation 603 to produce downsampled image610. The line of padding 609 does propagate through to image 610 in downsampled form 611.However, the number of padding lines added is significantly smaller than when we pad naivelyas was the case in Figure 5.The downsampled image may then be passed through an AI-based compression pipeline suchas that shown in Figures 1, 2 or 3. As described above, the downsampling operations maybe performed by one or more convolution operations and which may be performed by oneor more layers of the encoder neural network(s). Alternatively, they may be performed as astandalone step by a dedicated downsampling neural network that performs the same function.More generally, the intermediate outputs of one or more layers of the neural network may bedownsampled and accordingly, the above-described dynamic padding may be applied to theseintermediate outputs as applicable.Whilst not explicitly shown, the flow module 206 illustrated in Figure 3 is able to especiallybenefit from dynamic padding as flow may be calculated at a number of different resolutionsand combined to produce an overall flow ^^ . Accordingly, the number of downsamplingoperations in the flow module 206 may be higher than in the other modules and the total pixelreductions that arise from dynamic padding instead of naive padding is particularly pronouncedin the flow module 206.As described above, the downsampling occurs on the encode side of the AI-based compressionpipeline. A corresponding set of upsampling operations may be performed on the decode sideof the AI-based compression pipeline that mirror the downsampling operations but in reverse,for example by applying a transpose convolution or deconvolution operation.Figure 7 illustrates a sequence 700 of consecutive upsampling operations 701, 702, 703 areapplied to an image 704. In one approach, before each upsampling operation is applied, it isdetermined whether or not the dimension(s) being upsampled was downsampled on the encodeside. This may be determined from a priori knowledge of the original dimension of the imagebeing decoded, which may be transmitted in the form of side information in the bitstream. Forexample, once the original dimension is known, the padding schedule that was applied duringdownsampling on encode can be calculated in reverse and applied accordingly. Thus, in Figure7, which is a reverse of the operations of Figure 6, we upsample 701 the image 704 leaving thepadding lines 705 unchanged in the first upsample operation, and only remove the paddinglines 706 from the image 707 in the second step before the second upsample operation 702.Finally, as we have already removed all the padding lines from the image 708 before the finalupsample operation 703, there are no additional processing steps to remove any padding toproduce the final image 709.In another approach, the upsampling is not complementary (i.e. there is no determination ofwhether or not the dimension(s) being upsampled was downsampled on the encode side. Inthis case, the number of pixels to remove is calculated as set out below.For a particular dimension of the image (e.g. height or width or another dimension), thecalculation of the number of pixels to remove after upsampling is as follows:Let ^^ be the input dimension length and ^^^^ be the current length of the tensor in that dimensionafter an upsample. Let ^^1, .., ^^^^ be the factors of the remaining upsamples to perform inconsecutive sequence of upsampling operations. Then the amount of ’pixels’ to remove fromthe tensor can be calculated as:⌊ ∏^^⌋ ^^^^∏^^=1 ^^^^^^ − ^^^^^^ We now consider the concept of padding awareness in training in more detail. Provided belowis a pseudocode algorithm outlining a padding aware training regime:Algorithm 1 Padding aware trainingInputs: Neural network ^^ (^^) including a plurality of downsampling and upsamplingconvolution layers ^^(^^) where ^^ are learnable weightsVariables: Optimizer ^^, gradient computation flag ^^^^^^^^.Initialise: ^^^^^^^^ = ^^^^^^^^ for ^^ (^^) and ^^(^^), and Training loop counter ^^ = 0.Set config of downsample and upsample amount ^^ per convolution layer e.g. 2x, 3x, orsome other amountwhile ^^ < ^^ doForward pass through ^^ (^^) including downsample and upsample convolution layers ^^(^^), adding paddingonly before operations where input not divisible by ^^Compute loss and perform backward pass with ^^^^^^^^.Apply optimizer ^^ to update ^^ (^^) including ^^(^^).^^ ← ^^ + 1end whileA neural network ^^ (^^) with a plurality of downsampling and upsampling convolution layers^^(^^) is provided where ^^ and ^^ are learnable weights. A configuration file is providedspecifying the rules of the dynamic padding regime. That is, specifying an amount ^^ ofdownsampling or upsampling to be performed per convolution layer. In the examples providedthis may e.g. 2x whereby the downsampling operations applied consecutively facilitatedownsampling to e.g. 2x, 4x, 8x, 16x, 32x 64x 128x and so on. The neural networks areinitialised e.g. with random values, and the grad flag is set to true for both the weightsand / or biases of the neural network ^^ (^^) and the upsampling and downsampling layers ^^(^^),indicating that they are trainable. Training for for ^^ steps then commences by repeatedlyperforming a forward pass through ^^ (^^) including through ^^(^^). The forward pass comprisesapplying the dynamic padding based on amount ^^, whereby if the input to a downsamplinglayer is not divisible by the amount ^^ padding is added until it is, and downsampling canproceed. The loss is computed, which may be obtained with a rate distortion loss function (e.g.as described above in the introductory section), and back propagation is performed throughthe elements where ^^^^^^^^ is set to true. An optimiser ^^, e.g. SGD, Adam, or some otheroptimisation algorithm is applied to update the weights and / or biases of ^^ (^^) including tothose of ^^(^^). Training ends after some predetermined condition is met, for example after apredetermined number of epochs, training steps, or after the loss plateaus or in some otherway converges. A note on training data: including image dimensions of a variety of differentsizes can be beneficial as it ensures the model is exposed to the concept of adding paddingat different places within the consecutive downsampling operations when dynamic paddingis applied. This in turn helps the model generalise well to input images it receives duringinference "in the wild".A consequence of applying the dynamic padding approach during training is that the down-sampling and upsampling layers ^^(^^) are, as a whole, exposed to a representation of the sameimage both with and without padding. For example, if we take the toy example image 605 ofFigure 6, we see that padding is only added after two the first downsampling operations 601,602 have taken place. This means that, during training, ^^(^^) has downsampling layers that,together, get to see both a representation of the image in original form without any padding, aswell as a representaiton of that image, or at least its features, in padded form. As a result, ^^(^^)can be said to learn during training that the image does not originally contain padding, that thepadding may be added during some intermediate step of downsampling, that some images mayneed padding, some may not need any padding at all, and that, on decode side, the paddingmay be removed during some intermediate upsampling step if it was not originally present inthe input image (whereby if padding is incorrectly reconstructed in an output image by theneural networks, the distortion scores will increase and so this effect is penalised by the ratedistortion loss function).Thus, dynamic padding provides the neural networks with a mechanism by which to learnwhen and why padding is added sometimes but not at other times to the same input image- something which is not possible when the padding is added naively to all images beforedownsampling, or when attempting to achieve this effect through training data only. Thus,not only does this approach result in a runtime performance boost due to fewer pixels beingprocessed, but the networks become padding aware which in turn results in fewer artefacts andhallucinations in the reconstructed images for a given bitrate / compression rate, even whenpadding has been added.An additional, indirect consequence of the dynamic padding approach is that it allows thenetworks to learn to accommodate images or videos that natively contain padding. For example,a video sequence scraped from a video streaming site in wide screen mode is typically paddedwith black pixels above and below the substantive pixels that make up the video. Withoutdynamic padding, there is no architectural mechanism for the neural networks to learn whatpadding was in the original video stream as ingested, and what may have been added duringdownsampling. This makes the training signal very noisy when attempting to train on suchimages and videos. In contrast, with dynamic padding, the downsampling and upsamplinglayers ^^(^^) are exposed to both the padding of the original video, as well as any additionalpadding added during downsampling. The inventors have found that exposing ^^(^^) to bothduring training allows ^^(^^) to learn a set of weights that are able to accurately distinguishbetween native padding and added padding and thus allow the AI-compression pipeline toreconstruct the output images without hallucinating additional padding on top of the nativepadding or otherwise hallucinating some replacement content in place of the native videopadding of the original input.As described above, the dynamic padding approach described herein may be specified in aconfiguration file specifying rules or other heuristics of when to add padding. This may be inthe form of a function that takes as arguments the number of pixels in the dimension beingdownsampled, and a parameter corresponding to the amount being downsampled (e.g. 2, 3, 4,5, etc.), checking whether the pixel number is divisible by the downsampling parameter, andif yes, returning a "do not pad" flag, or if no, returning a "pad until divisible" flag. The flagmay then be passed to an another function that either adds empty pixels to the image matrix ortensor to pad it, or simply leaves it unchanged, as applicable.It is envisaged that the configuration file specifying the dynamic padding rules is also providedon the decode side, together with an indication of the dimensions of the original image, sothat the decode side of the pipeline is able to identify at which upsampling steps to removepadding, and how many lines of padding to remove. The original image dimensions may besent as side information in a bitstream, or specified in a user-set configuration file indicatingwhat resolution or dimension the AI-compression pipeline is being applied to.Figure 8 illustrates an AI-based compression pipeline corresponding to that shown in Figure 3,except now a plurality of downsampling layers 801a, 801b, 801c comprising convolution layerswith learnable weights are provided in the respective encoders of each of the I-frame module205, flow module 206, and residual module 210. The downsampling layers are configuredto apply padding according to the dynamic padding methods described herein. Further, acorresponding plurality of upsampling layers 802a, 802b, 802c are provided on the decodeside of the pipeline in the respective decoders of the I-frame module 205, flow module 206,and residual module 210. The upsampling layers are configured to remove the padding atthe corresponding upsampling step and by an amount as specified by the dynamic paddingconfiguration by which the padding was adding during the downsampling by the downsamplinglayers 801a, 801b, 801c.Concept 2: Spatially and / or Temporally Dependent Flow ModellingAs described above, in AI-based image compression, a neural network is used to map an inputimage ^^ to a latent variable ^^. The latent variable ^^ is losslessly encoded into a bitstream whichis transmitted, and then losslessly decoded back from the bitstream into the latent variable ^̂^,which is then passed through a further neural network to reconstruct an output image ^^ that isan approximation of the input image ^^.The step of losslessly encoding the latent ^^ first comprises quantising the latent variable ^^ andthen performing range or arithmetic encoding (or some other entropy encoding algorithm) onthe quantised latent variable ^^. Performing entropy encoding on the quantised latent ^^, relieson knowing a probability mass function (PMF) indicative of the distribution of the elementsof the quantised latent variable ^^. As the ground truth PMF (i.e. the true distribution) is notobtainable directly, it is instead modelled using an entropy model.One approach to entropy modelling the distribution of the elements of th quantised latent ^^ isto use a parametric distribution ^^(^^ = ^^^^ |^^), where ^^ are the parameters of the PMF. Knownparametric distributions include but are not limited to Gaussian or Laplacian distributions, inwhich case ^^ represents the mean ^^ and variance ^^ parameters of the distributions. In one toyexample, this gives us:∫^^(^^ = ^^ |^^ ^^+^^ / 2^^ ) =^^^^−^^ / 2 ^^(^^ |^^, ^^)^^^^ where ^^(^^ |^^, ^^) is a Gaussian or Laplacian, and ^^ is the quantisation bin size of the quantisationoperation (e.g. ^^ = 1 for integer quantisation in, for example, int8).This approach can be extended by conditioning the parameters of the PMF on some informationassociated with the latent ^^ being encoded. The idea is that the PMFs of the distributionsof each element of ^^ are not standalone functions but instead they are dependent on otherelements. This dependency may arise due to, for example, spatial or temporal correlationsin the input image(s). For example, adjacent pixels in an image are typically correlated witheach other and a corresponding correlation exists in the pixels of the latent ^^ representation.Capturing these correlations in the entropy model through the parameters of the PMF can bedone by making these parameters functions of some information ^^ associated with the ^^. Thatis:^^(^^ = ^^^^ |^^(^^), ^^(^^))The choice of ^^ (i.e. what information we condition ^^ and ^^ on is to an extent arbitrary buttwo known approaches are:(i) feed the latent representation ^^ into a hypernetwork (such as that shown in Figure 2)to produce a hyper latent representation which we refer to herein as ^^ on which ^^ and ^^ can bedirectly conditioned; or(ii) use an autoregressive probabilistic model that directly conditions ^^ and ^^ on someor all pixels of ^^, for example:^^(^^ = ^^^^ |^^(^^^^−1, ..., ^^^^−^^), ^^(^^^^−1, ..., ^^^^−^^))where ^^ is the number of pixels that ^^ and ^^ are conditioned on.A benefit of conditioning on the hyper latent representation ^^ is that quantisation bins can bedirectly centred around the values of ^^. That is, the quantisation of some unquantised latentrepresentation ^^ can be represented with the location shifted rounding function:^̂^^^ = ⌊^^^^ − ^^(^^)⌋ + ^^(^^)Location shifted rounding for quantisation is significantly more accurate than straight roundingbecause it allows the distribution of values to be centered closer to a true center of thedistribution to thereby minimise information loss (i.e. minimise the quantisation error orrounding error) when performing the rounding. The operation ⌊^^⌋ indicates a roundingoperation.Conversely, a benefit of an autoregressive approach that conditions ^^ and ^^ on other pixelsof the latent representation ^^ is that spatial context is directly captured. Whereby, as weautoregressively work through the ^^ pixels, each pixel of latent ^^ can in principle be modelledtaking into account the context of all previously decoded pixels of the latent ^^ to therebyproduce a highly expressive PMF for use in entropy encoding the quantised latent ^^ intothe bitstream. However, autoregressive approaches suffer from significant computationalcost as they must be performed serially. That is, to condition on ^^ and ^^ on ^^^^−1, we mustfirst condition ^^ and ^^ ^^^^−2, and so on up to ^^^^−^^. Further, autoregressive approaches arehighly sensitive to even very small errors so any very small errors introduced during locationshifted rounding due to tiny floating point rounding errors very quickly cascade into significantinaccuracies and mismatches between the encoder and decoder.A solution to this problem is to use a hybrid approach that conditions the location parameter ^^of the PMF on the previous pixels of the latent ^^ in an autoregressive manner, but conditioningthe scale parameter ^^ only on the hyper latent representation ^^, that is:^^(^^ = ^^^^ |^^(^^^^−1, ..., ^^^^−^^), ^^(^^))Practically, this PMF can be constructed or implemented by configuring the hyper-network toproduce not only the location and scale parameters ^^ and ^^, but also information indicativeof a dependence of one or more elements of ^^ on one or other elements of ^^. That is,information that attempts to predict what the results of a true autoregressive calculation wouldbe for each pixel without actually having to perform the full autoregressive calculation. Thisinformation may take the form of a spatial dependence tensor T or some other multidimensionaldata structure or format with elements having values that depend on an input hyper latent ^^ andon the weights of the hyper network learned during training. We thus get a PMF defined by:^^(^^ = ^^^^ |^^(T^^^^), ^^(^^))We can define this general approach for the specific case where we want to introduce locationshifted rounding as the method of quantisation of the latent ^^:^̂^∑^^−1∑^^−1^^ = ⌊^^^^ − ^^^^ − ^^=^^−^^ T^^ ^^ ^^ ^^⌋ + ^^^^ + ^^=^^−^^ T^^ ^^ ^^ ^^ As highlighted above, the spatial dependence tensor T has elements that, when applied to thevalues of pixels ^^ of latent ^^, modify those values to take into account how the unknowable, truedistribution of ^^ would have changed when modelled in an autoregressive manner to take intoaccount the dependence of those pixels on the values of other pixels ^^ of ^^. This is achievedwithout having to actually perform the full, computationally burdensome autoregressivemodelling. Note ^̂^ denotes a quantised representation of the latent ^^.Further, in practice, an AI-based compression pipeline may entropy encode not the quantisedlatent representation ^̂^ but the quantised normalised latent arising from the location shiftedquantisation process. That is, we can represent the quantised normalised latent ^^^^ (also referredto herein as information indicative of a quantisation residual or rounding residual or roundingamount) as follows:^̂^∑ ^^ = ⌊^^^^ − ^^^^ − ^^−1^^=^^−^^ T^^ ^^ ^^ ^^⌋ + ^^^^ +∑^^−1^^=^^−^^ T^^ ^^ ^^ ^^ The practical effect of this this approach is that the PMF of ^^^^ that we use for entropy encodingis guaranteed to have a ^^ parameter of zero (i.e. because of how we have constructed ^^^^, weare effectively guaranteeing that the ^^ parameters are substantially zero, that is: ^^(Ξ = |0, ^^).More specifically, if we return to our original hybrid approach formulation where we definedthe PMF as:^^(^^ = ^^^^ |^^(^^^^−1, ..., ^^^^−^^), ^^(^^)) if we apply this formulation to the quantised normalised latent representation ^^, the autoregres-sive part (i.e. ^^(^^^^−1, ..., ^^^^−^^)) is zero, and we get:) The profound effect of this formulation is that we can entropy decode the bitstream to produce^^ without needing anything but the bitstream and the scale parameter ^^ (which is obtainedfrom the decoding ^^ ^^ ^^^^^^ with the hyper decoder of the hyper network) as long as we use thespatial dependence tensor T when constructing ^^^^. Once we have obtained ^^^^, we can reconstruct ^̂^ by "undoing" the location shift using thelocation dependence tensor T by:^^^^ = ^^^^ + ^^^^ +∑^^−1^^=^^−^^ T^^ ^^ ^^ ^^ The spatial dependence tensor T is a lower triangular tensor because it emulates an autoregres-sive model (i.e. for a 2-dimensional tensor, the first row only has one non-zero (first) elementas there are no previous elements to depend on, the second row has two non-zero elements, thecurrent element and an element indicating a dependence on the first element, the third row hasthree non-zero elements, and so on. Because of this structure, the system can be treated as asystem of linear equations and can accordingly be solved using any known linear algebraicsolving algorithm in parallel (rather than serially) in a way that is accordingly significantlyquicker and more efficient than any autoregressive system which must be solved serially.However, the resulting reconstructed ^^^^ substantially corresponds to the latent representationthat would have been produced had the system been a purely autoregressive system and thus itis a more accurate reconstruction and requires fewer bits to produce than an entropy modelingapproach that relies on a PMF based purely on a hyper latent ^^ for both the ^^ and ^^ parameters.Example linear algebraic solving algorithms that are envisaged include but are not limited to:iterative solvers, directed acyclic graphs, implicit solvers, and so on.Note for completeness that, as described in the introductory section, we perform the hypernetwork encode and decode steps on the encode side which means we know, on the encode sideof the pipeline before transmitting the bitstream, what the ^^, ^^, and T parameters will be thatthe main decoder on the decode side will eventually get to use during decoding. Indeed it isthese parameters that are used to perform the entropy encoding on the encode side. Thus, oncewe have these parameters, we calculate the residual ^^^^ and pass this to the entropy encoderto produce a bitstream. We also pass the latent representation ^^ into the hyper network toproduce the hyper latent ^^ which is separately entropy encoded and included in the bitstreamas meta information, ready for the hyper deocder to receive to reconstruct ^^, ^^, and T.Once we have produced the reconstructed latent representation ^^^^, it is decoded with the maindecoder neural network to reconstruct the output image ^^ which is an approximation of theinput image.The above formulation for capturing spatial dependence in an autoregressive way in an entropymodel without having to actually perform a complete autoregressive computation can beextended to other components of an AI-based compression pipeline. This extension can beapplied to both spatial and / or temporal dependence (for example when pixels of one frame arein some way dependent on pixels of another frame for example where an object is movingacross a frame over the course of a sequence of frames of a video).Consider the illustrative flow-residual compression pipeline shown in Figure 9, which showsflow encoder / decoder networks and residual encoder / decoder networks of an AI basedcompression pipeline. This may be, for example, similar to the AI based compression pipelineof the type shown in Figure 3.The flow encoder neural network takes a current image ^^^^ and a previous image ^^^^−1, encodesthese into a flow latent representation which is optionally quantised and entropy encoded andtransmitted as a bit stream. On the decode side, the bitstream is received, entropy decoded intothe flow latent representation that the flow decoder neural network uses as input to producea representation of flow ^^ , e.g. a flow map. The flow map ^^ is applied to a previouslydecoded image ^^^^−1 to generate a warped version of that previously decoded image ^^^^−1,^^ . Thewarped version of the previously decoded image ^^^^−1,^^ is then fed into the residual encodernetwork, together with the current image ^^^^ to produce a residual latent representation which isoptionally quantised and entropy encoded and transmitted as a bit stream. On the decode side,the bitstream is received and entropy decoded back into the residual latent representation ^^ andused by the residual decoder neural network, in combination with information from the warpedversion of the previously decoded image ^^^^−1,^^ , to produce the reconstructed image ^^^^ . In thecase of Figure 9, the information associated with the warped previously decoded image ^^^^−1,^^is optionally first processed by module ^^ , referred to herein after as a composition adapter, forexample to downsample and / or pad it before it is fed into the residual decoder together withthe entropy decoded residual latent representation to produce the final reconstructed image^^^^ . This process may then be repeated for ^^^^+1 and so on to encode, transmit, and decode asequence of frames.As is further shown in Figure 9, the entropy encoding that produces the respective bitstreams204, 208, 212 is supported by corresponding a hyper network which in turn is supported bya hyper hyper network. The hyper network for the I-frame module 201 comprises a neuralnetwork 903a that is referred to as a hyper encoder, and a neural network 903b that is referredto as a hyper decoder. As is described in connection with Figure 2, the hyper encoder neuralnetwork 903a encodes the latent representation produced by the encoder neural network 203,and the hyper decoder neural network 905a produces the entropy parameters that define thedistribution of the latent representation. These parameters are used during entropy encoding toproduce the bitstream 204. The encoded latent representation produced by the hyper encoder903a is itself entropy encoded into a bitstream using an entropy model using entropy parametersobtained by encoding the encoded latent representation using the hyper hyper encoder 903b,and decoding that encoded latent representation using the hyper hyper decoder 905b. Thisnested structure can be extended as desired but it has been found that a hyper and hyper hypernetwork achieve good performance for run time trade off. The entropy parameters that areused to produce the bitstream associated with the hyper hyper network (or whatever the highestlevel of the nested structure is) may be fixed rather than produced by an encoder / decoder pair.As can be seen in Figure 9, this nested hyper network and hyper hyper network approach isalso provided in the optical flow module 206 in the form of hyper encoder neural network907a, hyper decoder neural network 909a, hyper hyper encoder neural network 909a and hyperhyper decoder neural network 909b. The same is also provided in the residual module 210 inthe form of hyper encoder neural network 911a, hyper decoder neural network 913a, hyperhyper encoder neural network 911b and hyper hyper decoder neural network 913b.In each case, the spatial dependence tensor formulation described above may be used as animproved entropy model for use when entropy encoding by the I-frame module 201 encoder203 and decoder 205, by the flow module 206 encoder 207 and decoder 209, and / or by theresidual module 210 encoder 211 and decoder 213. That is, the hyper encoders 903a, 907a,911a and the hyper decoders 905a, 909a, 913a may be configured to not only output entropyparameters ^^ and ^^ but also a spatial dependence tensor T, whereby for each encoder-decoderpair, it is quantised normalised latent representation arising from a location shifted quantisationoperation performed on the corresponding latent representation that is entropy encoded intothe bitstream and subsequently decoded.In the case of flow, the above approach is particularly advantageous because optical flow mapsoften have spatial correlations that can be exploited for more efficient compression.Figure 10 illustratively shows the above-described spatial tensor approach implemented ina flow module 206 such as that shown in Figure 9. As in Figure 9, an input image ^^^^ of asequence of images is received together with a second image from the sequence of images ^^^^−1.These are fed into an encoder to produce a latent representation of optical flow information^^ ^^ ^^^^^^ indicative of a difference between the two images. The latent representation of opticalflow information ^^ ^^ ^^^^^^ is fed into a hyper encoder 907a of a hyper network to produce hyperlatent representation of the optical flow information ^^ ^^ ^^^^^^ , which is fed into an entropy encoderto produce a first part of a bitstream 1002.The hyper latent representation of the optical flow information ^^ ^^ ^^^^^^ is also fed into a hyperdecoder 909a of the hyper network (all while still on the encode side) to produce the entropyparameters ^^ ^^ ^^^^^^ and ^^ ^^ ^^^^^^ , as well as a spatial dependence tensor T ^^ ^^^^^^ indicative of spacialdependencies that respective pixels of the latent representation of optical flow information^^ ^^ ^^^^^^ have with respect to each other.The entropy parameter ^^ ^^ ^^^^^^ and the spatial dependence tensor T ^^ ^^^^^^ are used to performlocation shifted quantisation 1003 on the latent representation of optical flow information^^ ^^ ^^^^^^, which in turn is used to calculate the quantised normalised latent representation ^^ ^^ ^^^^^^:^̂^ , ^^ ^^^^^^ = ⌊∑^^−1∑^^−1^^ ^^^^, ^^ ^^^^^^ − ^^^^, ^^ ^^^^^^ − ^^=^^−^^ T^^ ^^ , ^^ ^^^^^^^^ ^^ , ^^ ^^^^^^⌋ + ^^^^, ^^ ^^^^^^ + ^^=^^−^^ T^^ ^^ , ^^ ^^^^^^^^ ^^ , ^^ ^^^^^^ Note that, for ease of notation, we refer to latent representation of optical flow information inpre-quantised state in this equation as ^^^^, ^^ ^^^^^^ which corresponds to the latent representation ofoptical flow information ^^ ^^ ^^^^^^ flow encoder 207.With ^^ ^^ ^^^^^^ calculated, we feed it and ^^ ^^ ^^^^^^ output by the hyper decoder 909a into an entropyencoder 1004 to produce a second part of the bitstream 1005.The first part of the bitstream 1002 and the second part of the bitstream 1005 may be combinedinto a single bitstream 1006 which is transmitted 1007 to a the decode side, which may beimplemented on a physically and geographically separate system to that on which the steps ofFigure 10 are performed, illustrated by a cloud 1008Figure 11 illustratively shows the decode side of the compression pipeline described inconnection with Figure 10. The bitstream 1101 is received 1102 from the system of Figure10 and split back up into its constituent first part 1103 and second part 1104 respectivelyassociated with ^^ ^^ ^^^^^^ and ^^ ^^ ^^^^^^.The first part of the received bitstream 1103 is entropy decoded by an entropy decoder 1104 toproduce a resconstruction of the latent representation of optical flow information ^^ ^^ ^^^^^^. Thisis fed into a hyperdecoder 909a to reconstruct the entropy parameters ^^ ^^ ^^^^^^ and ^^ ^^ ^^^^^^ , as wellas a reconstruction of the spatial dependence tensor T ^^ ^^^^^^.The second part of the received bitstream 1105 is then fed into an entropy decoder 1106 (whichmay be the same entropy decoder as that used to entropy decode the first part of the bitstream1103, or a different one), to produce a reconstructed quantised normalised latent representation^^ ^^ ^^^^^^.The reconstructed quantised normalised latent representation ^^ ^^ ^^^^^^, reconstructed entropyparameter ^^ ^^ ^^^^^^, and reconstructed spatial dependence tensor T together form an algebraicsystem of equations that can be solved 1107 using any known system of linear equationsolver algorithm to reconstruct the latent representation of optical flow information ^̂^ ^^ ^^^^^^. Inthe example of Figure 11, it is envisaged that an iterative solver is used but this is illustrativeonly and other methods including directed acyclic graphs, implicit solvers, and so on may alsobe used. As will be appreciated, these methods are parallelisable so may be solved efficientlycompared to any non-parallelisable methods.The reconstructed latent representation of optical flow information ^̂^ ^^ ^^^^^^ is then fed into theflow decoder 209 to produce reconstructed optical flow information ^^^^ indicative of a differencebetween the two images ^^^^ and ^^^^−1 but now available on the decode side.The reconstructed optical flow information ^^^^ can accordingly be used on the decode side towarp a previously decoded image ^^^^−1 to produce a warped version of that image, that may beused for subsequent operations of the compression pipeline, such as for example in the residualmodule 210 as illustrated in Figures 3 and / or 9.Whilst not shown, the encoders and decoders of the residual module 210 may also be providedwith a hyper network that produces its own spatial dependence tensor indicative of spatialdependencies of the respective pixels of latent representations produced by the residual encoderwith respect to each other.In this way, the concept of a spatial dependence tensor may be extended from use with theencoder and decoders of the I-frame module 201, the encoders and decoders of the flow andresidual modules 206, 210, thereby enhancing the operation of the AI-pipeline by facilitatingaccuracy improvements that emulate autoregressive entropy modelling without incurring theassociated computational overheads.The above-described spatial dependence tensor approach may be applied to predict what spatialdependencies exist with respect to all previous pixels of a given input (e.g. if an input is a 1080x 1920 pixel image, the spatial dependence tensor for the final pixel may capture global spatialdependencies with respect to all other pixels). However, it is also envisaged that this approachmay be scaled down to sub-global level to predict spatial dependencies on a patch-wise basis.For example, a given input may be divided into patches of n by n pixels, for each of which aseparate spatial dependence kernal may be predicted, which together form a spatial dependencetensor.Note for completeness that the spatial dependence tensor is described as a tensor above forgenerality, however it may be any dimension, including for example a one-dimensional vector,a two-dimensional matrix, a three or more dimensional tensor and so on.For ease of understanding, the quantised normalised latent representation ^^ (i.e informationindicative of a quantisation residual) and the residual of a flow-residual image compressionpipeline (i.e. an image residual) are not the same. That is, information indicative of aquantisation residual ^^ represents the quantisation or rounding error of the quantisation processwhereas an image residual is a difference between two images indicating in some way howthey differ from each other and, in its simplest form, may be the direct subtraction of pixelvalues of one image from the other.Concept 3: Learned noise trainingAn example of a known neural network architecture comprises consecutive convolution layersand activation layers. For example, a convolution layer followed by an activation layer, followedby a convolution layer followed by an activation layer, and so on. The activation layersapply an activation function to their input, typically a rectified linear unit (ReLU) activationfunction. The outputs are then passed on to the next convolution layer, whose output is thenpassed onto the next activation layer and so on. In the context of an AI compression pipeline,this type of architecture may be used for various components. For example, on the encodeside it may be the encoder that produces the latent representation of the input image, thehyper-encoder that produces the hyper-latent representation, the hyper-hyper-encoder, thatproduces the hyper-hyper-latent representation, and so on. On the decode side it may be thedecoder, hyper-decoder, hyper-hyper-decoder and so on. In addition, in a flow residual modelused for video compression, it may be used in the flow and / or residual encoder and / or decoders.When implementing these architectures in practice, in an ideal scenario, during inference, theweights and biases of the layers would be floating point data types of arbitrary precision (e.g.fp16, fp32 or fp64). Operating in floating point precision allows activation functions like theReLU and others, such as leaky ReLU, to avoid becoming saturated and / or avoids ReLU deathoccurring. Indeed training is typically performed in floating point precision.However, when implementing AI compression pipelines where inference is intended to be runin real time on devices with limited memory, compute, and power available, real time with afocus on speed, operating on floating point data types results in an unacceptable draw on thesememory, compute, and power resources. For example, running an AI compression pipeline ona mobile device such as a smartphone or a laptop in floating point precision can in many casesresult in very high CPU (or NPU, or TPU) usage, very high power draw (reducing battery lifetimes), and very high memory usage (preventing other applications from functioning).Instead, in known AI compression pipelines, the weights and biases of the layers, and thelatent representation output by the encoder neural network are quantized to integer data typeprecision (also known as fixed-point precision). In the case of int8, this means quantising theweights, biases and latent representation to a discrete set of 256 values.Whilst it is possible for the operations used to apply the convolutions and activation function,and for their intermediate outputs (e.g. activations) to remain implemented in floating pointprecision this again results in memory, compute, and power resource difficulties. Instead, notonly are the weights, biases and inputs quantized into the lower precision integer space (i.e.fixed point precision of e.g. int8), but the outputs and inputs of the intermediate layers are alsoquantized into the lower precision space. For example, an int8 input to a convolution and / oractivation layer may result in a int32 output which is then requantised back into int8, whichmeans the data flow through the network remains in the fixed point precision space. Thisallows inference to be run on devices with lower hardware overheads but there are a number ofconsequences of this, particularly in training the neural networks of the pipeline.More generally, training of neural networks is often performed with the weights, biases andactivations kept in the highest precision possible, e.g. fp32, as this allows the values of theweights, biases and activations of the networks to be updated at a far more granular level thanin e.g. int8 which can contribute to greater training stability and convergence to a true globalminimum. However, subsequent quantisation of the fp32 values into a lower precision space,such as int8, can significantly alter the values of the weights, biases and activations. Theoverall difference between the high precision values and the low precision, quantised values ofweights, biases and / or activations of a network is referred to herein after as the quantisationgap. The larger the quantisation gap, the higher the likelihood that a network that works wellin high precision space will perform poorly or in unexpected ways in low precision space.Reducing the quantisation gap, that is helping the networks weights, biases and / or activationsto converge during training in high precision space to values that are approximately similar inlow precision space, is a challenging task.One approach to solving this task is known as quantisation aware training (QAT). QAT is atechnique used in deep learning to simulate the effects of quantization on the model duringthe training phase. In some examples, QAT incorporates the quantization process into thetraining pipeline. This means that during training, the weights and activations are actuallydirectly quantized (i.e., converted from floating-point to lower-precision formats like int8 orint16), and the model operates as if it were in a quantized state. This allows the model toadapt to the quantization effects, such as reduced numerical precision and the possibility ofsaturation. There are a number of variants of QAT, for example during the forward pass,weights and activations may be quantized, but for the backward pass (during which gradientsare computed for learning), some implementations may use higher precision to maintain theeffectiveness of the gradient descent optimization process. This mixed precision approachhelps in minimizing the performance degradation due to quantization. Finally, after the initialtraining with quantization simulation, the model might undergo fine-tuning to help to recoverany loss in accuracy introduced by quantization.An alternative approach to directly quantizing the weights, biases and / or activations is to addnoise to the values of one or more of the weights, biases and / or activations of the networkduring all or parts of training. The quantisation gap may be modelled as an error betweenthe high precision values and the low precision values. Accordingly, introducing an error inthe form of noise into one or more of the high precision (e.g. fp32) weights, biases and / oractivations during training emulates the quantisation process and thus makes the networksquantisation aware during training. However, naively adding some arbitrary noise to allweights, biases and / or activations in an uncontrolled manner is detrimental to the trainingstability and can destroy the training signal. Identifying what to add noise to and what formthat noise takes is a challenging problem.In the context of AI-based compression pipelines the inventors have realised that the quantisaiongap is disproportionately affected by the quantisation of the activations or outputs of the layersof the neural networks (e.g. the encoder neural networks that produce the latent representation,the hyper latent representation and where applicable a hyper hyper latent representation, as wellas their corresponding decoder neural networks), compared to the effect of the quantising ofthe weights and / or biases of the networks. Accordingly, adding noise during training to theseactivations (i.e. the outputs of the layers as they are passed between layers) disproportionatelyimproves the quantisation gap.We take the concept of adding noise to all outputs of all layers as a starting point. From thisstarting point we then consider whether there are any layers that we do not wish to add noiseto, and what is the form of noise that may be added in order to retain stability during training.In other words, which are the outputs where adding noise is detrimental to training stabilityand accordingly to which we do not wish to add noise, and what is the form of the noise that isto be added.The inventors have found that the training stability of the neural networks of an AI basedcompression pipeline can be made surprisingly robust, mostly irrespective of the layers thenoise is being added to, when the following condition is met: the noise that is added to theoutputs of one or more of the layers during training is itself learnable. This overcomes theproblem that naively adding some arbitrary noise to the outputs destroys the training signal.This can be understood at a high level as the networks being given the freedom to learn toadd the right form and amount of noise to to each output to maintain training stability. Forexample, during training the networks may learn to add a high amount of noise to one outputof one layer but almost none to another, and so on.This concept is illustrated in the algorithm below:

[0002] Algorithm 2 Noise training with learned noiseInputs: Neural network ^^ (^^)Config specifying which layer outputs to add noise ^^(^^) toVariables: Optimizer ^^, gradient computation flag ^^^^^^^^.Initialise: ^^^^^^^^ = ^^^^^^^^ for ^^ (^^) and ^^, and Training loop counter ^^ = 0.while ^^ < ^^ doForward pass through ^^ (^^).Compute loss and perform backward pass with ^^^^^^^^.Apply optimizer ^^ to update ^^ (^^) and ^^.^^ ← ^^ + 1end whileA neural network ^^ (^^) and an accompanying configuration is provided specifying the outputsof which layers noise ^^(^^) will be added to during training. The noise ^^(^^) may defined as anoise function or plurality of noise functions parameterised by one or variables ^^. The neuralnetworks and the noise function(s) are initialised e.g. with random values, and the grad flag isset to true for both the weights and / or biases of the neural network ^^ (^^) as well as the noise^^(^^), indicating that they are trainable. Training for for ^^ steps then commences by repeatedlyperforming a forward pass through ^^ (^^) with the noise ^^(^^) added to outputs of layersspecified in the configuration. The loss is computed, may be obtained with a rate distortionloss function (e.g. as described above in the introductory section), and back propagation isperformed through the elements where ^^^^^^^^ is set to true. An optimiser ^^, e.g. SGD, Adam,or some other optimisation algorithm is applied to update the weights and / or biases of ^^ (^^) andthe parameters of the noise ^^(^^). Training ends after some predetermined condition is met, forexample after a predetermined number of epochs, training steps, or after the loss plateaus or insome other way converges. As described above, it is envisaged that this training loop may beperformed in a high precision space such as fp32. After training, the implementation of ^^ (^^)may be converted to perform inference in a lower precision space such as int8 whereby theweights and biases may be converted into int8 values, and the operations (such as matmuls andso on) used during inference may be specified to operate fully or in part in a lower precisionspace as applicable.The noise function ^^(^^) may comprise any suitable parameterised function such as, but notlimited to, a Gaussian noise function, a Poisson noise function, a uniform noise function, andso on. The greater the number of parameters, the more flexibility the networks have in learningwhat noise to add to what layers, and so on. In one implementation, it is envisaged that thenoise function is defined as a Gaussian with learnable standard deviation and mean parameters.The above approach may be used to add noise to the outputs of one or more layers of thethe architectures of Figures 1 and 2 that comprise e.g. an encoder, decoder and / or hyperencoder and / or hyper decoder, and / or a hyper hyper encoder and hyper hyper decoder (notshown). It may also be used to add noise to the outputs of one or more layers of more complexarchitectures such as that of Figure 3, as well as that of Figure 12, which will now be describedConsider the illustrative flow-residual compression pipeline shown in Figure 12, whichshows flow encoder / decoder networks and residual encoder / decoder networks of an AI basedcompression pipeline. This may be, for example, similar to the AI based compression pipelineof the type shown in Figure 3.The flow encoder neural network takes a current image ^^^^ and a previous image ^^^^−1, encodesthese into a flow latent representation which is optionally quantised and entropy encoded andtransmitted as a bit stream. On the decode side, the bitstream is received, entropy decoded intothe flow latent representation that the flow decoder neural network uses as input to producea representation of flow ^^ , e.g. a flow map. The flow map ^^ is applied to a previouslydecoded image ^^^^−1 to generate a warped version of that previously decoded image ^^^^−1,^^ . Thewarped version of the previously decoded image ^^^^−1,^^ is then fed into the residual encodernetwork, together with the current image ^^^^ to produce a residual latent representation which isoptionally quantised and entropy encoded and transmitted as a bit stream. On the decode side,the bitstream is received and entropy decoded back into the residual latent representation ^^ andused by the residual decoder neural network, in combination with information from the warpedversion of the previously decoded image ^^^^−1,^^ , to produce the reconstructed image ^^^^ . In thecase of Figure 12, the information associated with the warped previously decoded image ^^^^−1,^^is optionally first processed by module ^^ , referred to herein after as a composition adapter, forexample to downsample and / or pad it before it is fed into the residual decoder together withthe entropy decoded residual latent representation to produce the final reconstructed image^^^^ . This process may then be repeated for ^^^^+1 and so on to encode, transmit, and decode asequence of frames.As is further shown in Figure 12, the entropy encoding that produces the respective bitstreams204, 208, 212 is supported by corresponding a hyper network which in turn is supported bya hyper hyper network. The hyper network for the I-frame module 201 comprises a neuralnetwork 1203a that is referred to as a hyper encoder, and a neural network 1203b that is referredto as a hyper decoder. As is described in connection with Figure 2, the hyper encoder neuralnetwork 1203a encodes the latent representation produced by the encoder neural network203, and the hyper decoder neural network 1205a produces the entropy parameters thatdefine the distribution of the latent representation. These parameters are used during entropyencoding to produce the bitstream 204. The encoded latent representation produced by thehyper encoder 1203a is itself entropy encoded into a bitstream using an entropy model usingentropy parameters obtained by encoding the encoded latent representation using the hyperhyper encoder 1203b, and decoding that encoded latent representation using the hyper hyperdecoder 1205b. This nested structure can be extended as desired but it has been found that ahyper and hyper hyper network achieve good performance for run time trade off. The entropyparameters that are used to produce the bitstream associated with the hyper hyper network (orwhatever the highest level of the nested structure is) may be fixed rather than produced by anencoder / decoder pair.As can be seen in Figure 12, this nested hyper network and hyper hyper network approachis also provided in the optical flow module 206 in the form of hyper encoder neural network1207a, hyper decoder neural network 1209a, hyper hyper encoder neural network 1209a andhyper hyper decoder neural network 1209b. The same is also provided in the residual module210 in the form of hyper encoder neural network 1211a, hyper decoder neural network 1213a,hyper hyper encoder neural network 1211b and hyper hyper decoder neural network 1213b.Returning to the concept of noise training in the context of the architecture of Figure 12,learned noise, as parameterised by one or more noise functions ^^(^^) may be added to one ormore outputs of one or more of the layers of any of the main encoder 203 and decoder 205 ofthe I-frame module 201, any of the hyper encoder 1203a and hyper decoder 1205a, and / or anyof the hyper hyper encoder 1203a and hyper hyper decoder 1205b.Similarly, learned noise may be added to one or more outputs of one or more of the layers ofany of the main encoder 207 and decoder 209 of the flow module 206, and / or to any of thehyper encoder 1207a and hyper decoder 1209a, and / or any of the hyper hyper encoder 1207band hyper hyper decoder 1209b.Similarly, learned noise may be added to one or more outputs of one or more of the layers ofany of the main encoder 211 and decoder 213 of the residual module 210, and / or to any of thehyper encoder 1211a and hyper decoder 1213a, and / or any of the hyper hyper encoder 1211band hyper hyper decoder 1213b.For completeness, adding noise may mean taking a value and adding or subtracting somenumber to that value to distort and introduce an error into that value compared to what itoriginally was. In the case of an AI compression pipeline, the outputs of the layers may bevectors, matrices or tensors and accordingly, adding noise to these outputs may mean addinga different value to each element of the vector, matrix or tensor, as determined by the noisefunction or noise functions ^^(^^) being used to add the noise. More generally the parametersof the noise function or functions ^^(^^) may be different for different elements of the vector,matrix or tensor they are being applied to and during training, the values of these plurality ofparameters (that may also be in the form of a tensor) are updated and thus learned.As described above, this allows the networks to decide whether or not it is advantageous tosignificantly distort the values of an output with a high degree of noise, or only partially or notdistort the values.Returning to Figure 12, the outputs of the hyper decoders 1205a, 1209a, 1213a compriseinformation associated with the distribution of output of the encoder of the main network towhich the hyper network is attached, e.g. the latent representation, the flow latent representationand the residual latent representation, and is used to efficiently entropy encode these elementsinto a bitstream, as well as entropy decode these elements from the bitstream. This distributioninformation may thus comprise one or more mean values ^^ (also referred to as a zero-pointvalue) and / or standard deviation values ^^ (also referred to as a scale value) associated withthose latent representations (e.g. in the form of a tensor). The inventors have found counter-intuitively that adding noise to the standard deviation values used for entropy encoding anddecoding during training does not improve the quantisation gap problem and in some casescan make it worse. Accordingly, it is envisaged that no noise be added to the outputs of layerscorresponding to the standard deviation values produced by the hyper decoders 1205a, 1209a,1213a. The same reasoning applies to the hyper hyper decoders 1205b, 1209b, 1213b in thatno noise is applied to the outputs corresponding to the standard deviation values associatedwith the entropy encoding and decoding.In contrast, adding learned noise to the mean values output by the hyper decoders 1205a,1209a, 1213a and / or hyper hyper decoders 1205b, 1209b, 1213b does reduce the size of thequantisation gap. Accordingly it is envisaged that noise is added to these outputs.In more general terms, each of the neural networks of the encoders and decoders, hyperencoders and hyper decoders, and hyper hyper encoders and hyper hyper decoders across allthe modules 201, 206, 210 in the compression pipeline may comprise one or more of thefollowing layers that define an operation or operation type: convolution layer (that performsa convolution operation), activation layer (that performs an activation operation e.g. ReLuand so on), linear layer (that performs a linear transformation), output layer (the final layer ofa network), and other layer types. Subject to not adding noise to the scale parameter outputby the hyper decoders and / or hyper hyper decoders, it is envisaged that learned noise may beadded to the outputs of substantially all of the other layers. As described above, by makingthe form of the noise learned through optimisable noise functions, the networks are able todecide during training whether to significantly distort the outputs significantly with noise,or leave them substantially unchanged with only very small or no noise added to the values.The difficulty of noise training thus turns into a "fire and forget" method whereby manuallyspecifying how much noise is added, and in what form it is added to different outputs ofdifferent layers is eliminated and the networks learn the optimum noise to use in each layerduring noise training. The resulting quantisation awareness of the networks so-trained isaccordingly improved over any handcrafted method.In a further implementation, it is envisaged that some layers and the operations that definesthem may be fused. For example, two consecutive layers that apply a linear operation can befused into a single operation. In this case, noise may be added to the fused layer as a whole.In a further implementation, at least some layers may share noise functions, for example wherelayers are identical similar (e.g. a convolution layer in the hyper decoder of the I-frame modulemay be the same in terms of dimensions, stride, and so on as in the hyper decoder of the flowmodule). In these types of case, the noise may be learnable for only one of the layers and thenshared to the other identical or similar layers across the pipeline. This can improve trainingstability as it synchronises the noise across identical or similar layers in preparation for howthe AI compression pipeline may operate in quantised space where the quantisation parametersused to quantise the neural networks from high precision to low precision space are sharedacross the pipeline to reduce resource overheads. This approach may also improve trainingstability and have a regularising effect as the overall number of learnable noise parametersacross the pipeline is smaller when at least some of them are shared.Finally, it will be understood that there may be some layers of one or more of the neuralnetworks that effectively operate as pre- or post- processing layers such as one or more pixelshuffle or unshuffle layers that convert spacial dimensions into channel dimensions and viceversa, or one or more layers that convert RGB data into YUV data, layers that quantise theinput / output images to match a predetermined data format, and so on. It is envisaged that noiseis not added to these types of pre- and / or post-processing layers.The subject matter and the functional operations described in this specification can beimplemented in digital electronic circuitry, in tangibly-embodied computer software orfirmware, in computer hardware, including the structures disclosed in this specification andtheir structural equivalents, or in combinations of one or more of them. The subject matterdescribed in this specification can be implemented as one or more computer programs, i.e.,one or more modules of computer program instructions encoded on a tangible non transitoryprogram carrier for execution by, or to control the operation of, data processing apparatus.Alternatively or in addition, the program instructions can be encoded on an artificially generatedpropagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, thatis generated to encode information for transmission to suitable receiver apparatus for executionby a data processing apparatus. The computer storage medium can be a machine-readablestorage device, a machine-readable storage substrate, a random or serial access memory device,or a combination of one or more of them. The computer storage medium is not, however, apropagated signal.Further, whilst the present disclosure is described as applying to an image or images, this mayalso refer to a representation of an image or images or information associated with an imageor images. For example, it may apply to optical flow information, residual information, andso on, or indeed to any feature map produced in the AI-based compression pipeline wheredownsampling may be used to reduce overall compute and memory overheads.The term “data processing apparatus” encompasses all kinds of apparatus, devices, andmachines for processing data, including by way of example a programmable processor, acomputer, or multiple processors or computers. The apparatus can include special purposelogic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specificintegrated circuit). The apparatus can also include, in addition to hardware, code that createsan execution environment for the computer program in question, e.g., code that constitutesprocessor firmware, a protocol stack, a database management system, an operating system, ora combination of one or more of them.A computer program (which may also be referred to or described as a program, software, asoftware application, a module, a software module, a script, or code) can be written in anyform of programming language, including compiled or interpreted languages, or declarative orprocedural languages, and it can be deployed in any form, including as a stand alone program oras a module, component, subroutine, or other unit suitable for use in a computing environment.A computer program may, but need not, correspond to a file in a file system. A program can bestored in a portion of a file that holds other programs or data, e.g., one or more scripts storedin a markup language document, in a single file dedicated to the program in question, or inmultiple coordinated files, e.g., files that store one or more modules, sub programs, or portionsof code. A computer program can be deployed to be executed on one computer or on multiplecomputers that are located at one site or distributed across multiple sites and interconnected bya communication network.The processes and logic flows described in this specification can be performed by one or moreprogrammable computers executing one or more computer programs to perform functionsby operating on input data and generating output. The processes and logic flows can also beperformed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g.,an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).Computers suitable for the execution of a computer program include, by way of example,can be based on general or special purpose microprocessors or both, or any other kind ofcentral processing unit. Generally, a central processing unit will receive instructions and datafrom a read only memory or a random access memory or both. The essential elements ofa computer are a central processing unit for performing or executing instructions and oneor more memory devices for storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer data to, or both, one or moremass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks.However, a computer need not have such devices. Moreover, a computer can be embedded inanother device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio orvideo player, a VR headset, a game console, a Global Positioning System (GPS) receiver, aserver, a mobile phones, a tablet computer, a notebook computer, a music player, an e-bookreader, a laptop or desktop computer, a PDAs, a smart phone, or other stationary or portabledevices, that includes one or more processors and computer readable media, or a portablestorage device, e.g., a universal serial bus (USB) flash drive, to name just a few.Computer readable media suitable for storing computer program instructions and data includeall forms of non-volatile memory, media and memory devices, including by way of examplesemiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magneticdisks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM andDVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in,special purpose logic circuitry.The subject matter described in this specification can be implemented in a computing systemthat includes a back end component, e.g., as a data server, or that includes a middlewarecomponent, e.g., an application server, or that includes a front end component, e.g., a clientcomputer having a graphical user interface or a Web browser through which a user can interactwith an implementation of the subject matter described in this specification, or any combinationof one or more such back end, middleware, or front end components. The components of thesystem can be interconnected by any form or medium of digital data communication, e.g., acommunication network. Examples of communication networks include a local area network(“LAN”) and a wide area network (“WAN”), e.g., the Internet.The computing system can include clients and servers. A client and server are generally remotefrom each other and typically interact through a communication network. The relationship ofclient and server arises by virtue of computer programs running on the respective computersand having a client-server relationship to each other.While this specification contains many specific implementation details, these should beconstrued as descriptions of features that may be specific to particular examples of particularinventions. Certain features that are described in this specification in the context of separateexamples can also be implemented in combination in a single example. Conversely, variousfeatures that are described in the context of a single example can also be implemented inmultiple examples separately or in any suitable subcombination.Similarly, while operations are depicted in the drawings in a particular order, this should notbe understood as requiring that such operations be performed in the particular order shownor in sequential order, or that all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processing may be advantageous.Moreover, the separation of various system modules and components in the examples describedabove should not be understood as requiring such separation in all examples, and it should beunderstood that the described program components and systems can generally be integratedtogether in a single software product or packaged into multiple software products.

Claims

1. CLAIMS1. A method for lossy image or video encoding, transmission and decoding, the methodcomprising the steps of:receiving an input image at a first computer system;performing consecutive downsampling operations on the input image to produce adownsampled input image and adding padding to the input image between one or more of theconsecutive downsampling operations;encoding the downsampled input image using a first trained neural network to produce alatent representation;transmitting the latent representation to a second computer system; anddecoding the latent representation using a second trained neural network to produce adownsampled output image;performing consecutive upsampling operations on the downsampled output image toproduce an output image and removing padding between one or more of the consecutiveupsampling operations, wherein the output image is an approximation of the input image.

2. The method of claim 1, comprising:adding padding to the input image between some but not all of the consecutive down-sampling operations; andremoving padding from the downsampled output image between some but not all of theconsecutive upsampling operations.

3. The method of claim 1 or 2, comprising adding padding to the input image before a down-sampling operation where a dimension being downsampled is indivisible by a predetermineddownsampling amount.

4. The method of any of claims 1 to 3, comprising adding padding to the input image before adownsampling operation until a dimension being downsampled is divisible by a predetermineddownsampling amount.

5. The method of any of claims 1 to 4, comprising:identifying a height and / or width dimension of the input image;calculating a sequence of height and / or width dimensions of the input image afterperforming each of the downsampling operations;generating an add padding schedule, the add padding schedule specifying between whichconsecutive downsampling operations the step of adding padding is to be performed; andusing the add padding schedule to perform said adding padding.

6. The method of any of claims 1 to 5, comprising:identifying a height and / or width dimension of the downsampled output image;calculating a sequence of height and / or width dimensions of the downsampled outputimage after performing each of the upsampling operations;generating a remove padding schedule, the remove padding schedule specifying betweenwhich consecutive downsampling operations the step of removing padding is to be performed;and using the remove padding schedule to perform said removing padding.

7. The method of claim 6, wherein the remove padding schedule corresponds to the addpadding schedule in reverse.

8. The method of any of claims 1 to 7, wherein the input image has a height dimensionof 1080 pixels, wherein the each downsampling operation halves the height dimension, andwherein said adding padding is performed in the height dimension when the input image heightdimension is 135 pixels.

9. The method of any of claims 1 to 8, wherein the input image has a height dimensionof 720 pixels, wherein the each downsampling operation halves the height dimension, andwherein said adding padding is performed in the height dimension when the input image heightdimension is 45 pixels.

10. The method of any of claims 1 to 9, wherein at least one of the consecutive downsamplingoperations comprises a convolution operation performed with one or more convolution layersof the first neural network.

11. The method of any of claims 1 to 10, wherien at least one of the consecutive upsamplingoperations comprises a convolution operation performed with one or more convolution layersof the second neural network.

12. A method of training one or more neural networks, the one or more neural networks beingfor use in lossy image or video encoding, transmission and decoding, the method comprisingthe steps of:receiving a first input image at a first computer system;performing consecutive downsampling operations on the input image to produce adownsampled input image and adding padding to the input image between one or more of theconsecutive downsampling operations;encoding the downsampled input image using a first trained neural network to produce alatent representation;decoding the latent representation using a second trained neural network to produce adownsampled output image;performing consecutive upsampling operations on the downsampled output image toproduce an output image and removing padding between one or more of the consecutiveupsampling operations, wherein the output image is an approximation of the input image.evaluating a function based on a difference between the input image and the outputimage; updating the parameters of the first neural network and the second neural network basedon the evaluated function; andrepeating the above steps using a first set of input images to produce a first trained neuralnetwork and a second trained neural network.

13. The method of claim 12, comprising:adding padding to the input image between some but not all of the consecutive down-sampling operations; andremoving padding from the downsampled output image between some but not all of theconsecutive upsampling operations.

14. The method of claim 12 or 13, comprising adding padding to the input image before a down-sampling operation where a dimension being downsampled is indivisible by a predetermineddownsampling amount.

15. The method of any of claims 12 to 14, comprising adding padding to the input imagebefore a downsampling operation until a dimension being downsampled is divisible by apredetermined downsampling amount.

16. The method of any of claims 12 to 15, wherein the downsampling comprises performingone or more convolution operation using one or more convolution layers of the first neuralnetwork, and wherein updating the parameters of the first neural network comprises updatingweights of the one or more convolution layers to produce a padding-aware first neural network.

17. The method of any of claims 12 to 16, wherein the upsampling comprises performingone or more convolution operations using one or more convolution layers of the secondneural network, and wherein updating the parameters of the second neural network comprisesupdating the weights of the one or more convolution layers to produce a padding-aware secondneural network.

18. A data processing system configured to perform the method of any one of claims 1 to 17.

19. A data processing apparatus configured to perform the method of claims 1 to 17.

20. A computer program comprising instructions which, when the program is executed by acomputer, cause the computer to carry out the method of claims 1 to 17.

21. A computer-readable storage medium comprising instructions which, when executed by acomputer, cause the computer to carry out the method of claims 1 to 17.

22. A method for lossy image or video encoding and transmission, and decoding, the methodcomprising the steps of:receiving a first image and a second image at a first computer system;with a first neural network, producing a latent representation of optical flow informationusing the first image and the second image, the optical flow information being indicative of adifference between the first image and the second image;quantising the latent representation using information indicative of a dependence of oneor more elements of the latent representation on one or more other elements of the latentrepresentation to produce a quantised normalised latent representation;transmitting the quantised normalised latent representation to a second computer system;reconstructing the latent representation from the quantised normalised latent representa-tion using the information indicative of the dependence of one or more elements of the latentrepresentation on one or more other elements of the latent representation;with a second neural network, decoding the latent representation of optical flow informa-tion to produce an approximation of the optical flow information; andwith a third neural network, producing an output image using the approximation ofoptical flow information, wherein the output image is an approximation of the first image.

23. The method of claim 22, wherein said reconstructing comprises solving a system of linearequations based on the information indicative of the dependence of one or more elements ofthe latent representation on one or more other elements of the latent representation.

24. The method of of claim 22 or 23, comprising:with a fourth neural network, producing a hyper latent representation of the optical flowinformation using the latent representation;transmitting the hyper latent representation to the second computer system; andwith a fifth neural network, producing said information indicative of the dependence ofone or more elements of the latent representation on one or more other elements of the latentrepresentation using the hyper latent representation.

25. The method of claim 24, wherein the values of the latent representation are distributedaccording to a distribution defined by one or more parameters, and wherein the methodcomprises: with the fifth neural network, producing the one or more parameters using the hyperlatent representation; andentropy encoding the quantised normalised latent representation into a bitstream usingthe one or more parameters, wherein said transmitting comprises transmitting the bitstream.

26. The method of claim 25, wherein the one or more parameters comprise one or more scaleparameters and / or one or more location parameters of the distribution.

27. The method of claim 26, wherein said quantising comprises performing location shiftedquantisation on the latent representation using the one or more parameters and the informationindicative of a dependence of one or more elements of the latent representation on one or moreother elements of the latent representation.

28. The method of any of claims 24 to 27, comprising entropy encoding the hyper latentrepresentation into the bitstream.

29. The method of claim 28, comprising:entropy decoding a first part of the bitstream to reconstruct the hyper latent representation;with the fifth neural network, decoding the hyper latent representation to produce the oneor more parameters and the information indicative of the dependence of one or more elementsof the latent representation on one or more other elements of the latent representation;entropy decoding a second part of the bitstream to reconstruct the quantised normalisedlatent representation using the one or more parameters; andperforming said reconstructing of the latent representation using the quantised normalisedlatent representation, the one or more parameters, and the information indicative of thedependence of one or more elements of the latent representation on one or more other elementsof the latent representation30. The method of any of claims 22 to 29, comprising:warping a representation of the second image with the approximation of optical flowinformation; andwith a sixth neural network, producing a latent representation of an image residualindicative of a difference between the first image and the second image;wherein producing the output image using the optical flow information comprisesdecoding the latent representation of the image residual with the third neural network.

31. The method of any of claim 30, comprising combining the warped representation of thesecond image with the latent representation of the image residual and decoding the combinedwarped representation and latent representation of the image residual with the third neuralnetwork.

32. The method of any of claims 22 to 31, wherein the information indicative of the dependenceof one or more elements of the latent representation on one or more other elements of thelatent representation is defined as a tensor comprising tensor elements that, when applied tothe elements of the latent representation modify said elements to be distributed according toan emulated autoregressive probability mass function.

33. The method of claim 32, when dependent on claim 24, wherein producing said informationindicative of the dependence of one or more elements of the latent representation on one ormore other elements of the latent representation using the hyper latent representation comprisespredicting the tensor elements with the fifth neural network.

34. A method for lossy image or video encoding and transmission,the method comprising thesteps of:receiving a first image and a second image at a first computer system;with a first neural network, producing a latent representation of optical flow informationusing the first image and the second image, the optical flow information being indicative of adifference between the first image and the second image;quantising the latent representation using information indicative of a dependence of oneor more elements of the latent representation on one or more other elements of the latentrepresentation to produce a quantised normalised latent representation;transmitting the quantised normalised latent representation to a second computer system.

35. A method for lossy image or video receipt and decoding, the method comprising the stepsof: receiving at a second computer system a quantised normalised latent representation, thequantised normalised latent representation produced from a latent representation of opticalflow information indiciative of a difference between first and second images and being pro-duced using information indicative of a dependence of one or more elements of the latentrepresentation on one or more other elements of the latent representation;reconstructing the latent representation from the quantised normalised latent representa-tion using the information indicative of the dependence of one or more elements of the latentrepresentation on one or more other elements of the latent representation;with a second neural network, decoding the latent representation of optical flow informa-tion to produce an approximation of the optical flow information; andwith a third neural network, producing an output image using the approximation ofoptical flow information, wherein the output image is an approximation of the first image.

36. A data processing apparatus configured to perform the method of any of claims 22 to 35.

37. A computer program comprising instructions which, when the program is executed by acomputer, cause the computer to carry out the method of any of claims 22 to 35.

38. A computer-readable storage medium comprising instructions which, when executed by acomputer, cause the computer to carry out the method of any of claims 22 to 35.

39. A method of training one or more neural networks, the one or more neural networks beingfor use in lossy image or video encoding, transmission and decoding, the method comprisingthe steps of:receiving a first input image at a first computer system;encoding the first input image using a first neural network to produce a latent representa-tion; decoding the latent representation using a second neural network to produce a first outputimage, wherein the first output image is an approximation of the first input image;evaluating a function based on a difference between the first output image and the firstinput image;updating the parameters of the first neural network and the second neural network basedon the evaluated function;repeating the above steps using a first set of input images to produce a first trained neuralnetwork and a second trained neural network; andadding noise to outputs of one or more of the layers of the one or more neural networksduring said training.

40. The method of claim 39, comprising adding noise to an output of one or more layers of thefirst neural network and / or the second neural network.

41. The method of claim 39 or 40, comprising adding noise to an output of one or moreactivation layers of the first neural network and / or the second neural network.

42. The method of any of claims 39 to 41, wherein the method comprises:encoding the latent representation using a third neural network to produce a first hyperlatent representation;decoding the first hyper latent representation using a fourth neural network to produceinformation associated with a distribution of the latent representation; andwith the information associated with the distribution of the latent representation, entropyencoding and entropy decoding the latent representation.

43. The method of claim 42, comprising adding noise to an output of one or more layers of thethird neural network and / or the fourth neural network.

44. The method of claim 43, comprising adding noise to an output of one or more activationlayers of the third neural network and / or the fourth neural network.

45. The method of claim 44, wherein the output of one or more activation layers of the fourthneural network comprises a standard deviation parameter associated with the distribution ofthe latent representation and a mean parameter associated with the distribution of the latentrepresentation, and wherein the noise is added to the mean parameter not the standard deviationparameter.

46. The method of any of claims 39 to 45 comprising:receiving a second input image at a first computer system;with a fifth neural network, producing a latent representation of optical flow informationusing the first input image and the second input image, the optical flow information beingindicative of a difference between the first input image and the second input image;with a sixth neural network, decoding the latent representation of optical flow informationto produce an approximation of the optical flow information; andusing the approximation of the optical flow information to produce a second outputimage, wherein the second output image is an approximation of the second input image.

47. The method of claim 46 comprising adding noise to an output of one or more layers of thefifth and / or the sixth neural network.

48. The method of claim 47, comprising adding noise to an output of one or more activationlayers of the fifth and / or the sixth neural network.

49. The method of any of claim 46 to 48, wherein the method comprises:encoding the latent representation of optical flow information using a seventh neuralnetwork to produce a second hyper latent representation;decoding the second hyper latent representation using an eighth neural network toproduce information associated with a distribution of the optical flow information; andwith the information associated with the distribution of the optical flow information,entropy encoding and entropy decoding the latent representation of optical flow information.

50. The method of claim 49, comprising adding noise to an output of one or more layers of theseventh neural network and / or the eighth neural network.

51. The method of claim 50, comprising adding noise to an output of one or more activationlayers of the seventh neural network and / or the eighth neural network.

52. The method of claim 51, wherein the output of one or more activation layers of the eighthneural network comprises a standard deviation parameter associated with the distribution of theoptical flow information and a mean parameter associated with the distribution of the opticalflow information, and wherein the noise is added to the mean parameter not the standarddeviation parameter.

53. The method of any of claims 39 to 52 comprising:receiving a second input image at a first computer system;with a ninth neural network, producing a latent representation of residual informationusing the first input image and the second input image, the residual information being indicativeof a difference between the first input image and the second input image;with an tenth neural network, decoding the latent representation of residual informationto produce a second output image, wherein the second output image is an approximation of thesecond input image.

54. The method of claim 53, comprising adding noise to an output of one or more layers of theninth and / or the tenth neural network.

55. The method of claim 54, comprising adding noise to an output of one or more activationlayers of the ninth and / or the tenth neural network.

56. The method of any of claims 53 to 55, wherein the method comprises:encoding the latent representation of residual information using an eleventh neuralnetwork to produce a third hyper latent representation;decoding the third hyper latent representation using a twelfth neural network to produceinformation associated with a distribution of the latent representation of residual information;and with the information associated with the distribution of the latent representation ofresidual information, entropy encoding and entropy decoding the latent representation ofresidual information.

57. The method of claim 56, comprising adding noise to an output of one or more layers of theeleventh neural network and / or the twelfth neural network.

58. The method of claim 57, comprising adding noise to an output of one or more activationlayers of the eleventh neural network and / or the twelfth neural network.

59. The method of claim 58, wherein the output of one or more activation layers of the twelfthneural network comprises a standard deviation parameter associated with the distribution ofthe latent representation of residual information and a mean parameter associated with thedistribution of the latent representation of residual information, and wherein the noise is addedto the mean parameter not the standard deviation parameter.

60. The method of any of claims 39 to 59, wherein the noise is defined by a noise functionparameterised by one or more variables, wherein the method comprises updating the oneor more variables based on the evaluated function, and wherein repeating the above stepscomprises repeating the updating of the one or more variables to produce a learned noisefunction during said training.

61. The method of claim 60, wherein the noise function is parameterised by one or morestandard deviation parameters and / or one or more mean parameters, and wherein repeating theabove steps comprises repeating the updating of the one or more standard deviation parametersand / or the one or more mean parameters.

62. The method of claim 60 or 61, wherein the noise added to an output of one or more layers ofone or more of the first, second, third, fourth, fifth, sixth, seventh, eighth, ninth, tenth, eleventhand / or twelfth neural networks is defined by a corresponding noise function of a pluralityof noise functions parameterised by respective one or more variables, wherein the methodcomprises updating the respective one or more variables based on the evaluated function, andwherein repeating the above steps comprises repeating the updating of the respective one ormore variables to produce a plurality of learned noise functions.

63. The method of claim 62, wherein at least one noise function of the plurality of noisefunctions is parameterised by one or more predetermined, fixed variables.

64. A method for lossy image or video encoding, transmission and decoding, the methodcomprising the steps of:receiving an input image at a first computer system;encoding the input image using a first trained neural network to produce a latentrepresentation; transmitting the latent representation to a second computer system; anddecoding the latent representation using a second trained neural network to produce anoutput image, wherein the output image is an approximation of the input image;wherein the first trained neural network and / or the second trained neural network aretrained according to any of the methods of claims 39 to 63.

65. A data processing system configured to perform the method of any one of claims 39 to 64.

66. A data processing apparatus configured to perform the method of claims 39 to 64.

67. A computer program comprising instructions which, when the program is executed by acomputer, cause the computer to carry out the method of claims 39 to 64.

68. A computer-readable storage medium comprising instructions which, when executed by acomputer, cause the computer to carry out the method of claims 39 to 64.

Images