Method and data processing system for lossy image or video encoding, transmission and decoding

Abstract

A method for lossy image or video encoding and transmission, and decoding, the method comprising the steps of: receiving a first image and a second image at a first computer system; with a first neural network, producing a latent representation of optical flow information using the first image and the second image, the optical flow information being indicative of a difference between the first image and the second image; transmitting the latent representation of optical flow information to a second computer system; with a second neural network, decoding the latent representation of optical flow information to produce an approximation of the optical flow information; scaling the optical flow information and with a third neural network, producing an output image using the scaled optical flow information, wherein the output image is an approximation of the first image.

Description

[0001] Method and data processing system for lossy image or videoencoding, transmission and decodingBACKGROUNDThis invention relates to a method and system for lossy image or video encoding, transmissionand decoding, a method, apparatus, computer program and computer readable storage mediumfor lossy image or video encoding and transmission, and a method, apparatus, computerprogram and computer readable storage medium for lossy image or video receipt and decoding.There is increasing demand from users of communications networks for images and videocontent. Demand is increasing not just for the number of images viewed, and for the playingtime of video; demand is also increasing for higher resolution content. This places increasingdemand on communications networks and increases their energy use because of the largeramount of data being transmitted.To reduce the impact of these issues, image and video content is compressed for transmissionacross the network. The compression of image and video content can be lossless or lossycompression. In lossless compression, the image or video is compressed such that all of theoriginal information in the content can be recovered on decompression. However, when usinglossless compression there is a limit to the reduction in data quantity that can be achieved. Inlossy compression, some information is lost from the image or video during the compressionprocess. Known compression techniques attempt to minimise the apparent loss of informationby the removal of information that results in changes to the decompressed image or video thatis not particularly noticeable to the human visual system. JPEG, JPEG2000, AVC, HEVC andAVI are examples of compression processes for image and / or video files.In general terms, known lossy image compression techniques use the spatial correlationsbetween pixels in images to remove redundant information during compression. For example,in an image of a blue sky, if a given pixel is blue, there is a high likelihood that the neighbouringpixels, and their neighbouring pixels, and so on, are also blue. There is accordingly no need toretain all the raw pixel data. Instead, we can retain only a subset of the pixels which take upfewer bits and infer the pixel values of the other pixels using information derived from spatialcorrelations.A similar approach is applied in known lossy video compression techniques. That is, spatialcorrelations between pixels allow the removal of redundant information during compression.However, in video compression, there is further information redundancy in the form of temporalcorrelations. For example, in a video of an aircraft flying across a blue-sky background, mostof the pixels of the blue sky do not change at all between frames of the video. The mostof the blue sky pixel data for the frame at position t = 0 in the video is identical to that atposition t = 10. Storing this identical, temporally correlated, information is inefficient. Instead,only the blue sky pixel data for a subset of the frames is stored and the rest are inferred frominformation derived from temporal correlations.In the realm of lossy video compression in particular, the removal of redundant temporallycorrelated information in a video sequence is known inter-frame redundancy.One technique using inter-frame redundancy that is widely used in standard video compressionalgorithms involves the categorization of video frames into three types: I-frames, P-frames, andB-frames. Each frame type carries distinct properties concerning their encoding and decodingprocess, playing different roles in achieving high compression ratios while maintainingacceptable visual quality.I-frames, or intra-coded frames, serve as the foundation of the video sequence. These framesare self-contained, each one encoding a complete image without reference to any other frame.In terms of compression, I-frames are least compressed among all frame types, thus carryingthe most data. However, their independence provides several benefits, including being thestarting point for decompression and enabling random access, crucial for functionalities likefast-forwarding or rewinding the video.P-frames, or predictive frames, utilize temporal redundancy in video sequences to achievegreater compression. Instead of encoding an entire image like an I-frame, a P-frame representsthe difference between itself and the closest preceding I- or P-frame. The process, known asmotion compensation, identifies and encodes only the changes that have occurred, therebysignificantly reducing the amount of data transmitted. Nonetheless, P-frames are dependent onprevious frames for decoding. Consequently, any error during the encoding or transmissionprocess may propagate to subsequent frames, impacting the overall video quality.B-frames, or bidirectionally predictive frames, represent the highest level of compression.Unlike P-frames, B-frames use both the preceding and following frames as references in theirencoding process. By predicting motion both forwards and backwards in time, B-framesencode only the differences that cannot be accurately anticipated from the previous and nextframes, leading to substantial data reduction. Although this bidirectional prediction makesB-frames more complex to generate and decode, it does not propagate decoding errors sincethey are not used as references for other frames. Artificial intelligence (AI) based compressiontechniques achieve compression and decompression of images and videos through the use oftrained neural networks in the compression and decompression process. Typically, duringtraining of the neutral networks, the difference between the original image and video and thecompressed and decompressed image and video is analyzed and the parameters of the neuralnetworks are modified to reduce this difference while minimizing the data required to transmitthe content. However, AI based compression methods may achieve poor compression resultsin terms of the appearance of the compressed image or video or the amount of informationrequired to be transmitted.An example of an AI based image compression process comprising a hyper-network is describedin Ballé, Johannes, et al. “Variational image compression with a scale hyperprior.” arXivpreprint arXiv:1802.01436 (2018), which is hereby incorporated by reference.An example of an AI based video compression approach is shown in Agustsson, E., Minnen, D.,Johnston, N., Balle, J., Hwang, S. J., and Toderici, G. (2020), Scale-space flow for end-to-endoptimized video compression. In Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition (pp. 8503-8512), which is hereby incorporated by reference.A further example of an AI based video compression approach is shown in Mentzer, F.,Agustsson, E., Ballé, J., Minnen, D., Johnston, N., and Toderici, G. (2022, November). Neuralvideo compression using gans for detail synthesis and propagation. In Computer Vision–ECCV2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, PartXXVI (pp. 562-578), which is hereby incorporated by reference.SUMMARYAccording to an aspect of the present disclosure, there is provided a method for lossy image orvideo encoding and transmission, and decoding. The method comprises the step of receivinga first image and a second image at a first computer system. In addition, the method furthercomprises the step of producing a latent representation of optical flow information using thefirst image and the second image, with a first neural network. The optical flow informationis indicative of a difference between the first image and the second image. The method alsocomprises the step of transmitting the latent representation of optical flow information to asecond computer system. Furthermore, the method comprises the step of decoding the latentrepresentation of optical flow information to produce an approximation of the optical flowinformation, with a second neural network. The method further comprises the step of scalingthe optical flow information. Moreover, the method comprises the step of producing an outputimage using the scaled optical flow information, with a third neural network. The output imageis an approximation of the first image.Optionally, the method as described above further comprises producing the output image witha fourth neural network. In this context, the fourth neural network is responsible for producingresidual information. The second image and the scaled optical flow information are utilizedby the fourth neural network in generating the residual information. Additionally, the fourthneural network decodes the residual information to produce the output image.Optionally, the method may involve scaling the optical flow information. As described above,the optical flow information comprises respective elements. The scaling of the optical flowinformation further comprises exponentially scaling these respective elements. The exponentialscaling is based on a value of each respective element.Optionally, the method may involve scaling the optical flow information. As described above,the optical flow information comprises respective elements. The scaling of the optical flowinformation is achieved by logarithmically scaling these respective elements. The value ofeach respective element is taken into consideration during this logarithmic scaling process.Optionally, the method may involve scaling the optical flow information. The scaling ofthe optical flow information may comprise scaling respective elements of the optical flowinformation. The scaling of the respective elements may be based on a value of the respectiveelements. The method may further comprise applying a fifth neural network to the optical flowinformation.Optionally, the method comprises quantising the scaled optical flow information. As describedabove, the scaled optical flow information is produced. Subsequently, the quantisation of thescaled optical flow information takes place. Following the quantisation process, the outputimage is produced.Optionally, the method as described above further comprises a step of quantisating. In thisstep, the respective elements of the scaled optical flow information are rounded into a pluralityof quantisation bins. The rounding is based on a value of the respective elements.According to an aspect of the present disclosure, there is provided a method for lossy imageor video receipt and decoding. The method comprises the step of decoding optical flowinformation at a first computer system. The decoding is performed using a first neuralnetwork. The optical flow information is indicative of a difference between a first image and asecond image. The method further comprises the step of scaling the optical flow information.Additionally, the method comprises the step of producing an output image using the scaledoptical flow information. The production of the output image is performed with a third neuralnetwork. The output image is an approximation of the first image.Optionally, there is provided a data processing apparatus. The data processing apparatus isconfigured to perform the method as described above.Optionally, there is provided a computer program. The computer program comprisesinstructions. When the computer program is executed by a computer, the instructions causethe computer to carry out the method as described above.Optionally, there is provided a computer-readable storage medium. The computer-readablestorage medium comprises instructions. When these instructions are executed by a computer,the computer carries out the method as described above.According to an aspect of the present disclosure, there is provided a method of training one ormore neural networks. The one or more neural networks are for use in lossy image or videoencoding, transmission, and decoding. The method comprises the steps of receiving a firstimage and a second image at a first computer system. The method further comprises, with a firstneural network, producing a latent representation of optical flow information. This is achievedusing the first image and the second image, wherein the optical flow information is indicative ofa difference between the first image and the second image. Additionally, the method comprises,with a second neural network, decoding the latent representation of optical flow information.This decoding process produces an approximation of the optical flow information. The methodalso comprises scaling the optical flow information. Furthermore, the method comprises, witha third neural network, producing an output image. This is achieved using the scaled opticalflow information, wherein the output image is an approximation of the first image. The methodfurther comprises evaluating a function. This evaluation is based on a difference between thefirst image and the output image. Moreover, the method comprises updating the parameters ofthe first, second, and / or third neural networks. This update is based on the evaluated function.Lastly, the method comprises repeating the above steps using a first set of input images. Thisrepetition produces first, second, and / or third trained neural networks.According to an aspect of the present disclosure, there is provided a method for lossy image orvideo encoding and transmission, and decoding. The method comprises the step of receivinga first image and a second image at a first computer system. In addition, the method furthercomprises the step of producing a latent representation of residual information using the firstimage and the second image, with a first neural network. The residual information is indicativeof a difference between the first image and the second image. The method also comprises thestep of transmitting the latent representation of residual information to a second computersystem. Furthermore, the method comprises the step of decoding the latent representation ofthe residual information to produce a representation of the first image, with a second neuralnetwork. Moreover, the method comprises the step of scaling the representation of the firstimage to produce an output image. The output image is an approximation of the first image.Optionally, the method as described above further comprises scaling the optical flow information.In this context, scaling the optical flow information involves clamping respective elements ofthe residual information between minimum and maximum values.Optionally, the method further comprises scaling the residual information. In this context, thescaling of the residual information involves exponentially scaling respective elements of theresidual information. The exponential scaling of the respective elements is based on a value ofthe respective elements.Optionally, the method further comprises scaling the residual information. In this context, thescaling of the residual information involves logarithmically scaling respective elements ofthe residual information. The basis for this logarithmic scaling is a value of the respectiveelements.Optionally, the method may involve scaling the optical flow information. The scaling of theoptical flow information may comprise scaling respective elements of the residual information.The scaling of the respective elements of the residual information may be based on a value ofthe respective elements. The method may further comprise applying a fourth neural networkto the residual information for the scaling of the respective elements.According to an aspect of the present disclosure, there is provided a method for lossy image orvideo receipt and decoding. The method comprises the step of decoding residual informationat a first computer system. The decoding is performed using a first neural network. The resultof this decoding process is the production of a representation of a first image. The residualinformation, which is decoded, is indicative of a difference between the first image and asecond image. The method further comprises the step of scaling the representation of thefirst image. The scaling is performed to produce an output image. The output image is anapproximation of the first image.Optionally, there is provided a data processing apparatus. The data processing apparatus isconfigured to perform the method as described above.Optionally, there is provided a computer program. The computer program comprisesinstructions. When the computer program is executed by a computer, the instructions causethe computer to carry out the method as described above.Optionally, there is provided a computer-readable storage medium. The computer-readablestorage medium comprises instructions. When these instructions are executed by a computer,they cause the computer to carry out the method as described above.According to an aspect of the present disclosure, there is provided a method of training one ormore neural networks. The one or more neural networks are for use in lossy image or videoencoding, transmission, and decoding. The method comprises the steps of receiving a firstimage and a second image at a first computer system. In the method, a first neural networkis used to produce a latent representation of residual information. This is achieved by usingthe first image and the second image. The residual information is indicative of a differencebetween the first image and the second image. The method also involves decoding the latentrepresentation of the residual information. This decoding is performed using a second neuralnetwork. The result of this decoding is a representation of the first image. Furthermore, themethod comprises scaling the representation of the first image. This scaling is performed toproduce an output image. The output image is an approximation of the first image. In themethod, a function is evaluated based on a difference between the first image and the outputimage. The parameters of the first and second neural networks are updated based on theevaluated function.Optionally, the method further comprises repeating the above steps using a first set of inputimages. This repetition is performed to produce first and second trained neural networks.According to an aspect of the present disclosure, there is provided a method for lossy image orvideo encoding, transmission and decoding, the method comprising the steps of: receiving aninput image at a first computer system; encoding the input image using a first trained neuralnetwork to produce a latent representation; transmitting the latent representation to a secondcomputer system; decoding the latent representation using a second trained neural network;scaling the decoded latent representation; and producing an output image using scaled decodedlatent representation, wherein the output image is an approximation of the input image.According to an aspect of the present disclosure, there is provided a method of training one ormore neural networks, the one or more neural networks being for use in lossy image or videoencoding, transmission and decoding, the method comprising the steps of:receiving a first image and a second image at a first computer system;with a first neural network, producing a latent representation of optical flow informationusing the first image and the second image, the optical flow information being indicative of adifference between the first image and the second image;with a second neural network, decoding the latent representation of optical flowinformation to produce an approximation of the optical flow information;with a third neural network, producing an output image using the optical flow information,wherein the output image is an approximation of the first image;evaluating a function based on a representation in first and second resolutions of adifference between the first image and the second image;updating the parameters of the first, second and / or third neural networks based on theevaluated function; andrepeating the above steps using a first set of input images to produce first, second and / orthird trained neural networks.Optionally, the representation of a difference in first and second resolutions between the firstimage and the second image comprises a difference between the first image and a warpedversion of the second image.Optionally, the first neural network comprises one or more downsampling layers configuredto produce the representation in first and second resolutions between the first image and thesecond image.Optionally, updating said parameters comprises updating the parameters of the downsamplinglayers.Optionally, the function is further based on a representation in a third resolution of a differencebetween the first image and the second image.Optionally, wherein the function is further based on a difference between the first image andthe output image.Optionally, the method comprises entropy encoding the latent representation of the opticalflow information into a bitstream, and wherein the function is further based on a bitrate of thebitstream.Optionally, the method comprises using the difference between the first and second images inthe first resolution to warp the first or second image in the second resolution, and using thewarped first or second image in the second resolution to estimate the difference between thefirst and second images in the second resolution.Optionally, the difference comprises a mean squared error.Optionally, the representations in the first and second resolutions each comprise a tensor, andwherein the method comprises estimating said difference by estimating a mean squared errorbetween said tensors.According to an aspect, there is provided a method for lossy image or video encoding,transmission and decoding, the method comprising the steps of:receiving an input image at a first computer system;encoding the input image using a first trained neural network to produce a latentrepresentation; transmitting the latent representation to a second computer system;decoding the latent representation using a second trained neural network to produce anoutput image, wherein the output image is an approximation of the input image;wherein the first neural network and the second neural network are trained according toany of the above methods.According to an aspect, there is provided a method for lossy image or video encoding andtransmission, the method comprising the steps of:receiving an input image at a first computer system;encoding the input image using a first trained neural network to produce a latentrepresentation; andtransmitting the latent representation;wherein the first neural network is trained according to any of the above methods.According to an aspect, there is provided a method for lossy image or video receipt anddecoding, the method comprising the steps of:receiving a latent representation transmitted by a first computer system at a secondcomputer system, the latent representation corresponding to an input image; anddecoding the latent representation using a second trained neural network to produce anoutput image, wherein the output image is an approximation of the input image;wherein the second neural network is trained according to any of the above methods.According to an aspect of the present disclosure, there is provided a data processing apparatusconfigured to perform the method of any of the above methods.According to an aspect of the present disclosure, there is provided a computer programcomprising instructions which, when the program is executed by a computer, cause thecomputer to carry out any of the above methods.According to an aspect of the present disclosure, there is provided a computer-readable storagemedium comprising instructions which, when executed by a computer, cause the computer tocarry out any of the above methods.According to an aspect of the present disclosure, there is provided a method of training one ormore neural networks, the one or more neural networks being for use in lossy image or videoencoding, transmission and decoding, the method comprising the steps of:receiving a first image at a first computer system;with a first neural network, producing a latent representation of the first image;with a second neural network, decoding the latent representation of the first image toproduce an output image;wherein at least one of producing the latent representation or decoding the latentrepresentation comprises an up- or down-sampling step;evaluating a function based on an output of the up- or down-sampling step;updating the parameters of the first and / or second neural networks based on the evaluatedfunction; andrepeating the above steps using a first set of input images to produce first and / or secondtrained neural networks.Optionally, the up- or down-sampling step is performed by one or more layers of the first neuralnetwork.Optionally, the up- or down-sampling step is performed by one or more layers of the secondneural network.Optionally, updating said parameters comprises updating the parameters of the up- or down-sampling layers.Optionally, the function is further based on a difference between the first image and the outputimage.Optionally, the method comprises entropy encoding the latent representation into a bitstream,and wherein the function is further based on a bitrate of the bitstream.According to an aspect, there is provided a method for lossy image or video encoding,transmission and decoding, the method comprising the steps of:receiving an input image at a first computer system;encoding the input image using a first trained neural network to produce a latentrepresentation; transmitting the latent representation to a second computer system;decoding the latent representation using a second trained neural network to produce anoutput image, wherein the output image is an approximation of the input image;wherein the first neural network and the second neural network are trained according toany of the above methods.According to an aspect, there is provided a method for lossy image or video encoding andtransmission, the method comprising the steps of:receiving an input image at a first computer system;encoding the input image using a first trained neural network to produce a latentrepresentation; andtransmitting the latent representation;wherein the first neural network is trained according to any of the above methods.According to an aspect, there is provided a method for lossy image or video receipt anddecoding, the method comprising the steps of:receiving a latent representation transmitted by a first computer system at a secondcomputer system, the latent representation corresponding to an input image; anddecoding the latent representation using a second trained neural network to produce anoutput image, wherein the output image is an approximation of the input image;wherein the second neural network is trained according to any of the above methods.According to an aspect of the present disclosure, there is provided a data processing apparatusconfigured to perform the method of any of the above methods.According to an aspect of the present disclosure, there is provided a computer programcomprising instructions which, when the program is executed by a computer, cause thecomputer to carry out any of the above methods.According to an aspect of the present disclosure, there is provided a computer-readable storagemedium comprising instructions which, when executed by a computer, cause the computer tocarry out any of the above methods.BRIEF DESCRIPTION OF THE DRAWINGSAspects of the invention will now be described by way of examples, with reference to thefollowing figures in which:Figure 1 illustrates an example of an image or video compression, transmission and decom-pression pipeline.Figure 2 illustrates a further example of an image or video compression, transmission anddecompression pipeline including a hyper-network.Figure 3 illustrates an example of a video compression, transmission and decompressionpipeline.Figure 4 illustrates an example of a video compression, transmission and decompressionsystem.Figure 5 illustrates an example of a video compression, transmission and decompressionpipeline.Figure 6 illustrates an example of a video compression, transmission and decompressionpipeline.Figure 7a illustrates an example of an output of an image or video compression, transmissionand decompression pipeline.Figure 7b illustrates an example of an output of an image or video compression, transmissionand decompression pipeline.Figure 8 illustrates an example of a video compression, transmission and decompressionpipeline.Figure 9 illustrates an example of a flow module.Figure 10a illustrates examples of two flow compression modules.Figure 10b illustrates an example of an image or video compression and decompressionpipeline.Figure 11 illustrates an example of an image or video compression and decompression pipeline.Figure 12 illustrates an example of an image or video compression and decompression pipeline.DETAILED DESCRIPTION OF THE DRAWINGSCompression processes may be applied to any form of information to reduce the amountof data, or file size, required to store that information. Image and video information is anexample of information that may be compressed. The file size required to store the information,particularly during a compression process when referring to the compressed file, may bereferred to as the rate. In general, compression can be lossless or lossy. In both forms ofcompression, the file size is reduced. However, in lossless compression, no information is lostwhen the information is compressed and subsequently decompressed. This means that theoriginal file storing the information is fully reconstructed during the decompression process.In contrast to this, in lossy compression information may be lost in the compression anddecompression process and the reconstructed file may differ from the original file. Image andvideo files containing image and video data are common targets for compression.In a compression process involving an image, the input image may be represented as ^^. Thedata representing the image may be stored in a tensor of dimensions ^^ × ^^ × ^^, where ^^represents the height of the image, ^^ represents the width of the image and ^^ represents thenumber of channels of the image. Each ^^ × ^^ data point of the image represents a pixel valueof the image at the corresponding location. Each channel ^^ of the image represents a differentcomponent of the image for each pixel which are combined when the image file is displayed bya device. For example, an image file may have 3 channels with the channels representing thered, green and blue component of the image respectively. In this case, the image informationis stored in the RGB colour space, which may also be referred to as a model or a format.Other examples of colour spaces or formats include the CMKY and the YCbCr colour models.However, the channels of an image file are not limited to storing colour information and otherinformation may be represented in the channels. As a video may be considered a series ofimages in sequence, any compression process that may be applied to an image may also beapplied to a video. Each image making up a video may be referred to as a frame of the video.The output image may differ from the input image and may be represented by ^^. The differencebetween the input image and the output image may be referred to as distortion or a differencein image quality. The distortion can be measured using any distortion function which receivesthe input image and the output image and provides an output which represents the differencebetween input image and the output image in a numerical way. An example of such a methodis using the mean square error (MSE) between the pixels of the input image and the outputimage, but there are many other ways of measuring distortion, as will be known to the personskilled in the art. The distortion function may comprise a trained neural network.Typically, the rate and distortion of a lossy compression process are related. An increase inthe rate may result in a decrease in the distortion, and a decrease in the rate may result in anincrease in the distortion. Changes to the distortion may affect the rate in a correspondingmanner. A relation between these quantities for a given compression technique may be definedby a rate-distortion equation.AI based compression processes may involve the use of neural networks. A neural network isan operation that can be performed on an input to produce an output. A neural network maybe made up of a plurality of layers. The first layer of the network receives the input. One ormore operations may be performed on the input by the layer to produce an output of the firstlayer. The output of the first layer is then passed to the next layer of the network which mayperform one or more operations in a similar way. The output of the final layer is the output ofthe neural network.Each layer of the neural network may be divided into nodes. Each node may receive at leastpart of the input from the previous layer and provide an output to one or more nodes in asubsequent layer. Each node of a layer may perform the one or more operations of the layer onat least part of the input to the layer. For example, a node may receive an input from one ormore nodes of the previous layer. The one or more operations may include a convolution, aweight, a bias and an activation function. Convolution operations are used in convolutionalneural networks. When a convolution operation is present, the convolution may be performedacross the entire input to a layer. Alternatively, the convolution may be performed on at leastpart of the input to the layer.Each of the one or more operations is defined by one or more parameters that are associatedwith each operation. For example, the weight operation may be defined by a weight matrixdefining the weight to be applied to each input from each node in the previous layer to eachnode in the present layer. In this example, each of the values in the weight matrix is a parameterof the neural network. The convolution may be defined by a convolution matrix, also knownas a kernel. In this example, one or more of the values in the convolution matrix may be aparameter of the neural network. The activation function may also be defined by values whichmay be parameters of the neural network. The parameters of the network may be varied duringtraining of the network.Other features of the neural network may be predetermined and therefore not varied duringtraining of the network. For example, the number of layers of the network, the number ofnodes of the network, the one or more operations performed in each layer and the connectionsbetween the layers may be predetermined and therefore fixed before the training process takesplace. These features that are predetermined may be referred to as the hyperparameters of thenetwork. These features are sometimes referred to as the architecture of the network.To train the neural network, a training set of inputs may be used for which the expected output,sometimes referred to as the ground truth, is known. The initial parameters of the neuralnetwork are randomized and the first training input is provided to the network. The output ofthe network is compared to the expected output, and based on a difference between the outputand the expected output the parameters of the network are varied such that the differencebetween the output of the network and the expected output is reduced. This process is thenrepeated for a plurality of training inputs to train the network. The difference between theoutput of the network and the expected output may be defined by a loss function. The result ofthe loss function may be calculated using the difference between the output of the networkand the expected output to determine the gradient of the loss function. Back-propagation ofthe gradient descent of the loss function may be used to update the parameters of the neuralnetwork using the gradients ^^^^ / ^^^^ of the loss function. A plurality of neural networks in asystem may be trained simultaneously through back-propagation of the gradient of the lossfunction to each network.In the context of image or video compression, this type of system, where simultaneous trainingwith back-propagation through each element or the whole network architecture may be referredto as end-to-end, learned image or video compression. Unlike in traditional compressionalgorithms that use primarily handcrafted, manually constructed steps, an end-to-end learnedsystem learns itself during training what combination of parameters best achieves the goal ofminimising the loss function. This approach is advantageous compared to systems that are notend-to-end learned because an end-to-end system has a greater flexibility to learn weights andparameters that might be counter-intuitive to someone handcrafting features.It will be appreciated that the term "training" or "learning" as used herein means the processof optimizing an artificial intelligence or machine learning model, based on a given set of data.This involves iteratively adjusting the parameters of the model to minimize the discrepancybetween the model’s predictions and the actual data, represented by the above-describedrate-distortion loss function.The training process may comprise multiple epochs. An epoch refers to one complete passof the entire training dataset through the machine learning algorithm. During an epoch, themodel’s parameters are updated in an effort to minimize the loss function. It is envisaged thatmultiple epochs may be used to train a model, with the exact number depending on variousfactors including the complexity of the model and the diversity of the training data.Within each epoch, the training data may be divided into smaller subsets known as batches.The size of a batch, referred to as the batch size, may influence the training process. A smallerbatch size can lead to more frequent updates to the model’s parameters, potentially leading tofaster convergence to the optimal solution, but at the cost of increased computational resources.Conversely, a larger batch size involves fewer updates, which can be more computationallyefficient but might converge slower or even fail to converge to the optimal solution.The learnable parameters are updated by a specified amount each time, determined by thelearning rate. The learning rate is a hyperparameter that decides how much the parametersare adjusted during the training process. A smaller learning rate implies smaller steps in theparameter space and a potentially more accurate solution, but it may require more epochs toreach that solution. On the other hand, a larger learning rate can expedite the training processbut may risk overshooting the optimal solution or causing the training process to diverge.The training described herein may involve use of a validation set, which is a portion of thedata not used in the initial training, which is used to evaluate the model’s performance and toprevent overfitting. Overfitting occurs when a model learns the training data too well, to thepoint that it fails to generalize to unseen data. Regularization techniques, such as dropout orL1 / L2 regularization, can also be used to mitigate overfitting.It will be appreciated that training a machine learning model is an iterative process thatmay comprise selection and tuning of various parameters and hyperparameters. As will beappreciated, the specific details, such as hyper parameters and so on, of the training processmay vary and it is envisaged that producing a trained model in this way may achieved in anumber of different ways with different epochs, batch sizes, learning rates, regularisations,and so on, the details of which are not essential to enabling the advantages and effects of thepresent disclosure, except where stated otherwise. The point at which an “untrained” neuralnetwork is considered be “trained” is envisaged to be case specific and depend on, for example,on a number of epochs, a plateauing of any further learning, or some other metric and is notconsidered to be essential in achieving the advantages described herein.More details of an end-to-end, learned compression process will now be described. It will beappreciated that in some cases, end-to-end, learned compression processes may be combinedwith one or more components that are handcrafted or trained separately.In the case of AI based image or video compression, the loss function may be defined by therate distortion equation. The rate distortion equation may be represented by ^^^^^^^^ = ^^ + ^^ ∗ ^^,where ^^ is the distortion function, ^^ is a weighting factor, and ^^ is the rate loss. ^^ may bereferred to as a lagrange multiplier. The langrange multiplier provides as weight for a particularterm of the loss function in relation to each other term and can be used to control which termsof the loss function are favoured when training the network.In the case of AI based image or video compression, a training set of input images maybe used. An example training set of input images is the KODAK image set (for exampleat www.cs.albany.edu / xypan / research / snr / Kodak.html). An example training set of inputimages is the IMAX image set. An example training set of input images is the Imagenetdataset (for example at www.image-net.org / download). An example training set of inputimages is the CLIC Training Dataset P (“professional”) and M (“mobile”) (for example athttp: / / challenge.compression.cc / tasks / ).An example of an AI based compression, transmission and decompression process 100 isshown in Figure 1. As a first step in the AI based compression process, an input image 5 isprovided. The input image 5 is provided to a trained neural network 110 characterized by afunction ^^^^ acting as an encoder. The encoder neural network 110 produces an output basedon the input image. This output is referred to as a latent representation of the input image 5. Ina second step, the latent representation is quantised in a quantisation process 140 characterisedby the operation ^^, resulting in a quantized latent. The quantisation process transforms thecontinuous latent representation into a discrete quantized latent. An example of a quantizationprocess is a rounding function.In a third step, the quantized latent is entropy encoded in an entropy encoding process 150 toproduce a bitstream 130. The entropy encoding process may be for example, range or arithmeticencoding. In a fourth step, the bitstream 130 may be transmitted across a communicationnetwork.In a fifth step, the bitstream is entropy decoded in an entropy decoding process 160. Thequantized latent is provided to another trained neural network 120 characterized by a function^^^^ acting as a decoder, which decodes the quantized latent. The trained neural network 120produces an output based on the quantized latent. The output may be the output image of theAI based compression process 100. The encoder-decoder system may be referred to as anautoencoder.Entropy encoding processes such as range or arithmetic encoding are typically able to losslesslycompress given input data up to close to the fundamental entropy limit of that data, as determinedby the total entropy of the distribution of that data. Accordingly, one way in which end-to-end,learned compression can minimise the rate loss term of the rate-distortion loss function andthereby increase compression effectiveness is to learn autoencoder parameter values thatproduce low entropy latent representation distributions. Producing latent representationsdistributed with as low an entropy as possible allows entropy encoding to compress the latentdistributions as close to or to the fundamental entropy limit for that distribution. The lowerthe entropy of the distribution, the more entropy encoding can losslessly compress it and thelower the amount of data in the corresponding bitstream. In some cases where the latentrepresentation is distributed according to a gaussian or Laplacian distribution, this learningmay comprise learning optimal location and scale parameters of the gaussian or Laplaciandistributions, in other cases, it allows the learning of more flexible latent representationdistributions which can further help to achieve the minimising of the rate-distortion lossfunction in ways that are not intuitive or possible to do with handcrafted features. Examples ofthese and other advantages are described in WO2021 / 220008A1, which is incorporated in itsentirety by reference.Something which is closely linked to the entropy encoding of the latent distribution and whichaccordingly also has an effect on the effectiveness of compression of end-to-end learnedapproaches is the quantisation step. During inference, a rounding function may be used toquantise a latent representation distribution into bins of given sizes, a rounding function isnot differentiable everywhere. Rather, a rounding function is effectively one or more stepfunctions whose gradient is either zero (at the top of the steps) or infinity (at the boundarybetween steps). Back propagating a gradient of a loss function through a rounding functionis challenging. Instead, during training, quantisation by rounding function is replaced byone or more other approaches. For example, the functions of a noise quantisation model aredifferentiable everywhere and accordingly do allow backpropagation of the gradient of theloss function through the quantisation parts of the end-to-end, learned system. Alternatively, astraight-through estimator (STE) quantisation model or one other quantisation models may beused. It is also envisaged that different quantisation models may be used for during evaluationof different term of the loss function. For example, noise quantisation may used to evaluate therate or entropy loss term of the rate-distortion loss function while STE quantisation may beused to evaluate the distortion term.In a similar manner to how learning parameters top produce certain distributions of the latentrepresentation facilitates achieving better rate loss term minimisation, end-to-end learning ofthe quantisation process achieves a similar effect. That is, learnable quantisation parametersprovide the architecture with a further degree of freedom to achieve the goal of minimising theloss function. For example, parameters corresponding to quantisation bin sizes may be learnedwhich is likely to result in an improved rate-distortion loss outcome compared to approachesusing hand-crafted quantisation bin sizes.Further, as the rate-distortion loss function constantly has to balance a rate loss term against adistortion loss term, it has been found that the more degrees of freedom the system has duringtraining, the better the architecture is at achieving optimal rate and distortion trade off.The system described above may be distributed across multiple locations and / or devices. Forexample, the encoder 110 may be located on a device such as a laptop computer, desktopcomputer, smart phone or server. The decoder 120 may be located on a separate device whichmay be referred to as a recipient device. The system used to encode, transmit and decode theinput image 5 to obtain the output image 6 may be referred to as a compression pipeline.The AI based compression process may further comprise a hyper-network 105 for thetransmission of meta-information that improves the compression process. The hyper-network105 comprises a trained neural network 115 acting as a hyper-encoder ^^ ℎ^^and a trained neuralnetwork 125 acting as a hyper-decoder ^^ℎ^^. An example of such a is shown in Figure 2.Components of the system not further discussed may be assumed to be the same as discussedabove. The neural network 115 acting as a hyper-decoder receives the latent that is the output ofthe encoder 110. The hyper-encoder 115 produces an output based on the latent representationthat may be referred to as a hyper-latent representation. The hyper-latent is then quantizedin a quantization process 145 characterised by ^^ℎ to produce a quantized hyper-latent. Thequantization process 145 characterised by ^^ℎ may be the same as the quantisation process 140characterised by ^^ discussed above.In a similar manner as discussed above for the quantized latent, the quantized hyper-latent isthen entropy encoded in an entropy encoding process 155 to produce a bitstream 135. Thebitstream 135 may be entropy decoded in an entropy decoding process 165 to retrieve thequantized hyper-latent. The quantized hyper-latent is then used as an input to trained neuralnetwork 125 acting as a hyper-decoder. However, in contrast to the compression pipeline 100,the output of the hyper-decoder may not be an approximation of the input to the hyper-decoder115. Instead, the output of the hyper-decoder is used to provide parameters for use in theentropy encoding process 150 and entropy decoding process 160 in the main compressionprocess 100. For example, the output of the hyper-decoder 125 can include one or more ofthe mean, standard deviation, variance or any other parameter used to describe a probabilitymodel for the entropy encoding process 150 and entropy decoding process 160 of the latentrepresentation. In the example shown in Figure 2, only a single entropy decoding process 165and hyper-decoder 125 is shown for simplicity. However, in practice, as the decompressionprocess usually takes place on a separate device, duplicates of these processes will be presenton the device used for encoding to provide the parameters to be used in the entropy encodingprocess 150.Further transformations may be applied to at least one of the latent and the hyper-latent at anystage in the AI based compression process 100. For example, at least one of the latent and thehyper latent may be converted to a residual value before the entropy encoding process 150,155is performed. The residual value may be determined by subtracting the mean value of thedistribution of latents or hyper-latents from each latent or hyper latent. The residual valuesmay also be normalised.To perform training of the AI based compression process described above, a training set ofinput images may be used as described above. During the training process, the parameters ofboth the encoder 110 and the decoder 120 may be simultaneously updated in each trainingstep. If a hyper-network 105 is also present, the parameters of both the hyper-encoder 115and the hyper-decoder 125 may additionally be simultaneously updated in each training step.The training process may further include a generative adversarial network (GAN). Whenapplied to an AI based compression process, in addition to the compression pipeline describedabove, an additional neutral network acting as a discriminator is included in the system. Thediscriminator receives an input and outputs a score based on the input providing an indicationof whether the discriminator considers the input to be ground truth or fake. For example, theindicator may be a score, with a high score associated with a ground truth input and a lowscore associated with a fake input. For training of a discriminator, a loss function is used thatmaximizes the difference in the output indication between an input ground truth and input fake.When a GAN is incorporated into the training of the compression process, the output image 6may be provided to the discriminator. The output of the discriminator may then be used in theloss function of the compression process as a measure of the distortion of the compressionprocess. Alternatively, the discriminator may receive both the input image 5 and the outputimage 6 and the difference in output indication may then be used in the loss function of thecompression process as a measure of the distortion of the compression process. Training ofthe neural network acting as a discriminator and the other neutral networks in the compressionprocess may be performed simultaneously. During use of the trained compression pipelinefor the compression and transmission of images or video, the discriminator neural network isremoved from the system and the output of the compression pipeline is the output image 6.Incorporation of a GAN into the training process may cause the decoder 120 to performhallucination. Hallucination is the process of adding information in the output image 6 thatwas not present in the input image 5. In an example, hallucination may add fine detail tothe output image 6 that was not present in the input image 5 or received by the decoder 120.The hallucination performed may be based on information in the quantized latent received bydecoder 120.Details of a video compression process will now be described. As discussed above, a video ismade up of a series of images arranged in sequential order. AI based compression process100 described above may be applied multiple times to perform compression, transmissionand decompression of a video. For example, each frame of the video may be compressed,transmitted and decompressed individually. The received frames may then be grouped toobtain the original video.The frames in a video may be labelled based on the information from other frames that is usedto decode the frame in a video compression, transmission and decompression process. Asdescribed above, frames which are decoded using no information from other frames may bereferred to as I-frames. Frames which are decoded using information from past frames may bereferred to as P-frames. Frames which are decoded using information from past frames andfuture frames may be referred to as B-frames. Frames may not be encoded and / or decoded inthe order that they appear in the video. For example, a frame at a later time step in the videomay be decoded before a frame at an earlier time.The images represented by each frame of a video may be related. For example, a number offrames in a video may show the same scene. In this case, a number of different parts of thescene may be shown in more than one of the frames. For example, objects or people in a scenemay be shown in more than one of the frames. The background of the scene may also beshown in more than one of the frames. If an object or the perspective is in motion in the video,the position of the object or background in one frame may change relative to the position ofthe object or background in another frame. The transformation of a part of the image froma first position in a first frame to a second position in a second frame may be referred to asflow, warping or motion compensation. The flow may be represented by a vector. One or moreflows that represent the transformation of at least part of one frame to another frame may bereferred to as a flow map.An example AI based video compression, transmission, and decompression process 200 isshown in Figure 3. The process 200 shown in Figure 3 is divided into an I-frame part 201for decompressing I-frames, and a P-frame part 202 for decompressing P-frames. It will beunderstood that these divisions into different parts are arbitrary and the process 200 may bealso be considered as a single, end-to-end pipeline.As described above, I-frames do not rely on information from other frames so the I-frame part201 corresponds to the compression, transmission, and decompression process illustrated inFigures 1 or 2. The specific details will not be repeated here but, in summary, an input image^^0 is passed into an encoder neural network 203 producing a latent representation which isquantised and entropy encoded into a bitstream 204. The subscript 0 in ^^0 indicates the inputimage corresponds to a frame of a video stream at position t = 0. This may be the first frame ofan entire video stream or the first frame of a chunk of a video stream made up of, for example,an I-frame and a plurality of subsequent P-frames and / or B-frames. The bitstream 204 is thenentropy decoded and passed into a decoder neural network 205 to reproduce a reconstructedimage ^^0 which in this case is an I-frame. The decoding step may be performed both locallyat the same location as where the input image compression occurs as well as at the locationwhere the decompression occurs. This allows the reconstructed image ^^0 to be available forlater use by components of both the encoding and decoding sides of the pipeline.In contrast to I-frames, P-frames (and B-frames) do rely on information from other frames.Accordingly, the P-frame part 202 at the encoding side of the pipeline takes as input not onlythe input image ^^^^ that is to be compressed (corresponding to a frame of a video stream atposition t), but also one or more previously reconstructed images ^^^^−1 from an earlier framet-1. As described above, the previously reconstructed ^^^^−1 is available at both the encodeand decode side of the pipeline and can accordingly be used for various purposes at both theencode and decode sides.At the encode side, previously reconstructed images may be used for generating a flow mapscontaining information indicative of inter-frame movement of pixels between frames. In theexample of Figure 3, both the image being compressed ^^^^ and the previously reconstructedimage from an earlier frame ^^^^−1 are passed into a flow module part 206 of the pipeline. Theflow module part 206 comprises an autoencoder such as that of the autoencoder systems ofFigures 1 and 2 but where the encoder neural network 207 has been trained to produce alatent representation of a flow map from inputs ^^^^−1 and ^^^^ , which is indicative of inter-framemovement of pixels or pixel groups between ^^^^−1 and ^^^^ . The latent representation of the flowmap is quantised and entropy encoded to compress it and then transmitted as a bitstream 208.On the decode side, the bitstream is entropy decoded and passed to a decoder neural network209 to produce a reconstructed flow map ^^ .The reconstructed flow map ^^ is applied to the previously reconstructed image ^^^^−1 to generatea warped image ^^^^−1,^^. It is envisaged that any suitable warping technique may be used, forexample bi-linear or tri-linear warping, as is described in Agustsson, E., Minnen, D., Johnston,N., Balle, J., Hwang, S. J., and Toderici, G. (2020), Scale-space flow for end-to-end optimizedvideo compression. In Proceedings of the IEEE / CVF Conference on Computer Vision andPattern Recognition (pp. 8503-8512), which is hereby incorporated by reference. It is furtherenvisaged that a scale-space flow approach as described in the above paper may also optionallybe used. The warped image ^^^^−1,^^ is a prediction of how the previously reconstructed image^^^^−1 might have changed between frame positions t-1 and t, based on the output flow mapproduced by the flow module part 206 autoencoder system from the inputs of ^^^^ and ^^^^−1.As with the I-frame, the reconstructed flow map ^^ and corresponding warped image ^^^^−1,^^may be produced both on the encode side and the decode side of the pipeline so they areavailable for use by other components of the pipeline on both the encode and decode sides.In the example of Figure 3, both the image being compressed ^^^^ and the ^^^^−1,^^ are passedinto a residual module part 210 of the pipeline. The residual module part 210 comprises anautoencoder system such as that of the autoencoder systems of Figures 1 and 2 but where theencoder neural network 211 has been trained to produce a latent representation of a residualmap indicative of differences between the input mage ^^^^ and the warped image ^^^^−1,^^. Thelatent representation of the residual map is then quantised and entropy encoded into a bitstream212 and transmitted. The bitstream 212 is then entropy decoded and passed into a decoderneural network 213 which reconstructs a residual map ^^ from the decoded latent representation.Alternatively, a residual map may first be pre-calculated between ^^^^ and the ^^^^−1,^^ and thepre-calculated residual map may be passed into an autoencoder for compression only. Thishand-crafted residual map approach is computationally simpler, but reduces the degrees offreedom with which the architecture may learn weights and parameters to achieve its goalduring training of minimising the rate-distortion loss function.Finally, on the decode side, the residual map ^^ is applied (e.g. combined by addition, subtractionor a different operation) to the warped image to produce a reconstructed image ^^^^ which is areconstruction of image ^^^^ and accordingly corresponds to a P-frame at position t in a sequenceof frames of a video stream. It will be appreciated that the reconstructed image ^^^^ can then beused to process the next frame. That is, it can be used to compress, transmit and decompress^^^^+1, and so on until an entire video stream or chunk of a video stream has been processed.Thus, for a block of video frames comprising an I-frame and ^^ subsequent P-frames, thebitstream may contain (i) a quantised, entropy encoded latent representation of the I-frameimage, and (ii) a quantised, entropy encoded latent representation of a flow map and residualmap of each P-frame image. For completeness, whilst not illustrated in Figure 3, any of theautoencoder systems of Figure 3 may comprise hyper and hyper-hyper networks such as thosedescribed in connection with Figure 2. Accordingly, the bitstream may also contain hyper andhyper-hyper parameters, their latent quantised, entropy encoded latent representations and soon, of those networks as applicable.Finally, the above approach may generally also be extended to B-frames, for example as isdescribed in Pourreza, R., and Cohen, T. (2021). Extending neural p-frame codecs for b-framecoding. In Proceedings of the IEEE / CVF International Conference on Computer Vision (pp.6680-6689).The above-described flow and residual based approach is highly effective at reducing theamount of data that needs to be transmitted because, as long as at least one reconstructed frame(e.g. I-frame ^^^^−1) is available, the encode side only needs to compress and transmit a flowmap and a residual map (and any hyper or hyper-hyper parameter information, as applicable)to reconstruct a subsequent frame.Figure 4 shows an example of an AI image or video compression process such as that describedabove in connection with Figures 1-3 implemented in a video streaming system 400. Thesystem 400 comprises a first device 401 and a second device 402. The first and seconddevices 401, 402 may be user devices such as smartphones, tablets, AR / VR headsets or otherportable devices. In contrast to known systems which primarily perform inference on GPUssuch as Nvidia A100, Geforce 3090, Gefore 4090 GPU cards, the system 400 of Figure 4performs inference on a CPU of the first and second devices respectively. That is, computefor performing both encoding and decoding are performed by the respective CPUs of the firstand second devices 401, 402. This places very different power usage, memory and runtimeconstraints on the implementation of the above methods than when implementing AI-basedcompression methods on GPUs. In one example, the CPU of first and second devices 401, 402may comprise a Qualcomm Snapdragon CPU.The first device 401 comprises a media capture device 403, such as a camera, arranged tocapture a plurality of images, referred to hereafter as a video stream 404, of a scene 404. Thevideo stream 404 is passed to a pre-processing module 406 which splits the video stream intoblocks of frames, various frames of which will be designated as I-frames, P-frames, and / orB-frames. The blocks of frames are then compressed by an AI-compression module 407comprising the encode side of the AI-based video compression pipeline of Figure 3. Theoutput of the AI-compression module is accordingly a bitstream 408a which is transmittedfrom the first device 401, for example via a communications channel, for example over oneor more of a WiFi, 3G, 4G or 5G channel, which may comprise internet or cloud-based 409communications.The second device 402 receives the communicated bitstream 408b which is passed to anAI-decompression module 410 comprising the decode side of the AI-based video compressionpipeline of Figure 3. The output of the AI-decompression module 402 is the reconstructedI-frames, P-frames and / or B-frames which are passed to a post-processing module 411 wherethey can prepared, for example passed into a buffer, in preparation for streaming 412 to andrendering on a display device 413 of the second device 402.It is envisaged that the system 400 of Figure 4 may be used for live video streaming at 30fps ofa 1080p video stream, which means a cumulative latency of both the encode and decode sideis below substantially 50ms, for example substantially 30ms or less. Achieving this level ofruntime performance with only CPU compute on user devices presents challenges which arenot addressed by known methods and systems or in the wider AI-compression literature.For example, execution of different parts of the compression pipeline during inferencemay be optimized by adjusting the order in which operations are performed using one ormore known CPU scheduling methods. Efficient scheduling can allow for operations to beperformed in parallel, thereby reducing the total execution time. It is also envisaged thatefficient management of memory resources may be implemented, including optimising cachingmethods such as storing frequently-accessed data in faster memory locations, and memoryreuse, which minimizes memory allocation and deallocation operations.A number of concepts related to the AI compression processes and / or their implementationin a hardware system discussed above will now be described. Although each concept isdescribed separately, one or more of the concepts described below may be applied in an AIbased compression process as described above.Concept 1: Flow ScalingAs described above, one of the challenges of implementing AI-based image and videocompression is reducing run time to allow a compression pipeline to run in real time or nearreal time. One approach that can substantially speed up run time is the quantisation of theneural network weights and biases, as well as the outputs of the neural networks and / or theindividual layers thereof. That is, a compression pipeline operating in fp16 will generallybe slower than one operating in int8, int4, and so on. The process of converting from oneprecision to another is known as quantisation which rounds a number into a given bin. Asdescribed in the above general sections, the quantisation parameters may be learned.However, effectively quantising models and / or outputs presents its own challenges in AI-basedimage and video compression given that each neural network behaves differently, producesoutputs with different min and max values, and so on. The consequence of this is that knownquantisation approaches typically result in substantial drops in accuracy when comparinge.g. a fp16 implementation against an int8 implementation. This in turn makes it difficult toimplement a fully AI-based image and video compression primarily in an int8 framework.One particularly challenging area for quantisation is optical flow.Consider the illustrative flow-residual compression pipeline shown in Figure 5, which showsflow encoder / decoder networks and residual encoder / decoder networks of an AI basedcompression pipeline. This may be, for example, similar to the AI based compression pipelineof the type shown in Figure 3.The flow encoder neural network takes a current image ^^^^ and a previous image ^^^^−1, encodesthese into a flow latent representation which is optionally quantised and entropy encoded andtransmitted as a bit stream. On the decode side, the bitstream is received, entropy decoded intothe flow latent representation that the flow decoder neural network uses as input to producea representation of flow ^^ , e.g. a flow map. The flow map ^^ is applied to a previouslydecoded image ^^^^−1 to generate a warped version of that previously decoded image ^^^^−1,^^ . Thewarped version of the previously decoded image ^^^^−1,^^ is then fed into the residual encodernetwork, together with the current image ^^^^ to produce a residual latent representation which isoptionally quantised and entropy encoded and transmitted as a bit stream. On the decode side,the bitstream is received and entropy decoded back into the residual latent representation ^^ andused by the residual decoder neural network, in combination with information from the warpedversion of the previously decoded image ^^^^−1,^^ , to produce the reconstructed image ^^^^ . In thecase of Figure 5, the information associated with the warped previously decoded image ^^^^−1,^^is optionally first processed by module ^^ , referred to herein after as a composition adapter, forexample to downsample and / or pad it before it is fed into the residual decoder together withthe entropy decoded residual latent representation to produce the final reconstructed image^^^^ . This process may then be repeated for ^^^^+1 and so on to encode, transmit, and decode asequence of frames.The inventors have realised that the output of the flow decoder 209, typically a tensor comprisinga plurality of values, has values with a very different overall ranges than outputs of the otherdecoders in the pipeline such as the i-frame decoder 205 and the residual decoder. For example,in some cases, the highest values of the tensor produced by the flow decoder 209 are higherthan values of the outputs of the other decoders, and / or the lowest values are lowest values ofthe outputs of the other decoders. Whilst this is not a difficulty when operating in fp16 space,it causes a substantial change in model behaviour when the values within the large range arequantised into the limited number of bins of e.g. int8 space or int4 space. For example inthe case of int8 quantisation, the full range of the flow decoder 209 output values must bequantised into only 256 bins. If the original output range is large, the quantisation into a smallnumber of bins results in a substantial loss in granularity. That is, if there are a small numberof very high values in the output tensor, and many small values, then in order to accommodatethe large range up to the very high values, the bin sizes will necessarily be large, and the manysmall values will end up in the same bin and the granularity and differences between thesesmall values will be lost after the quantisation process.In order to overcome this problem, the present inventors have introduced a flow scaling modulethat scales the output of the flow decoder 209 before quantisation takes place.Figure 6 illustratively shows the flow-residual compression pipeline of Figure 5 but with theflow scaling module 600 of the present disclosure.The flow scaling module 600 receives as input the output ^^ of the flow decoder 209, and thenproduces ^^^^^^^^^^^^^^ by performing an exponential or log scaling on it, for example:( )^^^^^^^^^^^^^^ = sign( ^^ ) ·2| ^^ |− 1 In more detail, in this example, the absolute value of each element in the tensor ^^ is computed.Negative values in ^^ become positive, and positive values (including zero) remain unchanged.This is then scaled by raising 2 to the power of each element. Essentially it’s calculating 2| ^^^^ |for each element ^^^^ in the tensor ^^ . Next, from each element, 1 is subtracted before the resultis multiplied by the original sign of that element in order to restore the original sign of eachelement to the transformed value.The overall effect of this expression is to exponentially amplify the magnitude of each elementin ^^ (base 2 exponential), while preserving their original signs (positive or negative). Theoptional subtraction of 1.0 ensures a zero value is presented.The consequence of scaling the flow tensor is that it can now be quantised without losingthe granulirity of the original floating point values. By using exponential (or log) scaling,the effect is to scale different elements of the range more or less depending on their values.This allows smaller (or larger) elements to be scaled more (or less, as applicable) so that thesubsequent quantisation bins the values into a wider range of bins across the 256 (or othernumber) quantisation bins. Taking the example of a flow tensor with many small values and afew large values, the small values will be scaled more so are "spread out" more evenly into thesubsequent quantisation bins, whereas the large values will be scaled less so will still only takeup a small number of bins at the upper end of the range.The above example is of course illustrative only, and other exponential or log scaling, or indeedother scaling type, approaches are also envisaged that can be used to spread the elementsthe flow tensor out more evenly across quantisation bins. For example, a logarithmic scalingapproach is also envisaged:(^ ^ )^^^^^^^^^^^^^^ = sign( ^^ ) · log ^ ^^^ + ^^ Here, it will be appreciated that the scaling function may be implemented whereby sign( ^^^^)^resents the sign of each element ^^^^ in the tensor ^^ , the term ^ ^rep ^^^^^ denotes the absolute valueof a given element ^^^^, the term ^^ is a small positive constant added to ensure the argument ofthe logarithm is always positive. This expression will handle zero and negative values in thetensor ^^ by using the absolute value and ensuring that the logarithm function always receives apositive argument, thereby allowing a log scaling to be applied to the elements of the flowtensor.In a further illustrative example, the above-described scaling may also be generalised to anyscaling function by introducing one or more additional neural network layers after the flowdecoder whose weights may be learned. These layers are are described hereinafter as a scalingnetwork and may be implemented as a separate module to the flow decoder or integratedwith it. The scaling network may approximate any function and accordingly may be trainedto approximate the above described exponential or logarithmic scaling functions, or indeedtrained to approximate another scaling function. The use of a scaling network provides agreater degree of flexibility to tailor the scaling function to a given use case and may, forexample, scale the flows associated with different types of input image pairs (e.g. imagesfrom static scenes, moving scenes, 3D animations, films, video games, and so on) to differentextents as determined by the learned weights. This scaling network approach may be indicatedas follows:^^^^^^^^^^^^^^ = ^^( ^^ ) where ^^ is a neural network comprising one or more layers that approximate a scaling function.The advantage of a scaling neural network is that it can produce improved overall performanceof the compression pipeline in terms of image reconstruction accuracy and lower bit rates butthis comes at a trade off of run time.Finally, it is envisaged that the scaling may be implemented in a more naive way by clampingthe outputs of the flow between minimum and maximum values. This typically results in someloss of accuracy but may be implemented efficiently and accordingly results in very fast runtimes. For example:min if ^^ < min into int8 results in the values being clamped betweenthe min and max values of defined by the borders of the highest and lowest quantisation bins,in which case clamping may have only negligible effect compared to the other methods ofscaling and may not be the optimum solution.Concept 2: Residual ScalingThe above described approach may also be used to scale the outputs of one or more othernetworks in the compression pipeline where a range of values of the output tensor of thosenetworks results in a loss of granularity after any subsequent quantisation. For example,consider the output tensor of the residual decoder. In AI based flow-residual compression,there exist a number of compression artefact types that occur that do not exist in traditionalcompression pipelines. One such artefact appears in image sequences where there is little tono movement between frames and manifests itself in the reconstructed images as halo-likerings around edges and objects in the images. These artefacts typically arise where small,barely noticeable changes in pixel values between frames such as tiny lighting changes, blockartefacts from traditional compression (if the input to the AI compression pipeline came froma traditionally compressed source), noise, and so on become amplified over time by the AIbased compression pipeline. This can, in some architectures, result in parts the networks ofthe compression pipeline treating image noise as real movement and then exaggerating theseas the sequence of frames of the video progress. For example, in some architectures, theseartefacts may be caused by the operation of the flow module(s), in other architectures thismay be caused by the operation of the residual module(s), in again other architectures thismay be caused by both modules or indeed other parts of the architectures, and so on. At theend of the sequence of frames, the artefacts can be quite severe and detrimental to the imagereconstruction quality. This type of artefact is referred to hereinafter as "unrolling".Unrolling artefacts occur across mostly all types of image sequences but are particularlyapparent when AI based flow-residual compression models are used to encode and decodeimage sequences of internet video calls and AR / VR scenes where a foreground user who movesvery little is surrounded by a largely static background which also does not move much), screensharing image sequences, security camera feeds and others. If there is no movement after aperiod of time, objects and edges start to exhibit halo-like ringing around their edges. Theartefacts also result in the compression pipeline assigning a disproportionate number of bits tohandle the hallucinated movement of the unrolling artefacts even though the correspondingparts of the actual images aren’t changing or are hardly changing between frames so ought notto have many bits assigned to them.Figures 7a and 7b respectively show comparative examples 700a, 700b of an output image ofan AI based compression pipeline with a flow-residual component. In the first example 700a,it can be seen that the image exhibits an unrolling type artefact 701 on a static backgroundobject. That is, the edges of the object exhibit halo-like ringing. In contrast, in the secondexample 500b which is generated using the solutions described below, the unrolling artefactsare mitigated.The inventors have realised that one way to mitigate unrolling artefacts is by scaling the valuesof the output tensor (i.e. the final output image) produced by the residual decoder. This maybe for example by scaling the extreme values towards the upper and / or lower end of the rangevalues of the tensor more than those further from the extremes of the range. This effectivelyresults in the tensor values being clamped in a range of values that prevent the outliers beingfed into the next frame reconstruction, thereby mitigating the noise amplification or blow upsthat can result in unrolling artefacts.As above in concept 1, the scaling may be exponential, logarithmic, or generalised by theapplication of one or more neural network layers, together a scaling network ^^, that approximatesome scaling function that may be learned. It will be appreciated that scaling by expotential orlogarithmic funciton may be implemented efficiently and without substantial run time increases.In contrast, a scaling network ^^ may produce better generalisability at the cost of slower runtimes. Finally, it is also envisaged that the scaling may be implemented with a direct clamp orthresholding, whereby any values of the tensor over and / or below the threshold value(s) may beclamped down to those threshold values. This may be implemented for example by applying ahardtanh function or by using any pre-existing clamp functions from machine learning librariessuch as the torch.clamp function in Pytorch.Figure 8 illustratively shows the flow-residual compression pipeline of Figure 5 but with theresidual scaling module 800 of the present disclosure.The residual scaling module 800 receives as input the output of the residual decoder 213,for example in its raw form, or in some post-processed form. It will be appreciated that thisoutput may accordingly correspond to the raw reconstructed image tensor ^^^^,^^^^^^, or somepost-processing thereof. From the input, the residual scaling module 800 produces a scaledreconstructed image tensor ^^^^,^^^^^^^^^^^^ . As has been described above, this may be by performingan exponential or log scaling on it in a similar manner to concept 1, or some generalisedfunction approximated by a scaling network, or indeed by clamping the values of a tensorbetween a min and / or max. For example, in the case of using clamping to scale the values thismay be:min if ^^ < min help to mitigate unrolling artefacts at all levels ofprecision (e.g. fp16, int8, and so on) by preventing extreme values of the output reconstructedimage from propagating across the sequence of frames being encoded and decoded, but it alsoensures that any reduction in granularity that results from any subsequent quantisation processis mitigated in the same way as in concept 1.Finally, it is more generally envisaged that any output of any of the decoders in the AIcompression may be scaled using any of the techniques, for example prior to subsequentquantisation, to ensure as much of the granularity of a floating point implementation ispreserved during quantisation. That is, the flow decoder and residual decoder examples aregiven above, but it is also envisaged that the I-frame decoder 205 from e.g. Figures 3, 6 and / or8 may also have a scaling module. It is also envisaged that should a pipeline be provided withone or more other decoders, one or more of their outputs may also be scaled.Concept 3: Multi-resolution flow lossFigure 9 illustrates an example of a flow module, in this case a network 900, configured toestimate information indicative of a difference between an image ^^^^−1 and an image ^^^^ , e.g.flow information. Figure 9 is provided as an example of how such flow information may becalculated between two input or output images.The network 900 comprises a set of layers 901a, 901b respectively for an image ^^^^−1 and animage ^^^^ from respective times or positions in a sequence ^^ − 1 and ^^ of a sequence of imageframes. The set of layers 901a, 901b may define one or more convolution operations and / ornonlinear activations in the layers to sequentially downsample the input images to producea pyramid of feature maps for different levels of coarseness or spatial resolution. This maycomprise performing ℎ / 2 ^^ / 2 downsampling in a first layer, ℎ / 4 ^^ / 4 downsampling in asecond layer ℎ / 8 ^^ / 8 downsampling in a third layer, ℎ / 16 ^^ / 16 downsampling in a fourthlayer, ℎ / 32 ^^ / 32 downsampling in a fifth layer, ℎ / 64 ^^ / 64 downsampling in a sixth layer,and so on. It will of course be appreciated that these downsampling operations and levels ofcoarseness or spatial resolution of a pyramid feature map are exemplary only and others levelsare also envisaged.With the downsampling operations performed and the corresponding pyramid of feature mapsgenerated, a first cost volume 902 is calculated at the most course level between the pixelsof the first image ^^^^−1 and the corresponding pixels of in the second image ^^^^ . Cost volumesdefine the matchmaking cost of matching the pixels in the initial image with the pixels in thelater image. That is, the closeness of each pixel, or a subset of all pixels, in the initial imageto one or more pixels in the later image is determined with a measure of closeness such as avector or dot product, a cosine similarity, a mean absolute difference, or some other measureof closeness. This metric may be calculated against all pixels in the later image, or only forpixels in a predetermined search radius such as a 1-10 pixel radius (preferably a 1, 2, 3, or 4pixel radius), or some other radius as described in connection with concept 4 below, aroundthe pixel coordinate corresponding to the pixel against which the measure of closeness is beingcalculated.Finally, a first flow 903 can be estimated from the first cost volume 902. This may be achievedusing, for example a flow extractor network which may comprise a convolutional neuralnetwork comprising a plurality of layers trained to output a tensor defining a flow map fromthe input cost volumes. Other methods of calculating flow information from cost volumes willalso be known to the skilled person.The same process is then repeated for the other levels of coarseness to calculate a second costvolume 904 and second flow 905, and so on for the cost volumes and flows associated witheach of the levels of coarseness have been calculated, up to the final cost volume 906 and flow907.The weights and / or biases of any activation layers in network 900 (e.g. optionally in thedownsampling convolution layers and / or in a flow extractor network that produces flow mapsfrom the cost volumes) are trainable parameters and can accordingly be updated during trainingeither alone, or in an end to end manner with the rest of the compression pipeline. The trainablenature of these parameters provides the network 900 with flexibility to produce feature maps ateach level of spatial resolution (i.e. pyramid feature maps) and / or at the flow outputs that areforced into a distribution that best allows the network to meet its training objective (e.g. bettercompression, better reconstruction accuracy, more accurate reconstruction of flow, and so on).For example, it allows the network 900 to produce feature maps that, when cost volumes and / orflow are calculated therefrom, produce cost volumes or flows that are distributed roughlymatching the latent representation distribution that would previously have been expected to beoutput by a dedicated flow encoder module. This effectively allows a dedicated flow encoderto be omitted entirely from the flow compression part of the pipeline, as is shown in theillustrative example of Figure 10a, descibed later.Optionally, for each level of coarseness or resolution, the flow of the previous level or levels ofcoarseness or resolution may be used to warp 908, 909, the feature maps before the cost volumeis calculated. This has the effect of artificially reducing the amount of relative movementbetween the pixels of the t and t - 1 images or feature maps when calculating the cost volumes,reducing flow errors for high movement details.As the warping process uses inputs from different levels of coarseness or spatial resolution, theflow estimation output may be upsampled 910, 911 first to match the coarseness resolution ofthe feature map to which the flow is being applied in the warping process.The outputs of the flow module may accordingly be one or more cost volumes or somerepresentation thereof, and / or one or more flows or some representation thereof).Note that whilst the term cost volume has been used above, the information indicative ofdifferences between the respective inputs need not be a strict cost volume in the mathematicalsense, but may be any representation of this information. For example a compressivelycalculated cost volume, applying for example the principles of compressive sensing to estimatean approximate cost volume by sampling only a small number of pixel differences comparedto performing a complete pixel-wise cost volume calculation. This compressively calculatedcost volume approach may be applied to all embodiments described herein.As described above, it will be appreciated that running a flow-based compression pipeline oncetraining has been completed relies on estimation of flow, whether by handcrafted algorithmor through some trained network. The output estimated flow itself may be compressed andtransmitted in a bitstream, which may be performed by a dedicated flow encoder networkthat encodes the flow information into a latent representation distributed according to adistribution that can be efficiently entropy encoded. Irrespective of the flow estimationapproach taken, dedicated flow encoders increase run time and partly contribute to preventinglearned compression codecs from running in real time or near real time.Consider for example the pipeline of Figure 3. At the encode side, a previously reconstructedimage and a new, to-be-encoded image may be used for generating flow maps containinginformation indicative of inter-frame movement of pixels between frames. In the exampleof Figure 3, both the current image being compressed ^^^^ and the previously reconstructedimage from an earlier frame ^^^^−1 are passed into a flow module 207 that typically has twoparts: a flow estimation part, and a dedicated encoder part that encodes the estimated flowsinto a latent representation that can be efficiently entropy encoded. The dedicated flow encodernetwork produces latent representations of flow that are distributed as close to an optimallyentropy encodable distribution as possible. In Figure 3, these two parts are shown as a singlecomponent.The flow estimation part may comprise, for example, the flow module 900 of Figure 9. That is,the flow module 900 produces its cost volumes and flows, then passes one or more of theseinto the second part: the dedicated flow encoder network that is trained to produce the latentrepresentation of flow that can be efficiently entropy encoded before being sent in a bitstreamto the decoder. Note that using a dedicated flow encoder network may be slow because first thecost volumes and flows are calculated before they can be encoded into a latent representation,which itself is a slow process, before the entropy encoding of the latent representation of thecost volume(s) and / or flow(s) into a bitstream can be performed which is finally transmitted.Using a dedicated flow encoded accordingly increases run time.Instead, in order reduce compute and runtime overhead, the present disclosure alternativelyenvisages omitting the dedicated flow encoder. For example, the outputs of the flow module 900output(s) are directly entropy encoded and fed into the bitstream without first encoding theminto a latent representation. Given that the process of encoding flow module output(s) into alatent representation with a flow compression encoder is computationally expensive, removingthis component entirely from the flow compression module 207 results in a significant decreasein runtime, thereby contributing to the goal of being able to run the pipeline in inference inreal time or near real time.This is, in part, made possible by virtue of the trainable nature of the flow module 900, wherebythe weights and biases of one or more of the activation layers of the flow module 900 result incost volumes or flows that are already distributed according to a distribution that correspondsroughly to that which a dedicated flow encoder may produce. This is illustrated in more detailin Figure 10a.Figure 10a illustratively shows two flow compression modules 1000a and 1000b which maybe used as flow module 207 in Figure 3. The same reference numerals are used for like-components. In the first module 1000a, two input images ^^^^ and ^^^^−1 are passed 1001a, 1001bare input into a flow module such as flow module 900 of Figure 9, which produces pyramidfeature maps 1002 of different levels of coarseness, and corresponding cost volumes and / orflows 1003. The final flow estimation is then passed to a dedicated flow compression encoder1004 which encodes it to produce a latent representation of the final flow estimation. Theoutput may thus be a latent representation of one or more of a H W, H / 2 W / 2, H / 4 W / 4, H / 8W / 8, H / 16 W16. H / 32 W / 32, H / 64 W / 64, or some other resolution, cost volumes and / or flows.This is finally entropy encoded into a bitstream 1005, and transmitted. The bitstream 1005 isentropy decoded and the decoded bitstream is passed to a decoder 1006 which reconstructs thecost volumes and / or flows which may be used in a flow-based approach as described above inthe general concepts section.However, as described above, the approach of the flow compression module 1000a with adedicated flow compression encoder 1004 is slow and computationally expensive.In contrast, the present inventors have realized that omitting the dedicated flow compressionencoder 1004 entirely and instead directly entropy encoding one or more outputs of theflow module 900 into the bitstream 1005 without first passing it through the dedicated flowencoder 1004 results in a substantial speed up at run time. Counter-intuitively, this removalof the dedicated flow compression encoder 1004 does not appreciably appear to effect theperformance of the rest of the compression pipeline both in terms of distortion and bit rate.This modified approach is illustrated with flow compression module 1000b in Figure 10a whereit is apparent that the flow encoder 1004 of flow compression module 1000b has effectivelybeen chopped out. Thus, instead of encoding the cost volumes and / or flows into a latentrepresentation before the entropy encoding step, the one or more of a H W, H / 2 W / 2, H / 4 W / 4,H / 8 W / 8, H / 16 W16. H / 32 W / 32, H / 64 W / 64, or some other resolution, cost volumes and / orflows are simply entropy encoded directly and sent out in the bitstream.As explained above, this approach is based on the insight that flow modules such as flowmodule 900 have trainable parameters and accordingly have a great deal of flexibility in termsof the distribution of outputs they can be trained to produce. The inventors have found thatwhen the dedicated flow compression encoder 1004 is included, the flow module 900 has noneed to produce outputs in a distribution that can be efficiently entropy encoded because theneural networks that make up the compression pipeline as a whole are simply able to rely on thededicated flow encoder 1004 to minimize any contribution to bitrate that the flow informationhas in the bitstream. However, when the dedicated flow compression encoder 1004 is removed,networks of the compression pipeline are no longer able to rely on the dedicated flow encoder1004. In its place, the inventors have found that the flow module 900 learns during trainingto compensate by outputting cost volumes and / or flows that are similarly distributed as thosethat would be output by a dedicated flow compression encoder 1004. This can be understoodas the flow module 900 (that is, the trainable networks within it such as the flow extractornetwork and / or activation layers in the convolution layers) effectively being forced to mimic thededicated flow compression encoder 1004 during training when the loss is being minimizedbecause the system can no longer rely on the (now removed) dedicated flow compressionencoder 1004 to perform that task.It is envisaged that the training of the flow module 900 may either be performed in an end-to-endmanner together with the rest of the compression pipeline or alternatively in a student-teacherapproach where the network of the flow module 900 is the student component, and a knownoptical flow model and pre-trained flow compression network is the teacher component.Additionally or alternatively, the training of the flow module 900 may be performed separatelyusing data on which the groundtruth flow is known. For example, by using 3D animation videodata whose groundtruth flow is known a priori through the animation program used to generateit, or using auto flow generation methods.This counter-intuitive removal of the dedicated flow compression encoder 1004 from the flowcompression module to force the other components to effectively take on the tasks previouslyperformed by the dedicated flow compression encoder 1004 contributes significantly to aspeed up at run time speed on the encoding side of the compression pipeline. This approachaccordingly makes a substantial stride forward towards the goal of achieving real time or nearreal time performance during inference.Figure 10b illustrates the introduction of the flow compression modules 1000b into a flow-residual compression pipeline, such as that of Figure 3, by illustratively showing the encodingand decoding of p-frames of an image stream, whereby a groundtruth image and its previouslyencoded and decoded reconstruction at t-1 are available. This corresponds generally to theflow-residual compression pipeline shown in Figure 3 and accordingly uses the same referencenumbers for corresponding features. However in this case, the output of the flow module 1002is natively in the distribution of a latent representation and can be immediately transmitted tothe decoder, without needing a standalone dedicated flow compression encoder. In addition, itis envisaged that the warped reconstructed, previously decoded image from t-1 may be passeddirectly to the residual decoder, in addition to whatever is output by the residual encoder 211.This provides the residual decoder with the additional context of the warped reconstructed,previously decoded image.Returning now to Figure 9, we can consider different training approaches that can be used toimprove the overall performance of the flow module 900 compared to naive training usinga known rate-distortion loss function. That is, in known flow-based approaches, the lossfunction is typically represented by ^^^^^^^^ = ^^ + ^^ ∗ ^^, where ^^ is the distortion function, ^^ isa weighting factor, and ^^ is the rate loss. ^^ may be referred to as a lagrange multiplier. Thelangrange multiplier provides a weight for a particular term of the loss function in relation toeach other term and can be used to control which terms of the loss function are favoured whentraining the network. This known, naive loss function does not explicitly take into account theindividual effects of the different components of the flow module of Figure 9 to overall loss.Specifically, during training with the above naive loss function, the updating of the trainableparameters of the layers of the different resolution pyramids that produce the above-describedflow pyramid features only effects the overall loss very indirectly through the final rate termand / or distortion term. The present inventors have found that this means the final values ofthe learned network parameters in the flow network are often sub-optimal in the context of anAI-based compression pipeline. They converge to values that produce a flow that optimallyreconstructs features associated with correct, real-world movement rules and concepts (e.g.rigid body physics in motion between frames) and so on. That is, when the final flow (e.g. aflow map) output by the flow decoder 1006 is analysed it is a reasonably good estimation ofreal-world movement rules and concepts. This is perhaps unsurprising when the distortionpart of the loss function is based solely on pixel differences between two frames in which(real-world) movement is occurring. For example, if a solid ball is thrown by a figure in frame1, then real-world solid-body physics and gravity indicate it should continue its trajectory andalso start to drop in subsequent frames. The flow map of such a frame sequence produced byflow module 1006 trained with a naive rate-distortion loss function would show this movementaccurately.However, counter-intuitively, in AI-based compression pipelines, we don’t actually care aboutthe specific form of the flow information or that that accurately reproduces information in away that models real-world movement rules and behaviours. Indeed constraining the flowinformation in a way that encourages real-world estimation of movement rules results in asub-optimal compression pipeline because such a representation of flow information may notbe optimal for helping the residual decoder 213 to produce the most accurate possible imagefor a given bit rate. That is, there exist other sets of learned parameters (weights, biases, etc.)that produce representations of flow information that, in the feature maps, look very little likeactual flow or "real world" movement, but nevertheless turn out to be more optimal for theresidual encoder to not only reconstruct frames with low distortion, but also good (indeedbetter) rate performance.Accordingly, the present inventors have realised that we actually want to encourage the flowmodule 1000b to learn weights that produce some representation of information indicative of adifference between two frames that optimally assists the residual decoder 213 in producing ^^^^and this may be a very different representation of flow than that which is used in traditionalimage and video compression or that which is learned by a flow module trained using a naiverate-distortion loss function.The question accordingly arises: how can we encourage the flow module 1000b to learn suchalternative representations of flow?The present disclosure solves this question by introducing additional terms into the loss functionbased on differences between output pyramid features of a first frame at time ^^, and warpedpyramid output features of a second frame at time ^^ − 1 associated with that pyramid level.This is illustrated in Figure 11.Figure 11 illustrates an example flow module such as that of Figure 9 but showing where,during training, variables are extracted and used to calculate additional loss terms in the lossfunction.Specifically, Figure 11 illustrates an example of a flow module, in this case a network 1100,configured to estimate information indicative of a difference between an image ^^^^−1 and animage ^^^^ , e.g. flow information. Figure 11 is provided as an example of how such flowinformation may be calculated between two input or output images. The flow module of Figure11a may be used in the place of, for example, the flow encoder 207 of Figure 3.The network 1100 comprises a set of layers 1101a, 1101b respectively for an image ^^^^−1 andan image ^^^^ from respective times or positions in a sequence ^^ − 1 and ^^ of a sequence of imageframes. The set of layers 1101a, 1101b may define one or more convolution operations and / ornonlinear activations in the layers to sequentially downsample the input images to producea pyramid of feature maps for different levels of coarseness or spatial resolution. This maycomprise performing ℎ / 2 ^^ / 2 downsampling in a first layer, ℎ / 4 ^^ / 4 downsampling in asecond layer ℎ / 8 ^^ / 8 downsampling in a third layer, ℎ / 16 ^^ / 16 downsampling in a fourthlayer, ℎ / 32 ^^ / 32 downsampling in a fifth layer, ℎ / 64 ^^ / 64 downsampling in a sixth layer,and so on. It will of course be appreciated that these downsampling operations and levels ofcoarseness or spatial resolution of a pyramid feature map are exemplary only and others levelsare also envisaged.With the downsampling operations performed and the corresponding pyramid of feature mapsgenerated, a first cost volume or some other pixel difference 1102, such as a compressivelycalculated cost volume, or mean absolute difference, or some other difference, is calculated atthe most course level between the pixels of the first image ^^^^−1 and the corresponding pixels ofin the second image ^^^^ . For completeness, this difference defines the matchmaking cost ofmatching the pixels in the initial image with the pixels in the later image. That is, the closenessof each pixel, or a subset of all pixels, in the initial image to one or more pixels in the laterimage is determined with a measure of closeness such as a vector or dot product, a cosinesimilarity, a mean absolute difference, or some other measure of closeness. This metric maybe calculated against all pixels in the later image, or only for pixels in a predetermined searchradius such as a 1-10 pixel radius (preferably a 1, 2, 3, or 4 pixel radius), or some other radiusaround the pixel coordinate corresponding to the pixel against which the measure of closenessis being calculated. Or, in the case of a compressively calculated cost volume, based on asampling of a subset of pixels around in a pixel radius to reconstruct an approximation of acost volume. The difference measure can be used to estimate a flow or some representation ofit that is not flow but that contains information that can be used for warping at each level of thepyramid in a more effective way than flow.During training of the flow module of Figure 11, the following additional loss term, referred toherein as a pyramid loss, is introduced:∑^^^^=1 ^^(^^^^,^^^^^^^^ , ^^(^^^^−1,^^^^^^^^ , ^^^^)) where ^^^^−1,^^^^^^^^ is the tensor output by the pyramid feature layer ^^ receiving ^^^^−1 as input, ^^^^,^^^^^^^^is the tensor output by the pyramid feature layer ^^ receiving ^^^^−1 as input, ^^ is a differencefunction such as a mean squared error, or some other difference function, ^^(·, ·) is a warpingoperation, and ^^^^ is the representation of flow at the pyramid feature layer ^^. Note that ^^ is apyramid layer index indicative of the resolution pairs being considered, e.g. ℎ / 2 ^^ / 2,ℎ / 4 ^^ / 4, and so on, up to ^^ layers.Including the pyramid loss in the known naive rate-distortion function we get:^^^^^^^^ = ^^ + ^^ ∗ ^^ + ∑^^^^=1 ^^(^^^^,^^^^^^^^ , ^^(^^^^−1,^^^^^^^^ , ^^^^)) By way of illustration, the variables from which the pyramid loss may be calculated in theexample of Figure 11 are indicated by box 1112 at the end the dashed line. Whilst only onesuch loss term is shown in Figure 11, it is envisaged that the number of such loss terms maycorrespond to the number of pyramid layers, taking into account that at the final layer (i.e.smallest layer) of the pyramid where there is no warp to perform there will be no pyramid lossfor that layer.At a high level, this additional loss term estimates the difference between between pyramidfeatures and warped pyramid features for one or more pyramid levels. The effect of introducingthe pyramid loss term is that the loss now depends explicitly on the tensors output by eachof the pyramid layers ^^ and a warped version thereof. This explicit dependence provides thenetworks with a stronger training signal during training to push the pyramid layer weights toconverge on values that produce output features that are optimal for warping at each pyramidlayer ^^. In turn, the final output representation of the information corresponding to flow isoptimal for warping a previously decoded frame ^^^^−1 in some way which in turn is optimal forthe residual encoder / decoder to finally produce an accurate reconstruction of an image ^^^^ , evenif these are quite different to what might "real world" flow might be expected to be.In other words, the pyramid layers learn to help the residual network to operate optimally evenif the overall flow information they produce is not a typical flow map but is nevertheless arepresentation of movement information between frames that is optimised for warping. Thiscontrasts to the naive rate-distortion loss function where the individual pyramid layers learnsome overall set of weights based on overall rate and / or distortion loss so there is very little orno training signal to encourage the individual layers to learn optimal weights that are distinctfrom traditional flows but which improve residual encoder / decoder performance.The inventors have found that the present approach can facilitate a reduction in final trainingloss of the AI-compression pipeline by up to substantially 5%. Whilst the additional loss term(s)can increase training time and reduce training stability both due to the additional computerequired and by virtue of a more complex loss landscape in which the global optimum isbeing sought, these effects are not prohibitive can be addressed using usual training techniquesknown to the skilled person.Below is illustrative pseudocode of a training loop for an AI-based compression pipeline usingthe above-described pyramid loss:

[0002] Algorithm 1 AI-based compression network training with custom loss functionInputs: Training dataset ^^, learning rate ^^, regularization parameter ^^, number of epochs ^^ ,network architecture ^^^^ including ^^ pyramid layersInitialize network parameters ^^for ^^^^^^^^ℎ = 1 to ^^ dofor each batch (^^^^−1, ^^^^ ) in ^^ doForward pass to compute predictions ^^^^ = ^^^^ (^^^^−1, ^^^^ )∑Compute loss ^^^^^^^^ = ^^ + ^^^^ + ^^^^=1 ^^(^^^^ , ^^^^^^^^ , ^^(^^^^−1, ^^^^^^^^ , ^^^^))Backward pass to compute gradients ∇^^ ^^^^^^^^Update parameters with optimizer O: ^^ ← O(^^, ∇^^ ^^^^^^^^, ^^)end forOptionally evaluate on validation setend forThat is, a training data set ^^ is provided. A learning rate ^^, regularisation parameter ^^, and anumber of training steps or epochs ^^ is selected. The network architecture of ^^^^ is defined, forexample as shown in Figures, 3, 5, 6a, 6b, and 7. The network parameters ^^ are randomlyinitialised and then the training loop is started. For each batch (^^^^−1, ^^^^) in the training data ^^,the forward pass is computed and the ^^^^^^^^ is calculated from a distortion term ^^, a rate term ^^,and a pyramid loss term∑^^^^=1 ^^(^^^^,^^^^^^^^ , ^^(^^^^−1,^^^^^^^^ , ^^^^)). The backwards pass is then performedto compute gradients ^^ are optimised using the optimiser,such as stochastic gradient descent SGD, or some other known optimiser. A validation lossthen optionally be calculated and the training loop is repeated until the predetermined numberof steps or epochs ^^ have been calculated, or some other criteria has been reached. Thelearning rate, batch size, and or number of epochs may be optimised during training, forexample using a learning rate scheduler or some other hyper parameter optimisation method.More generally, the hyper parameters may be optimised experimentally.Finally, it is envisaged that the above-described pyramid loss may be used together with amore complex loss function and may be one of any additional number of loss terms in the lossfunction.Concept 4: Multi-resolution general lossNext we generalise the above approach to all other cases where upsampling and / or downsamplingmay be performed in the AI-based compression pipeline, and where a difference may becomputed between pairs of the upsampled or downsampled information. In a first example,consider the general P-frame flow and residual modules of an AI-compression pipeline 1200shown in Figure 12, which corresponds to that of Figure 10b, but additionally includes pyramidfeature layers 1201 in the residual encoder.In the same way as with Concept 3 above, introducing additional loss terms based on theoutputs of one or more trainable up- and / or down-sampling layers encourages the weightsof those layers to converge to values that are closer to a global optimum than using a naiverate-distortion loss function that does explicitly take the overall effect of these layers intoaccount directly.In the case of Figure 12, the pyramid feature layers of Concept 3 are denoted by layers 1002but now the inputs to the residual encoder 211 are also downsampled, and the output of theflow decoder 1006 is optionally upsampled 1203 and the output of the residual decoder 213may also be optionally upsampled 1202. In this generalised example there are accordingly foursets of up and / or downsample layers with learnable parameters. If the naive rate-distortionloss function is used, the weights these layers converge due during training is not optimal asthe loss function does not explicitly take their effect into account.In contrast, in accordance with Concept 4, we introduce additional loss terms associated oneor more of these additional up- and / or down-sampling layers to encourage the learning duringtraining of a set of weights that directly optimises the outputs of these layers.More formally, we define general down- and up-sampling loss terms:∑^^^^=1 ^^(^^^^^^^^^^^^ , ^^^^^^^^^^^^ )∑^^^^=1 ^^(^^^^^^^^ , ^^^^^^^^ )The terms ^^ and ^^ may be any pair of inputs and / or outputs down- or up-sampled to a givenresolution by a layer ^^ with trainable weights. In the example of Figure 12, these general down-and up-sampling losses are applied respectively to the down- and up-sampling layers 1002,1201, 1202, and 1203, giving an overall loss function of:^^^^^^^^ = ^^ + ^^ ∗ ^^ + ∑^^ ∑^^^^=1 ^^(^^^^^^^^^^^^ , ^^^^^^^^^^^^ ) + ^^=1 ^^(^^^^^^^^ , ^^^^^^^^ ) Note in the specific case of Figure 12, at least one or more of ^^ or ^^ may comprise, forexample, flow information from a previous pair of frames, pairs of previous frames themselves,latent representations thereof, and so on all in any possible combination of ^^ and ^^.Applying the above general, multi-resolution loss function in pseudocode for training anAI-compression pipeline, we get the following generalised training algorithm:Algorithm 2 AI-based compression network training with custom loss functionInputs: Training dataset ^^, learning rate ^^, regularization parameter ^^, number of epochs ^^ ,network architecture ^^^^ including ^^ up- and or down-sampling layersInitialize network parameters ^^for ^^^^^^^^ℎ = 1 to ^^ dofor each batch (^^^^−1, ^^^^ ) in ^^ doForward pass to compute predictions ^^^^ = ^^^^ (^^^^−1, ^^^^ )∑mpute loss ^^^^^^^^ = ^^ + ^^ ∗ ^^ + ^^ ∑Co ^^^^=1 ^^(^^^^^^^^^^^^ , ^^^^^^^^^^^^ ) + ^^=1 ^^(^^^^^^^^ , ^^^^^^^^ )Backward Update parameters with optimizer O: ^^ ← O(^^, ∇^^ ^^^^^^^^, ^^)end forOptionally evaluate on validation setend for

[0003] Finally, it will be appreciated that the approach of concept two may also be used to introduceloss terms any any up- and / or downsampling layers not currently shown in Figure 12, forexample, any up- and / or downsampling performed in one or more I-frame modules (such asshown in Figure 3), or in the context of any resolution or dimension changes performed to anyinput or output tensors being passed between networks that are expecting to receive a giveninput and / or output dimension.The subject matter and the functional operations described in this specification can beimplemented in digital electronic circuitry, in tangibly-embodied computer software orfirmware, in computer hardware, including the structures disclosed in this specification andtheir structural equivalents, or in combinations of one or more of them. The subject matterdescribed in this specification can be implemented as one or more computer programs, i.e.,one or more modules of computer program instructions encoded on a tangible non transitoryprogram carrier for execution by, or to control the operation of, data processing apparatus.Alternatively or in addition, the program instructions can be encoded on an artificially generatedpropagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, thatis generated to encode information for transmission to suitable receiver apparatus for executionby a data processing apparatus. The computer storage medium can be a machine-readablestorage device, a machine-readable storage substrate, a random or serial access memory device,or a combination of one or more of them. The computer storage medium is not, however, apropagated signal.The term “data processing apparatus” encompasses all kinds of apparatus, devices, andmachines for processing data, including by way of example a programmable processor, acomputer, or multiple processors or computers. The apparatus can include special purposelogic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specificintegrated circuit). The apparatus can also include, in addition to hardware, code that createsan execution environment for the computer program in question, e.g., code that constitutesprocessor firmware, a protocol stack, a database management system, an operating system, ora combination of one or more of them.A computer program (which may also be referred to or described as a program, software, asoftware application, a module, a software module, a script, or code) can be written in anyform of programming language, including compiled or interpreted languages, or declarative orprocedural languages, and it can be deployed in any form, including as a stand alone program oras a module, component, subroutine, or other unit suitable for use in a computing environment.A computer program may, but need not, correspond to a file in a file system. A program can bestored in a portion of a file that holds other programs or data, e.g., one or more scripts storedin a markup language document, in a single file dedicated to the program in question, or inmultiple coordinated files, e.g., files that store one or more modules, sub programs, or portionsof code. A computer program can be deployed to be executed on one computer or on multiplecomputers that are located at one site or distributed across multiple sites and interconnected bya communication network.The processes and logic flows described in this specification can be performed by one or moreprogrammable computers executing one or more computer programs to perform functionsby operating on input data and generating output. The processes and logic flows can also beperformed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g.,an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).Computers suitable for the execution of a computer program include, by way of example,can be based on general or special purpose microprocessors or both, or any other kind ofcentral processing unit. Generally, a central processing unit will receive instructions and datafrom a read only memory or a random access memory or both. The essential elements ofa computer are a central processing unit for performing or executing instructions and oneor more memory devices for storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer data to, or both, one or moremass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks.However, a computer need not have such devices. Moreover, a computer can be embedded inanother device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio orvideo player, a VR headset, a game console, a Global Positioning System (GPS) receiver, aserver, a mobile phones, a tablet computer, a notebook computer, a music player, an e-bookreader, a laptop or desktop computer, a PDAs, a smart phone, or other stationary or portabledevices, that includes one or more processors and computer readable media, or a portablestorage device, e.g., a universal serial bus (USB) flash drive, to name just a few.Computer readable media suitable for storing computer program instructions and data includeall forms of non-volatile memory, media and memory devices, including by way of examplesemiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magneticdisks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM andDVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in,special purpose logic circuitry.The subject matter described in this specification can be implemented in a computing systemthat includes a back end component, e.g., as a data server, or that includes a middlewarecomponent, e.g., an application server, or that includes a front end component, e.g., a clientcomputer having a graphical user interface or a Web browser through which a user can interactwith an implementation of the subject matter described in this specification, or any combinationof one or more such back end, middleware, or front end components. The components of thesystem can be interconnected by any form or medium of digital data communication, e.g., acommunication network. Examples of communication networks include a local area network(“LAN”) and a wide area network (“WAN”), e.g., the Internet.The computing system can include clients and servers. A client and server are generally remotefrom each other and typically interact through a communication network. The relationship ofclient and server arises by virtue of computer programs running on the respective computersand having a client-server relationship to each other.While this specification contains many specific implementation details, these should beconstrued as descriptions of features that may be specific to particular examples of particularinventions. Certain features that are described in this specification in the context of separateexamples can also be implemented in combination in a single example. Conversely, variousfeatures that are described in the context of a single example can also be implemented inmultiple examples separately or in any suitable subcombination.Similarly, while operations are depicted in the drawings in a particular order, this should notbe understood as requiring that such operations be performed in the particular order shownor in sequential order, or that all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processing may be advantageous.Moreover, the separation of various system modules and components in the examples describedabove should not be understood as requiring such separation in all examples, and it should beunderstood that the described program components and systems can generally be integratedtogether in a single software product or packaged into multiple software products.

Claims

CLAIMS1. A method for lossy image or video encoding and transmission, and decoding, the methodcomprising the steps of:receiving a first image and a second image at a first computer system;with a first neural network, producing a latent representation of optical flow informationusing the first image and the second image, the optical flow information being indicative of adifference between the first image and the second image;transmitting the latent representation of optical flow information to a second computersystem; with a second neural network, decoding the latent representation of optical flowinformation to produce an approximation of the optical flow information;scaling the optical flow information; andwith a third neural network, producing an output image using the scaled optical flowinformation, wherein the output image is an approximation of the first image.

2. The method of claim 1, wherein producing the output image further comprises, with afourth neural network, producing residual information using the second image and the scaledoptical flow information, and decoding the residual information to produce the output image.

3. The method of claims 1 or 2, wherein scaling the optical flow information comprisesexponentially scaling respective elements of the optical flow information based on a value ofthe respective elements.

4. The method of claims 1 or 2, wherein scaling the optical flow information compriseslogarithmically scaling respective elements of the optical flow information based on a value ofthe respective elements.

5. The method of claims 1 or 2, wherein scaling the optical flow information comprises scalingrespective elements of the optical flow information based on a value of the respective elementsby applying a fifth neural network to the optical flow information.

6. The method of any of claims 1 to 5, comprising quantising the scaled optical flow informationbefore producing the output image.

7. The method of claim 6, wherein quantising comprises rounding respective elements of thescaled optical flow information into a plurality of quantisation bins based on a value of therespective elements.

8. A method for lossy image or video receipt and decoding, the method comprising the stepsof: decoding optical flow information at a first computer system using a first neural network,the optical flow information being indicative of a difference between a first image and thesecond image;scaling the optical flow information; andwith a third neural network, producing an output image using the scaled optical flowinformation, wherein the output image is an approximation of the first image.

9. A data processing apparatus configured to perform the method of claim 8.

10. A computer program comprising instructions which, when the program is executed by acomputer, cause the computer to carry out the method of any of claims 1 to 8.

11. A computer-readable storage medium comprising instructions which, when executed by acomputer, cause the computer to carry out the method of any of claims 1 to 8.

12. A method of training one or more neural networks, the one or more neural networks beingfor use in lossy image or video encoding, transmission and decoding, the method comprisingthe steps of:receiving a first image and a second image at a first computer system;with a first neural network, producing a latent representation of optical flow informationusing the first image and the second image, the optical flow information being indicative of adifference between the first image and the second image;with a second neural network, decoding the latent representation of optical flowinformation to produce an approximation of the optical flow information;scaling the optical flow information;with a third neural network, producing an output image using the scaled optical flowinformation, wherein the output image is an approximation of the first image;evaluating a function based on a difference between the first image and the outputimage;and updating the parameters of the first, second and / or third neural networks based on theevaluated function; andrepeating the above steps using a first set of input images to produce first, second and / orthird trained neural networks.

13. A method for lossy image or video encoding and transmission, and decoding, the methodcomprising the steps of:receiving a first image and a second image at a first computer system;with a first neural network, producing a latent representation of residual informationusing the first image and the second image, the residual information being indicative of adifference between the first image and the second image;transmitting the latent representation of residual information to a second computersystem; with a second neural network, decoding the latent representation of the residual informa-tion to produce a representation of the first image;scaling the representation of the first image to produce an output image, wherein theoutput image is an approximation of the first image.

14. The method of claim 13, wherein scaling the optical flow information comprises clampingrespective elements of the residual information between minimum and maximium values.

15. The method of claim 13, wherein scaling the residual information comprises exponentiallyscaling respective elements of the residual information based on a value of the respectiveelements.

16. The method of claim 13, wherein scaling the residual information comprises logarithmicallyscaling respective elements of the residual information based on a value of the respectiveelements.

17. The method of claim 13, wherein scaling the optical flow information comprises scalingrespective elements of the residual information based on a value of the respective elements byapplying a fourth neural network to the residual information.

18. A method for lossy image or video receipt and decoding, the method comprising the stepsof: decoding residual information at a first computer system using a first neural networkto produce a representation of a first image, the residual information being indicative of adifference between the first image and a second image; andscaling the representation of the first image to produce an output image, wherein theoutput image is an approximation of the first image.

19. A data processing apparatus configured to perform the method of claim 18.

20. A computer program comprising instructions which, when the program is executed by acomputer, cause the computer to carry out the method of any of claims 13 to 19.

21. A computer-readable storage medium comprising instructions which, when executed by acomputer, cause the computer to carry out the method of any of claims 13 to 19.

22. A method of training one or more neural networks, the one or more neural networks beingfor use in lossy image or video encoding, transmission and decoding, the method comprisingthe steps of:receiving a first image and a second image at a first computer system;with a first neural network, producing a latent representation of residual informationusing the first image and the second image, the residual information being indicative of adifference between the first image and the second image;with a second neural network, decoding the latent representation of the residual informa-tion to produce an a representation of the first image;scaling the representation of the first image to produce an output image, wherein theoutput image is an approximation of the first image; andevaluating a function based on a difference between the first image and the output image;updating the parameters of the first and second neural networks based on the evaluatedfunction; andrepeating the above steps using a first set of input images to produce first and secondtrained neural networks.

23. A method for lossy image or video encoding, transmission and decoding, the methodcomprising the steps of:receiving an input image at a first computer system;encoding the input image using a first trained neural network to produce a latentrepresentation; transmitting the latent representation to a second computer system;decoding the latent representation using a second trained neural network;scaling the decoded latent representation; andproducing an output image using scaled decoded latent representation, wherein theoutput image is an approximation of the input image.

24. A method of training one or more neural networks, the one or more neural networks beingfor use in lossy image or video encoding, transmission and decoding, the method comprisingthe steps of:receiving a first image and a second image at a first computer system;with a first neural network, producing a latent representation of optical flow informationusing the first image and the second image, the optical flow information being indicative of adifference between the first image and the second image;with a second neural network, decoding the latent representation of optical flowinformation to produce an approximation of the optical flow information;with a third neural network, producing an output image using the optical flow information,wherein the output image is an approximation of the first image;evaluating a function based on a representation in first and second resolutions of adifference between the first image and the second image;updating the parameters of the first, second and / or third neural networks based on theevaluated function; andrepeating the above steps using a first set of input images to produce first, second and / orthird trained neural networks.

25. The method of claim 24, wherein the representation of a difference in first and secondresolutions between the first image and the second image comprises a difference between thefirst image and a warped version of the second image.

26. The method of claim 24 or 25, wherein the first neural network comprises one or moredownsampling layers configured to produce the representation in first and second resolutionsbetween the first image and the second image.

27. The method of claim 26, wherein updating said parameters comprises updating theparameters of the downsampling layers.

28. The method of any of claims 24 to 27, wherein the function is further based on arepresentation in a third resolution of a difference between the first image and the secondimage.

29. The method of any of claims 24 to 28, wherein the function is further based on a differencebetween the first image and the output image.

30. The method of any of claims 24 to 29, comprising entropy encoding the latent representationof the optical flow information into a bitstream, and wherein the function is further based on abitrate of the bitstream.

31. The method of any of claims 24 to 30, comprising using the difference between the first andsecond images in the first resolution to warp the first or second image in the second resolution,and using the warped first or second image in the second resolution to estimate the differencebetween the first and second images in the second resolution.

32. The method of any of claims 24 to 31, wherein the the difference comprises a meansquared error.

33. The method of any of claims 24 to 32, wherein the representations in the first andsecond resolutions each comprise a tensor, and wherein the method comprises estimating saiddifference by estimating a mean squared error between said tensors.

34. A method for lossy image or video encoding, transmission and decoding, the methodcomprising the steps of:receiving an input image at a first computer system;encoding the input image using a first trained neural network to produce a latentrepresentation; transmitting the latent representation to a second computer system;decoding the latent representation using a second trained neural network to produce anoutput image, wherein the output image is an approximation of the input image;wherein the first neural network and the second neural network are trained according toany of the methods of claims 24 to 33.

35. A method for lossy image or video encoding and transmission, the method comprising thesteps of:receiving an input image at a first computer system;encoding the input image using a first trained neural network to produce a latentrepresentation; andtransmitting the latent representation;wherein the first neural network is trained according to any of the methods of claims 24to 33.

36. A method for lossy image or video receipt and decoding, the method comprising the stepsof: receiving a latent representation transmitted by a first computer system at a secondcomputer system, the latent representation corresponding to an input image; anddecoding the latent representation using a second trained neural network to produce anoutput image, wherein the output image is an approximation of the input image;wherein the second neural network is trained according to any of the methods of claims24 to 33.

37. A data processing apparatus configured to perform the method of any of claims 24 to 36.

38. A computer program comprising instructions which, when the program is executed by acomputer, cause the computer to carry out the method of any of claims 24 to 36.

39. A computer-readable storage medium comprising instructions which, when executed by acomputer, cause the computer to carry out the method of any of claims 24 to 36.

40. A method of training one or more neural networks, the one or more neural networks beingfor use in lossy image or video encoding, transmission and decoding, the method comprisingthe steps of:receiving a first image at a first computer system;with a first neural network, producing a latent representation of the first image;with a second neural network, decoding the latent representation of the first image toproduce an output image;wherein at least one of producing the latent representation or decoding the latentrepresentation comprises an up- or down-sampling step;evaluating a function based on an output of the up- or down-sampling step;updating the parameters of the first and / or second neural networks based on the evaluatedfunction; andrepeating the above steps using a first set of input images to produce first and / or secondtrained neural networks.

41. The method of claim 40, wherein the up- or down-sampling step is performed by one ormore layers of the first neural network.

42. The method of claim 41, wherein the up- or down-sampling step is performed by one ormore layers of the second neural network.

43. The method of any of claims 40 to 42, wherein updating said parameters comprisesupdating the parameters of the up- or down-sampling layers.

44. The method of any of claims 40 to 43, wherein the function is further based on a differencebetween the first image and the output image.

45. The method of any of claims 40 to 44, comprising entropy encoding the latent representationinto a bitstream, and wherein the function is further based on a bitrate of the bitstream.

46. A method for lossy image or video encoding, transmission and decoding, the methodcomprising the steps of:receiving an input image at a first computer system;encoding the input image using a first trained neural network to produce a latentrepresentation; transmitting the latent representation to a second computer system;decoding the latent representation using a second trained neural network to produce anoutput image, wherein the output image is an approximation of the input image;wherein the first neural network and the second neural network are trained according toany of the methods of claims 40 to 45.

47. A method for lossy image or video encoding and transmission, the method comprising thesteps of:receiving an input image at a first computer system;encoding the input image using a first trained neural network to produce a latentrepresentation; andtransmitting the latent representation;wherein the first neural network is trained according to any of the methods of claims 40to 45.

48. A method for lossy image or video receipt and decoding, the method comprising the stepsof: receiving a latent representation transmitted by a first computer system at a secondcomputer system, the latent representation corresponding to an input image; anddecoding the latent representation using a second trained neural network to produce anoutput image, wherein the output image is an approximation of the input image;wherein the second neural network is trained according to any of the methods of claims40 to 45.

49. A data processing apparatus configured to perform the method of any of claims 40 to 48.

50. A computer program comprising instructions which, when the program is executed by acomputer, cause the computer to carry out the method of any of claims 40 to 48.

51. A computer-readable storage medium comprising instructions which, when executed by acomputer, cause the computer to carry out the method of any of claims 40 to 48.

Images