Method and data processing system for lossy image or video encoding, transmission and decoding

Abstract

A method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first neural network to produce a latent representation; decoding the latent representation using a second neural network to produce an output image, wherein the output image is an approximation of the input image; evaluating a function based on a difference between the output image and the input image; updating the parameters of the first neural network and the second neural network based on the evaluated function; and wherein the method comprises regularising a term of the function using one or more regularisation parameters based on a target compression rate and / or a target distortion amount.

Description

[0001] Method and data processing system for lossy image or videoencoding, transmission and decodingBACKGROUNDThis invention relates to a method and system for lossy image or video encoding, transmissionand decoding, a method, apparatus, computer program and computer readable storage mediumfor lossy image or video encoding and transmission, and a method, apparatus, computerprogram and computer readable storage medium for lossy image or video receipt and decoding.There is increasing demand from users of communications networks for images and videocontent. Demand is increasing not just for the number of images viewed, and for the playingtime of video; demand is also increasing for higher resolution content. This places increasingdemand on communications networks and increases their energy use because of the largeramount of data being transmitted.To reduce the impact of these issues, image and video content is compressed for transmissionacross the network. The compression of image and video content can be lossless or lossycompression. In lossless compression, the image or video is compressed such that all of theoriginal information in the content can be recovered on decompression. However, when usinglossless compression there is a limit to the reduction in data quantity that can be achieved. Inlossy compression, some information is lost from the image or video during the compressionprocess. Known compression techniques attempt to minimise the apparent loss of informationby the removal of information that results in changes to the decompressed image or video thatis not particularly noticeable to the human visual system. JPEG, JPEG2000, AVC, HEVC andAVI are examples of compression processes for image and / or video files.In general terms, known lossy image compression techniques use the spatial correlationsbetween pixels in images to remove redundant information during compression. For example,in an image of a blue sky, if a given pixel is blue, there is a high likelihood that the neighbouringpixels, and their neighbouring pixels, and so on, are also blue. There is accordingly no need toretain all the raw pixel data. Instead, we can retain only a subset of the pixels which take upfewer bits and infer the pixel values of the other pixels using information derived from spatialcorrelations.A similar approach is applied in known lossy video compression techniques. That is, spatialcorrelations between pixels allow the removal of redundant information during compression.However, in video compression, there is further information redundancy in the form of temporalcorrelations. For example, in a video of an aircraft flying across a blue-sky background, mostof the pixels of the blue sky do not change at all between frames of the video. The mostof the blue sky pixel data for the frame at position t = 0 in the video is identical to that atposition t = 10. Storing this identical, temporally correlated, information is inefficient. Instead,only the blue sky pixel data for a subset of the frames is stored and the rest are inferred frominformation derived from temporal correlations.In the realm of lossy video compression in particular, the removal of redundant temporallycorrelated information in a video sequence is known inter-frame redundancy.One technique using inter-frame redundancy that is widely used in standard video compressionalgorithms involves the categorization of video frames into three types: I-frames, P-frames, andB-frames. Each frame type carries distinct properties concerning their encoding and decodingprocess, playing different roles in achieving high compression ratios while maintainingacceptable visual quality.I-frames, or intra-coded frames, serve as the foundation of the video sequence. These framesare self-contained, each one encoding a complete image without reference to any other frame.In terms of compression, I-frames are least compressed among all frame types, thus carryingthe most data. However, their independence provides several benefits, including being thestarting point for decompression and enabling random access, crucial for functionalities likefast-forwarding or rewinding the video.P-frames, or predictive frames, utilize temporal redundancy in video sequences to achievegreater compression. Instead of encoding an entire image like an I-frame, a P-frame representsthe difference between itself and the closest preceding I- or P-frame. The process, known asmotion compensation, identifies and encodes only the changes that have occurred, therebysignificantly reducing the amount of data transmitted. Nonetheless, P-frames are dependent onprevious frames for decoding. Consequently, any error during the encoding or transmissionprocess may propagate to subsequent frames, impacting the overall video quality.B-frames, or bidirectionally predictive frames, represent the highest level of compression.Unlike P-frames, B-frames use both the preceding and following frames as references in theirencoding process. By predicting motion both forwards and backwards in time, B-framesencode only the differences that cannot be accurately anticipated from the previous and nextframes, leading to substantial data reduction. Although this bidirectional prediction makesB-frames more complex to generate and decode, it does not propagate decoding errors sincethey are not used as references for other frames. Artificial intelligence (AI) based compressiontechniques achieve compression and decompression of images and videos through the use oftrained neural networks in the compression and decompression process. Typically, duringtraining of the neutral networks, the difference between the original image and video and thecompressed and decompressed image and video is analyzed and the parameters of the neuralnetworks are modified to reduce this difference while minimizing the data required to transmitthe content. However, AI based compression methods may achieve poor compression resultsin terms of the appearance of the compressed image or video or the amount of informationrequired to be transmitted.An example of an AI based image compression process comprising a hyper-network is describedin Ballé, Johannes, et al. “Variational image compression with a scale hyperprior.” arXivpreprint arXiv:1802.01436 (2018), which is hereby incorporated by reference.An example of an AI based video compression approach is shown in Agustsson, E., Minnen, D.,Johnston, N., Balle, J., Hwang, S. J., and Toderici, G. (2020), Scale-space flow for end-to-endoptimized video compression. In Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition (pp. 8503-8512), which is hereby incorporated by reference.A further example of an AI based video compression approach is shown in Mentzer, F.,Agustsson, E., Ballé, J., Minnen, D., Johnston, N., and Toderici, G. (2022, November). Neuralvideo compression using gans for detail synthesis and propagation. In Computer Vision–ECCV2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, PartXXVI (pp. 562-578), which is hereby incorporated by reference.SUMMARYAccording to an aspect, there is provided a method of training one or more neural networks,the one or more neural networks being for use in lossy image or video encoding, transmissionand decoding, the method comprising the steps of:receiving an input image at a first computer system;encoding the input image using a first neural network to produce a latent representation;decoding the latent representation using a second neural network to produce an outputimage, wherein the output image is an approximation of the input image;evaluating a function based on a difference between the output image and the inputimage; updating the parameters of the first neural network and the second neural network basedon the evaluated function; andrepeating the above steps using a first set of input images to produce a first trained neuralnetwork and a second trained neural network;wherein the method comprises regularising a term of the function using one or moreregularisation parameters based on a target compression rate and / or a target distortion amount.Optionally, said regularising comprises:updating the one or more regularisation parameters using a difference between a targetdistortion amount and an observed distortion amount indicative of a difference between theinput image and the output image.Optionally, said regularising comprises, after a number of said steps:updating the target distortion amount using a difference between a target compressionrate and an observed compression rate.Optionally, said observed compression rate is obtained from encoding one or more images of asecond set of input images using the first neural network.Optionally, said regularising comprises:updating the regularisation parameter using a difference between a target compressionrate and an observed compression rate of the input image after said encoding.Optionally, said regularising comprises, after a number of said steps:updating the target compression rate using a difference between a target distortion amountand an observed distortion amount indicative of a difference between the input image and theoutput image.Optionally, using the difference between the target distortion amount and the observed distortionamount comprises:applying a proportional–integral control loop function to the target distortion amountand the observed distortion amount, whereby the target distortion amount comprises a setpoint variable of the proportional–integral control loop function and the observed distortionamount comprises a process variable of the proportional-integral control loop.Optionally, using the difference between the target compression rate and the observed targetcompression rate comprises:applying a proportional–integral control loop function to the target compression rateand the observed compression rate, whereby the target compression rate comprises a set pointvariable of the proportional–integral control loop function and the observed compression ratecomprises a process variable of the proportional-integral control loop.Optionally, said regularising comprises, after a number of said steps:updating the target distortion amount or target compression rate using a differencebetween an observed metric and a target metric associated with the first neural network orsecond neural network.Optionally, the function is defined by a rate distortion equation L: L = ^^(^^) + ^^^^ (^^, ^^)where ^^ is said one or more regularisation parameters, ^^ (^^, ^^) is a distortion loss term basedon an observed difference between the input image ^^ and the output image ^^, and ^^(^^) is arate loss term based on an observed compression rate of the input image ^^ after said encoding;wherein the method comprises controlling ^^ with a first proportional-integral controlloop having a set point comprising a target distortion amount and a process variable comprisingan observed distortion amount; andwherein the method comprises controlling the target distortion amount with a secondproportional-integral control loop having a set point comprising a target compression rate anda process variable comprising an observed compression rate.Optionally, the observed compression rate is estimated by encoding one or more images of asecond set of input images using the first neural network, and estimating a compression rate ofthe one or more images after said encoding.Optionally, the observed compression rate is estimated by encoding one or more images of aplurality of sets of input images using the first neural network, and estimating a compressionrate of the one or more images after said encoding.Optionally, said controlling by the first proportional-integral controller comprises updating ^^and wherein said controlling by the second proportional-integral controller comprises updatingthe target distortion amount.Optionally, updating the target distortion amount by the second proportional-integral controlloop is performed less frequently than the updating of ^^.Optionally, mapping an output of the second proportional-integral control loop to a targetdistortion amount.Optionally, the output of the second proportional-integral control loop comprises a maximumvalue and a minimum value, and wherein the method comprises:mapping the maximum value and the minimum value to a maximum target distortionamount and a minimum target distortion amount; andinterpolating intermediate target distortion amounts from the maximum target distortionamount and the minimum target distortion amount.Optionally, the method comprises updating the set point of the first proportional-integralcontrol loop using one of the intermediate target distortion amounts.According to an aspect of the present disclosure, there is provided method of training one ormore neural networks, the one or more neural networks being for use in lossy image or videoencoding, transmission and decoding, the method comprising the steps of:receiving an input image at a first computer system;encoding the input image using a first neural network to produce a latent representation;decoding the latent representation using a second neural network to produce an outputimage, wherein the output image is an approximation of the input image;evaluating a function based on a difference between the output image and the inputimage; updating the parameters of the first neural network and the second neural network basedon the evaluated function; andrepeating the above steps using a first set of input images to produce a first trained neuralnetwork and a second trained neural network;wherein the method comprises regularising a term of the function using one or moreregularisation parameters based on a first target training metric; andupdating the first target training metric based on a second target training metric.Optionally, the first target training metric comprises a target distortion amount, and whereinthe second target training metric comprises a target compression rate.Optionally, the second target training metric comprises a plurality of target compression rates,and wherein the one or more regularisation parameters comprises a plurality of values, eachvalue associated with a different target compression rate.Optionally, the second target training metric comprises a plurality of target compression rates,and wherein the one or more regularisation parameters comprises a plurality of values, eachvalue associated with a different image of the first set of input images.Optionally, the first set of input images comprises an I-frame and one or more P- and / orB-frames, wherein the second target training metric comprises a plurality of target compressionrates, and wherein the one or more regularisation parameters comprises a plurality of values,each value associated with an I-frame, a P-frame or a B-frame.According to an aspect of the present disclosure, there is provided a method for lossy image orvideo encoding, transmission and decoding, the method comprising the steps of:receiving an input image at a first computer system;encoding the input image using a first trained neural network to produce a latentrepresentation; transmitting the latent representation to a second computer system;decoding the latent representation using a second trained neural network to produce anoutput image, wherein the output image is an approximation of the input image;wherein the first neural network and the second neural network are trained according toany of the above methods.According to an aspect of the present disclosure, there is provided a method for lossy image orvideo encoding and transmission, the method comprising the steps of:receiving an input image at a first computer system;encoding the input image using a first trained neural network to produce a latentrepresentation; andtransmitting the latent representation;wherein the first neural network is trained according to any of the above methods.According to an aspect of the present disclosure, there is provided a method for lossy image orvideo receipt and decoding, the method comprising the steps of:receiving a latent representation transmitted by a first computer system at a secondcomputer system, the latent representation corresponding to an input image; anddecoding the latent representation using a second trained neural network to produce anoutput image, wherein the output image is an approximation of the input image;wherein the second neural network is trained according to any of the above methods.According to an aspect of the present disclosure, there is provided a data processing systemconfigured to perform the method of any one of the above methods.According to an aspect of the present disclosure, there is provided a data processing apparatusconfigured to perform any one of the above methodsAccording to an aspect of the present disclosure, there is provided a computer programcomprising instructions which, when the program is executed by a computer, cause thecomputer to carry out any one of the above methods.According to an aspect of the present disclosure, there is provided a computer-readable storagemedium comprising instructions which, when executed by a computer, cause the computer tocarry out any one of the above methods.According to an aspect of the present disclosure, there is provided a method of training one ormore neural networks, the one or more neural networks being for use in lossy image or videoencoding, transmission and decoding, the method comprising the steps of:receiving an input image at a first computer system;encoding the input image using a first neural network to produce a latent representation;decoding the latent representation using a second neural network to produce an outputimage, wherein the output image is an approximation of the input image;evaluating a function based on a difference between the output image and the input imageand based on a difference between an observed compression rate and a target compression rateof the input image, the target compression rate sampled from a range of target compressionrates; updating the parameters of the first neural network and the second neural network basedon the evaluated function;repeating, using a set of input images, the receiving, encoding, decoding, evaluating andupdating steps a first number of times;changing the range of target compression rates; andrepeating, using the set of input images, the receiving, encoding, decoding, evaluatingand updating steps a second number of times to produce a first trained neural network and asecond trained neural network.Optionally, changing the range of target compression rates comprises increasing a maximumof the range and lowering a minimum of the range.Optionally, the method comprises sampling the target compression rate from a distribution oftarget compression rates in the range of target compression rates.Optionally, the method comprises biasing said sampling towards an upper end and / or lowerend of the distribution of target compression rates.Optionally, said biasing is performed after changing the range of target compression rates.Optionally, the method comprises regularising a term of the function using one or moreregularisation parameters based on the target compression rate and / or a target distortionamount indicative of the difference between the output image and the input image.Optionally, the method comprises updating the one or more regularisation parameters using adifference between an observed distortion amount and the target distortion amount.Optionally, the method comprises updating the target distortion amount using the differencebetween the observed compression rate and the target compression rate.Optionally, said changing the range of target compression rates is based on the differencebetween the observed compression rate and a target compression rate.Optionally, the method comprises performing said changing the range of target compressionrates when the difference between the observed compression rate and the target compressionrate falls below a threshold for a predetermined number of consecutive training steps.Optionally, changing the range of target compression rates comprises increasing a maximumof the range from 3 Mbps to 6 Mbps, and lowering a minimum of the range from 1 Mbps to0.1 Mbps.Optionally, the method further comprises:encoding the input image using a third neural network to produce a latent representation;decoding the latent representation using a fourth neural network to produce an outputimage, wherein the output image is an approximation of the input image;evaluating a function based on a difference between the output image and the input imageand based on a difference between an observed compression rate and a target compressionrate of the input image, the target compression rate sampled from a second range of targetcompression rates;updating the parameters of the third neural network and the fourth neural network basedon the evaluated function;repeating, using a set of input images, the receiving, encoding, decoding, evaluating andupdating steps a first number of times;changing the second range of target compression rates;repeating, using the set of input images, the receiving, encoding, decoding, evaluatingand updating steps a second number of times to produce a third trained neural network and afourth trained neural network;using the first and second trained neural networks to perform lossy video compression ata first level of an adaptive bitrate ladder; andusing the third and fourth trained neural networks to perform lossy video compression ata second level of the adaptive bitrate ladder.According to an aspect of the present disclosure, there is provided a method for lossy image orvideo encoding, transmission and decoding, the method comprising the steps of:receiving an input image at a first computer system;encoding the input image using a first trained neural network to produce a latentrepresentation; transmitting the latent representation to a second computer system;decoding the latent representation using a second trained neural network to produce anoutput image, wherein the output image is an approximation of the input image;wherein the first neural network and the second neural network are trained according toany of the above methods.According to an aspect of the present disclosure, there is provided a method for lossy image orvideo encoding and transmission, the method comprising the steps of:receiving an input image at a first computer system;encoding the input image using a first trained neural network to produce a latentrepresentation; andtransmitting the latent representation;wherein the first neural network is trained according to any of the above methods.According to an aspect of the present disclosure, there is provided a method for lossy image orvideo receipt and decoding, the method comprising the steps of:receiving a latent representation transmitted by a first computer system at a secondcomputer system, the latent representation corresponding to an input image; anddecoding the latent representation using a second trained neural network to produce anoutput image, wherein the output image is an approximation of the input image;wherein the second neural network is trained according to any of the above methods.According to an aspect of the present disclosure, there is provided a data processing systemconfigured to perform the method of any one of the above methods.According to an aspect of the present disclosure, there is provided a data processing apparatusconfigured to perform any one of the above methodsAccording to an aspect of the present disclosure, there is provided a computer programcomprising instructions which, when the program is executed by a computer, cause thecomputer to carry out any one of the above methods.According to an aspect of the present disclosure, there is provided a computer-readable storagemedium comprising instructions which, when executed by a computer, cause the computer tocarry out any one of the above methods.BRIEF DESCRIPTION OF THE DRAWINGSAspects of the invention will now be described by way of examples, with reference to thefollowing figures in which:Figure 1 illustrates an example of an image or video compression, transmission and decom-pression pipeline.Figure 2 illustrates a further example of an image or video compression, transmission anddecompression pipeline including a hyper-network.Figure 3 illustrates an example of a video compression, transmission and decompressionpipeline.Figure 4 illustrates an example of a video compression, transmission and decompressionsystem.Figure 5 illustrates an example of a control loop implementation for use with training a videocompression, transmission and decompression system.Figure 6 illustrates an example distortion against rate curve.Figure 7 illustrates an example distortion against rate curve.Figure 8 illustrates an example of a control loop implementation for use with training a videocompression, transmission and decompression system.DETAILED DESCRIPTION OF THE DRAWINGSCompression processes may be applied to any form of information to reduce the amountof data, or file size, required to store that information. Image and video information is anexample of information that may be compressed. The file size required to store the information,particularly during a compression process when referring to the compressed file, may bereferred to as the rate. In general, compression can be lossless or lossy. In both forms ofcompression, the file size is reduced. However, in lossless compression, no information is lostwhen the information is compressed and subsequently decompressed. This means that theoriginal file storing the information is fully reconstructed during the decompression process.In contrast to this, in lossy compression information may be lost in the compression anddecompression process and the reconstructed file may differ from the original file. Image andvideo files containing image and video data are common targets for compression.In a compression process involving an image, the input image may be represented as ^^. Thedata representing the image may be stored in a tensor of dimensions ^^ × ^^ × ^^, where ^^represents the height of the image, ^^ represents the width of the image and ^^ represents thenumber of channels of the image. Each ^^ × ^^ data point of the image represents a pixel valueof the image at the corresponding location. Each channel ^^ of the image represents a differentcomponent of the image for each pixel which are combined when the image file is displayed bya device. For example, an image file may have 3 channels with the channels representing thered, green and blue component of the image respectively. In this case, the image informationis stored in the RGB colour space, which may also be referred to as a model or a format.Other examples of colour spaces or formats include the CMKY and the YCbCr colour models.However, the channels of an image file are not limited to storing colour information and otherinformation may be represented in the channels. As a video may be considered a series ofimages in sequence, any compression process that may be applied to an image may also beapplied to a video. Each image making up a video may be referred to as a frame of the video.The output image may differ from the input image and may be represented by ^^. The differencebetween the input image and the output image may be referred to as distortion or a differencein image quality. The distortion can be measured using any distortion function which receivesthe input image and the output image and provides an output which represents the differencebetween input image and the output image in a numerical way. An example of such a methodis using the mean square error (MSE) between the pixels of the input image and the outputimage, but there are many other ways of measuring distortion, as will be known to the personskilled in the art. The distortion function may comprise a trained neural network.Typically, the rate and distortion of a lossy compression process are related. An increase inthe rate may result in a decrease in the distortion, and a decrease in the rate may result in anincrease in the distortion. Changes to the distortion may affect the rate in a correspondingmanner. A relation between these quantities for a given compression technique may be definedby a rate-distortion equation.AI based compression processes may involve the use of neural networks. A neural network isan operation that can be performed on an input to produce an output. A neural network maybe made up of a plurality of layers. The first layer of the network receives the input. One ormore operations may be performed on the input by the layer to produce an output of the firstlayer. The output of the first layer is then passed to the next layer of the network which mayperform one or more operations in a similar way. The output of the final layer is the output ofthe neural network.Each layer of the neural network may be divided into nodes. Each node may receive at leastpart of the input from the previous layer and provide an output to one or more nodes in asubsequent layer. Each node of a layer may perform the one or more operations of the layer onat least part of the input to the layer. For example, a node may receive an input from one ormore nodes of the previous layer. The one or more operations may include a convolution, aweight, a bias and an activation function. Convolution operations are used in convolutionalneural networks. When a convolution operation is present, the convolution may be performedacross the entire input to a layer. Alternatively, the convolution may be performed on at leastpart of the input to the layer.Each of the one or more operations is defined by one or more parameters that are associatedwith each operation. For example, the weight operation may be defined by a weight matrixdefining the weight to be applied to each input from each node in the previous layer to eachnode in the present layer. In this example, each of the values in the weight matrix is a parameterof the neural network. The convolution may be defined by a convolution matrix, also knownas a kernel. In this example, one or more of the values in the convolution matrix may be aparameter of the neural network. The activation function may also be defined by values whichmay be parameters of the neural network. The parameters of the network may be varied duringtraining of the network.Other features of the neural network may be predetermined and therefore not varied duringtraining of the network. For example, the number of layers of the network, the number ofnodes of the network, the one or more operations performed in each layer and the connectionsbetween the layers may be predetermined and therefore fixed before the training process takesplace. These features that are predetermined may be referred to as the hyperparameters of thenetwork. These features are sometimes referred to as the architecture of the network.To train the neural network, a training set of inputs may be used for which the expected output,sometimes referred to as the ground truth, is known. The initial parameters of the neuralnetwork are randomized and the first training input is provided to the network. The output ofthe network is compared to the expected output, and based on a difference between the outputand the expected output the parameters of the network are varied such that the differencebetween the output of the network and the expected output is reduced. This process is thenrepeated for a plurality of training inputs to train the network. The difference between theoutput of the network and the expected output may be defined by a loss function. The result ofthe loss function may be calculated using the difference between the output of the networkand the expected output to determine the gradient of the loss function. Back-propagation ofthe gradient descent of the loss function may be used to update the parameters of the neuralnetwork using the gradients ^^^^ / ^^^^ of the loss function. A plurality of neural networks in asystem may be trained simultaneously through back-propagation of the gradient of the lossfunction to each network.In the context of image or video compression, this type of system, where simultaneous trainingwith back-propagation through each element or the whole network architecture may be referredto as end-to-end, learned image or video compression. Unlike in traditional compressionalgorithms that use primarily handcrafted, manually constructed steps, an end-to-end learnedsystem learns itself during training what combination of parameters best achieves the goal ofminimising the loss function. This approach is advantageous compared to systems that are notend-to-end learned because an end-to-end system has a greater flexibility to learn weights andparameters that might be counter-intuitive to someone handcrafting features.It will be appreciated that the term "training" or "learning" as used herein means the processof optimizing an artificial intelligence or machine learning model, based on a given set of data.This involves iteratively adjusting the parameters of the model to minimize the discrepancybetween the model’s predictions and the actual data, represented by the above-describedrate-distortion loss function.The training process may comprise multiple epochs. An epoch refers to one complete passof the entire training dataset through the machine learning algorithm. During an epoch, themodel’s parameters are updated in an effort to minimize the loss function. It is envisaged thatmultiple epochs may be used to train a model, with the exact number depending on variousfactors including the complexity of the model and the diversity of the training data.Within each epoch, the training data may be divided into smaller subsets known as batches.The size of a batch, referred to as the batch size, may influence the training process. A smallerbatch size can lead to more frequent updates to the model’s parameters, potentially leading tofaster convergence to the optimal solution, but at the cost of increased computational resources.Conversely, a larger batch size involves fewer updates, which can be more computationallyefficient but might converge slower or even fail to converge to the optimal solution.The learnable parameters are updated by a specified amount each time, determined by thelearning rate. The learning rate is a hyperparameter that decides how much the parametersare adjusted during the training process. A smaller learning rate implies smaller steps in theparameter space and a potentially more accurate solution, but it may require more epochs toreach that solution. On the other hand, a larger learning rate can expedite the training processbut may risk overshooting the optimal solution or causing the training process to diverge.The training described herein may involve use of a validation set, which is a portion of thedata not used in the initial training, which is used to evaluate the model’s performance and toprevent overfitting. Overfitting occurs when a model learns the training data too well, to thepoint that it fails to generalize to unseen data. Regularization techniques, such as dropout orL1 / L2 regularization, can also be used to mitigate overfitting.It will be appreciated that training a machine learning model is an iterative process thatmay comprise selection and tuning of various parameters and hyperparameters. As will beappreciated, the specific details, such as hyper parameters and so on, of the training processmay vary and it is envisaged that producing a trained model in this way may achieved in anumber of different ways with different epochs, batch sizes, learning rates, regularisations,and so on, the details of which are not essential to enabling the advantages and effects of thepresent disclosure, except where stated otherwise. The point at which an “untrained” neuralnetwork is considered be “trained” is envisaged to be case specific and depend on, for example,on a number of epochs, a plateauing of any further learning, or some other metric and is notconsidered to be essential in achieving the advantages described herein.More details of an end-to-end, learned compression process will now be described. It will beappreciated that in some cases, end-to-end, learned compression processes may be combinedwith one or more components that are handcrafted or trained separately.In the case of AI based image or video compression, the loss function may be defined by therate distortion equation. The rate distortion equation may be represented by ^^^^^^^^ = ^^ + ^^ ∗ ^^,where ^^ is the distortion function, ^^ is a weighting factor, and ^^ is the rate loss. ^^ may bereferred to as a lagrange multiplier. The langrange multiplier provides as weight for a particularterm of the loss function in relation to each other term and can be used to control which termsof the loss function are favoured when training the network.In the case of AI based image or video compression, a training set of input images maybe used. An example training set of input images is the KODAK image set (for exampleat www.cs.albany.edu / xypan / research / snr / Kodak.html). An example training set of inputimages is the IMAX image set. An example training set of input images is the Imagenetdataset (for example at www.image-net.org / download). An example training set of inputimages is the CLIC Training Dataset P (“professional”) and M (“mobile”) (for example athttp: / / challenge.compression.cc / tasks / ).An example of an AI based compression, transmission and decompression process 100 isshown in Figure 1. As a first step in the AI based compression process, an input image 5 isprovided. The input image 5 is provided to a trained neural network 110 characterized by afunction ^^^^ acting as an encoder. The encoder neural network 110 produces an output basedon the input image. This output is referred to as a latent representation of the input image 5. Ina second step, the latent representation is quantised in a quantisation process 140 characterisedby the operation ^^, resulting in a quantized latent. The quantisation process transforms thecontinuous latent representation into a discrete quantized latent. An example of a quantizationprocess is a rounding function.In a third step, the quantized latent is entropy encoded in an entropy encoding process 150 toproduce a bitstream 130. The entropy encoding process may be for example, range or arithmeticencoding. In a fourth step, the bitstream 130 may be transmitted across a communicationnetwork.In a fifth step, the bitstream is entropy decoded in an entropy decoding process 160. Thequantized latent is provided to another trained neural network 120 characterized by a function^^^^ acting as a decoder, which decodes the quantized latent. The trained neural network 120produces an output based on the quantized latent. The output may be the output image of theAI based compression process 100. The encoder-decoder system may be referred to as anautoencoder.Entropy encoding processes such as range or arithmetic encoding are typically able to losslesslycompress given input data up to close to the fundamental entropy limit of that data, as determinedby the total entropy of the distribution of that data. Accordingly, one way in which end-to-end,learned compression can minimise the rate loss term of the rate-distortion loss function andthereby increase compression effectiveness is to learn autoencoder parameter values thatproduce low entropy latent representation distributions. Producing latent representationsdistributed with as low an entropy as possible allows entropy encoding to compress the latentdistributions as close to or to the fundamental entropy limit for that distribution. The lowerthe entropy of the distribution, the more entropy encoding can losslessly compress it and thelower the amount of data in the corresponding bitstream. In some cases where the latentrepresentation is distributed according to a gaussian or Laplacian distribution, this learningmay comprise learning optimal location and scale parameters of the gaussian or Laplaciandistributions, in other cases, it allows the learning of more flexible latent representationdistributions which can further help to achieve the minimising of the rate-distortion lossfunction in ways that are not intuitive or possible to do with handcrafted features. Examples ofthese and other advantages are described in WO2021 / 220008A1, which is incorporated in itsentirety by reference.Something which is closely linked to the entropy encoding of the latent distribution and whichaccordingly also has an effect on the effectiveness of compression of end-to-end learnedapproaches is the quantisation step. During inference, a rounding function may be used toquantise a latent representation distribution into bins of given sizes, a rounding function isnot differentiable everywhere. Rather, a rounding function is effectively one or more stepfunctions whose gradient is either zero (at the top of the steps) or infinity (at the boundarybetween steps). Back propagating a gradient of a loss function through a rounding functionis challenging. Instead, during training, quantisation by rounding function is replaced byone or more other approaches. For example, the functions of a noise quantisation model aredifferentiable everywhere and accordingly do allow backpropagation of the gradient of theloss function through the quantisation parts of the end-to-end, learned system. Alternatively, astraight-through estimator (STE) quantisation model or one other quantisation models may beused. It is also envisaged that different quantisation models may be used for during evaluationof different term of the loss function. For example, noise quantisation may used to evaluate therate or entropy loss term of the rate-distortion loss function while STE quantisation may beused to evaluate the distortion term.In a similar manner to how learning parameters top produce certain distributions of the latentrepresentation facilitates achieving better rate loss term minimisation, end-to-end learning ofthe quantisation process achieves a similar effect. That is, learnable quantisation parametersprovide the architecture with a further degree of freedom to achieve the goal of minimising theloss function. For example, parameters corresponding to quantisation bin sizes may be learnedwhich is likely to result in an improved rate-distortion loss outcome compared to approachesusing hand-crafted quantisation bin sizes.Further, as the rate-distortion loss function constantly has to balance a rate loss term against adistortion loss term, it has been found that the more degrees of freedom the system has duringtraining, the better the architecture is at achieving optimal rate and distortion trade off.The system described above may be distributed across multiple locations and / or devices. Forexample, the encoder 110 may be located on a device such as a laptop computer, desktopcomputer, smart phone or server. The decoder 120 may be located on a separate device whichmay be referred to as a recipient device. The system used to encode, transmit and decode theinput image 5 to obtain the output image 6 may be referred to as a compression pipeline.The AI based compression process may further comprise a hyper-network 105 for thetransmission of meta-information that improves the compression process. The hyper-network105 comprises a trained neural network 115 acting as a hyper-encoder ^^ ℎ^^and a trained neuralnetwork 125 acting as a hyper-decoder ^^ℎ^^. An example of such a is shown in Figure 2.Components of the system not further discussed may be assumed to be the same as discussedabove. The neural network 115 acting as a hyper-decoder receives the latent that is the output ofthe encoder 110. The hyper-encoder 115 produces an output based on the latent representationthat may be referred to as a hyper-latent representation. The hyper-latent is then quantizedin a quantization process 145 characterised by ^^ℎ to produce a quantized hyper-latent. Thequantization process 145 characterised by ^^ℎ may be the same as the quantisation process 140characterised by ^^ discussed above.In a similar manner as discussed above for the quantized latent, the quantized hyper-latent isthen entropy encoded in an entropy encoding process 155 to produce a bitstream 135. Thebitstream 135 may be entropy decoded in an entropy decoding process 165 to retrieve thequantized hyper-latent. The quantized hyper-latent is then used as an input to trained neuralnetwork 125 acting as a hyper-decoder. However, in contrast to the compression pipeline 100,the output of the hyper-decoder may not be an approximation of the input to the hyper-decoder115. Instead, the output of the hyper-decoder is used to provide parameters for use in theentropy encoding process 150 and entropy decoding process 160 in the main compressionprocess 100. For example, the output of the hyper-decoder 125 can include one or more ofthe mean, standard deviation, variance or any other parameter used to describe a probabilitymodel for the entropy encoding process 150 and entropy decoding process 160 of the latentrepresentation. In the example shown in Figure 2, only a single entropy decoding process 165and hyper-decoder 125 is shown for simplicity. However, in practice, as the decompressionprocess usually takes place on a separate device, duplicates of these processes will be presenton the device used for encoding to provide the parameters to be used in the entropy encodingprocess 150.Further transformations may be applied to at least one of the latent and the hyper-latent at anystage in the AI based compression process 100. For example, at least one of the latent and thehyper latent may be converted to a residual value before the entropy encoding process 150,155is performed. The residual value may be determined by subtracting the mean value of thedistribution of latents or hyper-latents from each latent or hyper latent. The residual valuesmay also be normalised.To perform training of the AI based compression process described above, a training set ofinput images may be used as described above. During the training process, the parameters ofboth the encoder 110 and the decoder 120 may be simultaneously updated in each trainingstep. If a hyper-network 105 is also present, the parameters of both the hyper-encoder 115and the hyper-decoder 125 may additionally be simultaneously updated in each training step.The training process may further include a generative adversarial network (GAN). Whenapplied to an AI based compression process, in addition to the compression pipeline describedabove, an additional neutral network acting as a discriminator is included in the system. Thediscriminator receives an input and outputs a score based on the input providing an indicationof whether the discriminator considers the input to be ground truth or fake. For example, theindicator may be a score, with a high score associated with a ground truth input and a lowscore associated with a fake input. For training of a discriminator, a loss function is used thatmaximizes the difference in the output indication between an input ground truth and input fake.When a GAN is incorporated into the training of the compression process, the output image 6may be provided to the discriminator. The output of the discriminator may then be used in theloss function of the compression process as a measure of the distortion of the compressionprocess. Alternatively, the discriminator may receive both the input image 5 and the outputimage 6 and the difference in output indication may then be used in the loss function of thecompression process as a measure of the distortion of the compression process. Training ofthe neural network acting as a discriminator and the other neutral networks in the compressionprocess may be performed simultaneously. During use of the trained compression pipelinefor the compression and transmission of images or video, the discriminator neural network isremoved from the system and the output of the compression pipeline is the output image 6.Incorporation of a GAN into the training process may cause the decoder 120 to performhallucination. Hallucination is the process of adding information in the output image 6 thatwas not present in the input image 5. In an example, hallucination may add fine detail tothe output image 6 that was not present in the input image 5 or received by the decoder 120.The hallucination performed may be based on information in the quantized latent received bydecoder 120.Details of a video compression process will now be described. As discussed above, a video ismade up of a series of images arranged in sequential order. AI based compression process100 described above may be applied multiple times to perform compression, transmissionand decompression of a video. For example, each frame of the video may be compressed,transmitted and decompressed individually. The received frames may then be grouped toobtain the original video.The frames in a video may be labelled based on the information from other frames that is usedto decode the frame in a video compression, transmission and decompression process. Asdescribed above, frames which are decoded using no information from other frames may bereferred to as I-frames. Frames which are decoded using information from past frames may bereferred to as P-frames. Frames which are decoded using information from past frames andfuture frames may be referred to as B-frames. Frames may not be encoded and / or decoded inthe order that they appear in the video. For example, a frame at a later time step in the videomay be decoded before a frame at an earlier time.The images represented by each frame of a video may be related. For example, a number offrames in a video may show the same scene. In this case, a number of different parts of thescene may be shown in more than one of the frames. For example, objects or people in a scenemay be shown in more than one of the frames. The background of the scene may also beshown in more than one of the frames. If an object or the perspective is in motion in the video,the position of the object or background in one frame may change relative to the position ofthe object or background in another frame. The transformation of a part of the image froma first position in a first frame to a second position in a second frame may be referred to asflow, warping or motion compensation. The flow may be represented by a vector. One or moreflows that represent the transformation of at least part of one frame to another frame may bereferred to as a flow map.An example AI based video compression, transmission, and decompression process 200 isshown in Figure 3. The process 200 shown in Figure 3 is divided into an I-frame part 201for decompressing I-frames, and a P-frame part 202 for decompressing P-frames. It will beunderstood that these divisions into different parts are arbitrary and the process 200 may bealso be considered as a single, end-to-end pipeline.As described above, I-frames do not rely on information from other frames so the I-frame part201 corresponds to the compression, transmission, and decompression process illustrated inFigures 1 or 2. The specific details will not be repeated here but, in summary, an input image^^0 is passed into an encoder neural network 203 producing a latent representation which isquantised and entropy encoded into a bitstream 204. The subscript 0 in ^^0 indicates the inputimage corresponds to a frame of a video stream at position t = 0. This may be the first frame ofan entire video stream or the first frame of a chunk of a video stream made up of, for example,an I-frame and a plurality of subsequent P-frames and / or B-frames. The bitstream 204 is thenentropy decoded and passed into a decoder neural network 205 to reproduce a reconstructedimage ^^0 which in this case is an I-frame. The decoding step may be performed both locallyat the same location as where the input image compression occurs as well as at the locationwhere the decompression occurs. This allows the reconstructed image ^^0 to be available forlater use by components of both the encoding and decoding sides of the pipeline.In contrast to I-frames, P-frames (and B-frames) do rely on information from other frames.Accordingly, the P-frame part 202 at the encoding side of the pipeline takes as input not onlythe input image ^^^^ that is to be compressed (corresponding to a frame of a video stream atposition t), but also one or more previously reconstructed images ^^^^−1 from an earlier framet-1. As described above, the previously reconstructed ^^^^−1 is available at both the encodeand decode side of the pipeline and can accordingly be used for various purposes at both theencode and decode sides.At the encode side, previously reconstructed images may be used for generating a flow mapscontaining information indicative of inter-frame movement of pixels between frames. In theexample of Figure 3, both the image being compressed ^^^^ and the previously reconstructedimage from an earlier frame ^^^^−1 are passed into a flow module part 206 of the pipeline. Theflow module part 206 comprises an autoencoder such as that of the autoencoder systems ofFigures 1 and 2 but where the encoder neural network 207 has been trained to produce alatent representation of a flow map from inputs ^^^^−1 and ^^^^ , which is indicative of inter-framemovement of pixels or pixel groups between ^^^^−1 and ^^^^ . The latent representation of the flowmap is quantised and entropy encoded to compress it and then transmitted as a bitstream 208.On the decode side, the bitstream is entropy decoded and passed to a decoder neural network209 to produce a reconstructed flow map ^^ .The reconstructed flow map ^^ is applied to the previously reconstructed image ^^^^−1 to generatea warped image ^^^^−1,^^. It is envisaged that any suitable warping technique may be used, forexample bi-linear or tri-linear warping, as is described in Agustsson, E., Minnen, D., Johnston,N., Balle, J., Hwang, S. J., and Toderici, G. (2020), Scale-space flow for end-to-end optimizedvideo compression. In Proceedings of the IEEE / CVF Conference on Computer Vision andPattern Recognition (pp. 8503-8512), which is hereby incorporated by reference. It is furtherenvisaged that a scale-space flow approach as described in the above paper may also optionallybe used. The warped image ^^^^−1,^^ is a prediction of how the previously reconstructed image^^^^−1 might have changed between frame positions t-1 and t, based on the output flow mapproduced by the flow module part 206 autoencoder system from the inputs of ^^^^ and ^^^^−1.As with the I-frame, the reconstructed flow map ^^ and corresponding warped image ^^^^−1,^^may be produced both on the encode side and the decode side of the pipeline so they areavailable for use by other components of the pipeline on both the encode and decode sides.In the example of Figure 3, both the image being compressed ^^^^ and the ^^^^−1,^^ are passedinto a residual module part 210 of the pipeline. The residual module part 210 comprises anautoencoder system such as that of the autoencoder systems of Figures 1 and 2 but where theencoder neural network 211 has been trained to produce a latent representation of a residualmap indicative of differences between the input mage ^^^^ and the warped image ^^^^−1,^^. Thelatent representation of the residual map is then quantised and entropy encoded into a bitstream212 and transmitted. The bitstream 212 is then entropy decoded and passed into a decoderneural network 213 which reconstructs a residual map ^^ from the decoded latent representation.Alternatively, a residual map may first be pre-calculated between ^^^^ and the ^^^^−1,^^ and thepre-calculated residual map may be passed into an autoencoder for compression only. Thishand-crafted residual map approach is computationally simpler, but reduces the degrees offreedom with which the architecture may learn weights and parameters to achieve its goalduring training of minimising the rate-distortion loss function.Finally, on the decode side, the residual map ^^ is applied (e.g. combined by addition, subtractionor a different operation) to the warped image to produce a reconstructed image ^^^^ which is areconstruction of image ^^^^ and accordingly corresponds to a P-frame at position t in a sequenceof frames of a video stream. It will be appreciated that the reconstructed image ^^^^ can then beused to process the next frame. That is, it can be used to compress, transmit and decompress^^^^+1, and so on until an entire video stream or chunk of a video stream has been processed.Alternatively, the residual autoencoder may be trained to reconstruct the frame ^^^^ directlyfrom the entropy decoded bitstream by removing the connection between ^^^^−1,^^ and the outputof the residual block 210, thereby eliminating any direct combination step with the warpedpreviously decoded image to speed up inference. In this case, the flow information is intuitivelyunderstood to be indirectly captured within the residual information, which the residual decoderis able to learn to use to directly reconstruct the output image ^^^^ .Alternatively, the residual autoencoder may be trained to reconstruct the frame ^^^^ directly fromthe entropy decoded bitstream in combination with some representation of flow injected intoone or more layers of the residual decoder. In this case, the flow information is intuitivelyunderstood to be indirectly captured within the injected information, which the residual decoderis able to learn to use while decoding the latent representation of flow information to directlyreconstruct the output image ^^^^ .Thus, for a block of video frames comprising an I-frame and ^^ subsequent P-frames, thebitstream may contain (i) a quantised, entropy encoded latent representation of the I-frameimage, and (ii) a quantised, entropy encoded latent representation of a flow map and residualmap of each P-frame image. For completeness, whilst not illustrated in Figure 3, any of theautoencoder systems of Figure 3 may comprise hyper and hyper-hyper networks such as thosedescribed in connection with Figure 2. Accordingly, the bitstream may also contain hyper andhyper-hyper parameters, their latent quantised, entropy encoded latent representations and soon, of those networks as applicable.Finally, the above approach may generally also be extended to B-frames, for example as isdescribed in Pourreza, R., and Cohen, T. (2021). Extending neural p-frame codecs for b-framecoding. In Proceedings of the IEEE / CVF International Conference on Computer Vision (pp.6680-6689).The above-described flow and residual based approach is highly effective at reducing theamount of data that needs to be transmitted because, as long as at least one reconstructed frame(e.g. I-frame ^^^^−1) is available, the encode side only needs to compress and transmit a flowmap and a residual map (and any hyper or hyper-hyper parameter information, as applicable)to reconstruct a subsequent frame.Figure 4 shows an example of an AI image or video compression process such as that describedabove in connection with Figures 1-3 implemented in a video streaming system 400. Thesystem 400 comprises a first device 401 and a second device 402. The first and seconddevices 401, 402 may be user devices such as smartphones, tablets, AR / VR headsets or otherportable devices. In contrast to known systems which primarily perform inference on GPUssuch as Nvidia A100, Geforce 3090, Gefore 4090 GPU cards, the system 400 of Figure 4performs inference on a CPU of the first and second devices respectively. That is, computefor performing both encoding and decoding are performed by the respective CPUs of the firstand second devices 401, 402. This places very different power usage, memory and runtimeconstraints on the implementation of the above methods than when implementing AI-basedcompression methods on GPUs. In one example, the CPU of first and second devices 401, 402may comprise a Qualcomm Snapdragon CPU.The first device 401 comprises a media capture device 403, such as a camera, arranged tocapture a plurality of images, referred to hereafter as a video stream 404, of a scene 404. Thevideo stream 404 is passed to a pre-processing module 406 which splits the video stream intoblocks of frames, various frames of which will be designated as I-frames, P-frames, and / orB-frames. The blocks of frames are then compressed by an AI-compression module 407comprising the encode side of the AI-based video compression pipeline of Figure 3. Theoutput of the AI-compression module is accordingly a bitstream 408a which is transmittedfrom the first device 401, for example via a communications channel, for example over oneor more of a WiFi, 3G, 4G or 5G channel, which may comprise internet or cloud-based 409communications.The second device 402 receives the communicated bitstream 408b which is passed to anAI-decompression module 410 comprising the decode side of the AI-based video compressionpipeline of Figure 3. The output of the AI-decompression module 402 is the reconstructedI-frames, P-frames and / or B-frames which are passed to a post-processing module 411 wherethey can prepared, for example passed into a buffer, in preparation for streaming 412 to andrendering on a display device 413 of the second device 402.It is envisaged that the system 400 of Figure 4 may be used for live video streaming at 30fps ofa 1080p video stream, which means a cumulative latency of both the encode and decode sideis below substantially 50ms, for example substantially 30ms or less. Achieving this level ofruntime performance with only CPU compute on user devices presents challenges which arenot addressed by known methods and systems or in the wider AI-compression literature.For example, execution of different parts of the compression pipeline during inferencemay be optimized by adjusting the order in which operations are performed using one ormore known CPU scheduling methods. Efficient scheduling can allow for operations to beperformed in parallel, thereby reducing the total execution time. It is also envisaged thatefficient management of memory resources may be implemented, including optimising cachingmethods such as storing frequently-accessed data in faster memory locations, and memoryreuse, which minimizes memory allocation and deallocation operations.A number of concepts related to the AI compression processes and / or their implementationin a hardware system discussed above will now be described. Although each concept isdescribed separately, one or more of the concepts described below may be applied in an AIbased compression process as described above.Concept 1: Regularisation ControllersTraining an AI-based compression pipeline to be able to compress input images to specificcompression rates is a challenging problem. A reason for this is that the interaction betweenthe compression rate and distortion in AI-based compression pipelines is unpredictable. Thismeans it is not easy to know a priori for a given training run what the final rate distortioncurve will look like, or indeed if the networks will be able to compress input images down to abitrate that a user will find useful. For example, in some cases a training run may producenetworks that can achieve state of the art reconstruction accuracy and very low distortion butcannot achieve high compression ratios (low compression rates). Other times, a training runmay produce networks that produce very small compression ratios (high compression rate) butvery poor image reconstruction accuracy. There are currently no reliable methods to controlthe desired bitrate range of the functioning model at inference. A workaround is to do multipletraining runs or select multiple training checkpoints and manually select the weights that areable to compress images down to rates that are closest to the rates that are desired for a givendistortion. This manual approach is hugely time consuming and error prone. The presentdisclosure is directed to solving this problem by introducing a novel control mechanism duringtraining.First, some additional context will be introduced. As described above, training of the neuralnetworks of an AI-based compression pipeline relies on a rate distortion loss function orobjective function. One or more terms of this loss function L may be regularisd by one ormore regularisation parameters:L = ^^(^^) + ^^^^ (^^, ^^)where ^^ is the one or more regularisation parameters, ^^ (^^, ^^) is a distortion loss term basedon an observed difference between the input image ^^ and the output image ^^, and ^^(^^) is a rateloss term based on an observed compression rate of the input image ^^ after encoding has beenperformed, for example the compression rate may be determined by a length of a bitstream, abits per pixel rate or any other compression metric, each of which is indicative of how small afile size an input image or images has been compressed to.The regularisation parameter ^^ may be a scalar, however as inputs to the AI-based compressionpipelines are typically in the form of tensors (e.g. images having height and width dimensions,and one or more channel dimensions), and because the inputs may be associated with differentframe types of a sequence (e.g. an I-frame and / or a P- or B-frame), the regularisation parameter^^ may instead comprise a plurality of values such that a different value of ^^ may be applieddepending on the inputs on which the loss is based at any given training step. Thus theregularisation parameter ^^ may be a matrix or tensor or other data structure which can facilitatethis. By constructing ^^ as a plurality of values, it introduces far greater control over howthe loss function is being regularised for different inputs and input types. In this case theloss L may be defined for a whole group of pictures (GOP) (e.g. L(^^^^^^) = ∑^^ L^^ ( ^^ ^^^^^^^^^^)whereby ^^ = (^^^^ (^^))^^)) which contrasts with the loss being associated with a frame (e.g.L( ^^ ^^^^^^^^) whereby ^^ = ^^(^^)).Second, the concept of a control loop in the context of a rate distortion loss function isintroduced. Control loops may be used in dynamic systems to force one or more processvariables in a system to converge to some set point. A known type of control loop is aproportional-integral controller. A proportional-integral controller may be defined by:∫^^(^^) = ^^^^ · ^^(^^) + ^^^^^^(^^) ^^^^where ^^^^ is a proportional gain, ^^^^ is an integral gain, ^^(^^) is a control signal or controlparameter and ^^(^^) is difference or error between a process variable and a set point at time ^^.Proportional-integral controllers force the system it is being applied to to correct errors ordifferences between a target and an observed value of a process variable based on both thecurrent value of the error and the accumulation of past errors, providing a balance betweenrapid response (due to the proportional action) and elimination of residual offset (due to theintegral action).A generalisation of a proportional-integral controller is a proportional-integral-derivativecontroller which also takes into account a derivative of the error. Whilst we refer generally to aproportional-integral controller herein. The same concepts may be applied and implementedequally well with a proportional-integral-derivative controller.The present inventors have found that applying a proportional-integral controller (or indeed aproportional-integral-derivative controller) to a rate-distortion loss function during training ofan AI-based compression pipeline, and specifically to the value or values of the regularisationparameter ^^ allows for fine-level control and tuning of the neural networks during trainingto force them to converge to weights that produce a target compression rate (or distortionrate) as applicable. At a high level, this can be understood as controlling how much the rateis contributing to the loss versus how much the distortion is contributing to the loss at eachtraining step (or after some predetermined number of training steps if less granular control isdesired) by dynamically updating ^^ as training progresses. More generally, it is envisaged thatthe PI or indeed PID controller may comprise an optimizer-based controller. That is, e(t) valuemay be passed through an optimiser, e.g. an Adam optimiser, SGD optimiser or some otheroptimiser, before the value of ^^ is updated.Thus, if after the application of an optimiser function during training the updated weightscause too much of an increase in a distortion loss resulting in neural networks that can producevery high compression rates but poor image reconstruction accuracy, the control loop adjuststhe regularisation parameter(s) ^^ such that, for the next training step, the optimiser functionafter backpropagation will adjust the weights to focus more on improving image reconstructionaccuracy over compression rates. This process is repeated for the next training step, and nexttraining step, and so on.The manner in which the control loop adjusts ^^ can be understood using the pseudocodealgorithm below:Algorithm 1 PI Controller for Adjusting to Target MSEInitialize ^^^^ , ^^^^ , ^^^^Initialize control signal ^^(0) ← 0Initialize cumulative error ^^ ← 0Initialize time step ^^ ← 0while system is running doMeasure ^^observed (^^)Calculate error: ^^(^^) = ^^target − ^^observed (^^)Update cumulative error: ^^ ← ^^ + ^^(^^) · ^^^^Compute control signal:^^(^^) = ^^^^ · ^^(^^) + ^^^^ · ^^Apply control signal ^^(^^) to regularisation parameter ^^^^ ← ^^ + ^^^^end while

[0002] That is, for a given training step at time ^^, the distortion value ^^^^^^^^^^^^^^^^^^ (e.g. an MSE value)between an input image ^^ and an output image ^^ is measured and compared to a target MSEvalue ^^^^^^^^^^^^^^ to produce an error. The cumulative error (i.e. the integral) is updated based onthe calculated error, and the error and cumulative error are scaled by ^^^^ and ^^^^ respectively tocalculate a control signal value ^^(^^) for that training step ^^. The control signal value ^^(^^) isthen applied to ^^ e.g. by multiplication or some other operation to modify the values of ^^ forthe next training step, thereby modifying the regularisation in the loss function and changinghow emphasis of rate versus distortion in the next training step ^^ + ^^^^.

[0003] Note that this pseudocode does not explicitly distinguish between the multiple values of thematrix or tensor of ^^ but, as described above, ^^ may comprise a plurality of values. Forexample, there may be different MSE targets ^^^^^^^^^^^^^^ and observed MSE values ^^^^^^^^^^^^^^^^^^ fordifferent frames or frame types of a sequence of frames G as well as for different compressionrate targets T (for example T = 0 may correspond to a low bit rate, and T = 10 may correspondto a high bit rate, and the values in between may correspond to intermediate target bit rates).In practice, we may thus define:^^^^ ^^ ← ^^(^^^^^^^^^^^^^^^^^^,^^ ^^ , ^^^^^^^^^^^^^^,^^ ^^ ) Whereby ^^^^ ^^ may be considered some data structure such as a tensor or matrix or array such as:0 1 · · · ^^ − 1^^00 ^^01 · · · ^^0,^^−1^^10 ^^11 · · · ^^1,^^−1...... ...... ^^^^−1,0 ^^^^−1,1 · · · ^^^^−1,^^−1During training, each of the individual values of ^^^^ ^^ , or some subset thereof, may be updatedby the control signal ^^(^^) produced by the control loop. Accordingly, the target MSE ^^^^^^^^^^^^^^and observed MSE ^^^^^^^^^^^^^^^^^^ may be calculated respectively for each target compression rate T(index ^^), as well as for each frame index or frame type of a sequence of frames (index ^^). Oneadvantage of this approach is that using a different ^^ value for different frames of a sequenceallows the regularisation of the loss to adapt to increases in distortion that are typically observedacross a sequence of frames of a video during training. This can be intuitively understood asbeing caused by P-frames (that re-use information from an I-frame) being less and less accuratethe further away they are in a sequence from the I-frame that contained the most information.In the example where we use different ^^ values for different frame types, ^^ may take thefollowing form:0 1 · · · ^^ − 1 ^^00 ^^01 · · · ^^0,^^−1P-frame ^^10 ^^11 · · · ^^1,^^−1B-frame ^^20 ^^21 · · · ^^2,^^−1Thus, applying the above to the rate distortion loss function, we get:L^^ ^^ (^^^^, ^^^^ ^^ ) := ^^(^^^^) + ^^^^ ^^^^ (^^^^, ^^^^ ^^ ) where ^^ and ^^ denote one of the respective values of ^^ such as target compression rate level,frame, or frame index and so on.Consider the following toy example with three compression target rates: low quality where T =0, medium quality where T = 1 and high quality T = 2. We also assume we have two differentframe types: I-frames and P-frames. In this toy example, we are training three sets of weights,each set associated with one of the compression target rates low quality, medium quality orhigh quality.Our ^^ in this example may accordingly be an array of 2x3 values, namely: (I-frame, P-frame)x(T=0, T=1, T=2). During each training step we, we check what kind of frame the MSEvalues are being calculated from (I-frame or P-frame), and then, for each set of weights, weapply one of the three ^^ values to the loss terms when calculating the loss. The weights arethen updated by an optimiser function and we then apply the control loop to update the array of^^ values for the next training step. In this way, as we train, our regularisation adapts accordingto the output of the control loop, which in turn is based on how close or how far the observedMSE is from the target MSE.However, whilst the above approach lets us target a specific distortion amount, there is notyet a link between the values of ^^ as they are updated and the compression rates. That is, thecontrol loop defined above is based solely on the target distortion amount (e.g. a target MSEvalue). Setting the target distortion amount (^^^^^^^^^^^^^^,^^ ^^ )is a non-trivial problem. If we naively guess what the ^^^^^^^^^^^^^^,^^ ^^ values should be, we end up with the same problemof not knowing a priori where our compression rates will settle as training progresses. Thatis, the control loop will cause the neural network’s weights to converge to some values thatcan output images with our desired, target distortion amounts (e.g. a predetermined MSEvalue) but we have no control over what rate at which the network compresses so as to achievethe target distortion. Whilst a low distortion amount may correspond to a small amount ofcompression (referred to hereinafter as a "high" compression rate or "low" compression ratio)whereby a high quality image reconstruction uses more bits, and a high distortion amount maycorrespond to a higher amount of compression (referred to hereinafter as a "low" compressionrate or "high" compression ratio) whereby a low quality image reconstruction uses fewer bits,the exact compression rates can vary significantly and are very difficult to guess solely fromsetting a target distortion amount.To solve this problem, the present disclosure introduces a second control loop. However, thissecond control loop does not target the rate or distortion loss by direct control of ^^. Instead, itindirectly influences ^^ by controlling the the target distortion amounts (^^^^^^^^^^^^^^,^^ ^^ ) used insideof the first control loop. That is, the first control loop can be considered a regularisationcontroller, whereas the second control loop can be considered a meta-parameter controllerthat updates the set point of the first control loop. Unlike the first control loop, this secondcontrol loop is directly tied to the target compression rates, as will be described below. Thissecond control loop thus introduces the (indirect) link between the values of ^^ and the targetcompression rates.As specified above we control the values of ^^ using the control loop function ^^(^^), which isbased on a difference between an observed MSE score ^^^^^^^^^^^^^^^^^^,^^ ^^ and a target MSE score^^^^^^^^^^^^^^,^^ ^^: ^^^^ ^^ ← ^^(^^^^^^^^^^^^^^^^^^,^^ ^^ , ^^^^^^^^^^^^^^,^^ ^^ ) To control the target MSE scores, we now introduce second control loop function and one ormore meta parameters ^^0, ^^1 ,^^2, ..., ^^^^ for each target compression bitrate ^^^^^^^^^^^^^^^^,^^ we wantto train for:^^^^ ← ^^(^^^^^^^^^^^^^^^^^^^^,^^ , ^^^^^^^^^^^^^^^^,^^ )where ^^ is a control loop function, ^^^^^^^^^^^^^^^^^^^^,^^ is an observed compression rate (bitrate) of avalidation dataset, and ^^^^^^^^^^^^^^^^,^^ is a target compression rate. The control loop function ^^ mayhave the same form as the first control loop ^^ described above. For example ^^ may comprise aPI controller or some other control loop function.The parameters ^^ℓ produced by ^^ are mapped to the distortion target values. For example,any mapping from {0...^^} of (^^ℓ)^^ℓ=0 to {1...^^} × {1...^^} could be appropriate. For example,^^0...^^^^ may be parameters spline system or parameters to a radial basis functionapproximation on the space of all MSE targets. For example, the mapping may be done explicitlyby hard-coding the mapping manually, or by associating the meta-parameter associated withthe lowest bitrate target ^^0 with the highest MSE target ^^0, ^^ , and the highest bitrate target^^^^ with the lowest MSE target ^^^^, ^^ , and then interpolating where the intermediate targetbitrates would have corresponding highest and lowest MSE targets. This interpolation maybe performed by a predefined interpolation function. The highest and lowest MSE targetvalues may be determined a priori either experimentally, or by setting values that are close tomaximum and minimum values that AI-based compression pipelines are able to achieve inthe wider literature. In this way, we only need to identify the end point MSE target values,associate these with our endpoint meta parameters ^^0 and ^^^^ , and interpolate the intermediateMSE target values from these endpoints. More generally, it is noted that this idea can beexpanded whereby the MSE targets or more general distortion targets need not be the sameas those used in the rate distortion loss estimation. For example, the distortion term in theabove-described loss functions may be some distortion ^^1 while the regularisation term ^^that regularises the distortion ^^1 may itself be based on a different distortion ^^2 based ondistortion target ^^^^^^^^^^^^^^ by way of operation of the controllers as described above.In a further generalisation, consider the spacing between the MSE targets. As described above,the association of metaparameters and MSE targets on the dataset being used by the controllersto estimate bitrate (referred to hereinafter as a holdout dataset and described in more detailbelow) is structured so there’s an abstraction between the metaparameters ^^0, ... ^^^^ and theMSE targets ^^0, ... ^^^^ . This means that under a suitable model for the rate-distortion function(for example a linear model or a hyperbolic model or some other parameterised model) theinter-level spacing of the holdout dataset bitrates (and therefore generalisation bitrates) can becontrolled directly by appropriately setting the spacing of the MSE targets. e.g., if you want thebitrate spacing to be linear (i.e., ^^^^^^^^^^^^^^^^^^^^,^^ - ^^^^^^^^^^^^^^^^^^^^,^^−1 is constant for all k) then you candirectly compute the required spacing of the T MSE targets; if you want it to be log-linear (i.e.,^^^^^^^^^^^^^^^^^^^^,^^ / ^^^^^^^^^^^^^^^^^^^^,^^−1 is constant for all k) then you can directly compute the requiredspacing of the T MSE targets. This results in a training strategy wherein sampling among theT levels during training thus prioritises bitrates equally, or log-equally or equally according towhatever parameterised model is being used.The result of this is that it allows for the bitrate level-sampling strategy to be adapted dependingon whether your model has an "easier time" learning low-bitrate structure or high-bitratestructure. This additional linking mechanism of the setting of the model spacing by someparametrised model is permitted directly by holdout dataset targeting using regularisationcontrollers, in conjunction with the design of a suitable rate–distortion model (i.e. that connectsdistortion targets to holdout dataset rate via meta-parameters). It is also envisaged that, duringtraining, spacing type (e.g. log-linear or linear or some other parameterised model) maybe switched on the fly. For example, if the model is having a challenging time learning alow-bitrate level structure with a log-linear spacing model, then this can be switched to linearafter some number of training steps or responsive to some trigger, and vice versa, to try toimprove the model’s ability to learn a given bitrate level structure.Taking all of the above into account, consider the following toy example: we have a highestbitrate target k = 9 and a lowest bitrate target k = 0, each with associated meta parameters ^^9and ^^0, and we want to train a set of networks to a total of 10 different levels between andthe highest and lowest bitrate targets. A typical highest MSE score may be e.g. 110(a unitless value) and a typical lowest MSE score may be e.g. 45 (another unitless value). Wecreate a mapping between these and the highest and lowest meta parameter scores wherebywhatever value ^^9 converges to, this will be mapped to 110, and whatever value ^^0 convergesto, this will be mapped to 45. We can then interpolate the MSE scores of total of 10 differentlevels, each associated with a ^^ value between the two endpoints.Next, we start training the network using the rate distortion loss in the same way as set outabove, that is, we estimate our lossL = ^^(^^) + ∑^^ ^^ ^^^^ ^^^^ (^^, ^^) and, at each weight update step, we update the weights of the network using an optimiser.After each step or a first predetermined number of steps, we run the first control loop to update^^^^ ^^ for the next step or number of steps:^^^^ ^^ ← ^^(^^^^^^^^^^^^^^^^^^,^^ ^^ , ^^^^^^^^^^^^^^,^^ ^^ ) Next, after a second predetermined number of steps, we run the second control loop to updatethe values of ^^^^^^^^^^^^^^,^^ ^^ :^^^^ ← ^^(^^^^^^^^^^^^^^^^^^^^,^^ , ^^^^^^^^^^^^^^^^,^^ ) We then continue training for the next second predetermined number of steps in the samemanner as described above, until we run the second control loop again to update the values of^^^^^^^^^^^^^^,^^ ^^ again and so on.The observed bitrate values ^^^^^^^^^^^^^^^^^^^^,^^ are calculated from a validation dataset that is separatefrom and not contaminated with either the training dataset or any other validation dataset usedduring training. This validation dataset is referred to above as a holdout dataset. The targetbitrate values ^^^^^^^^^^^^^^^^,^^ may correspond to some user-specified bitrates that may correspond toa the bitrates of a desired bitrate ladder or some other user-determined values and may be setmanually by the user, for example in a config file.Described below is an example training schedule implementing the above steps:First, we initialise ^^^^ ^^ , ^^^^^^^^^^^^^^,^^ ^^ , and ^^^^ with starting values, interpolate ^^^^ to the ^^^^^^^^^^^^^^,^^ ^^values, and example a set of 24 video each comprisinga number of frames.Second, we run 250 training steps using the these initial values, running the first control loopafter each training step to update the values of ^^^^ ^^ based on the observed MSE scores againstthe target MSE scores. After 250 training steps, we run a forward pass using the weights atstep 250 for each of the 24 video sequences and for each one calculate a compression rate (e.g.bpp or some other bitrate metric). These are then averaged to produce the observed bitrate^^^^^^^^^^^^^^^^^^^^,^^.Third, we run the second control loop using the estimated ^^^^^^^^^^^^^^^^^^^^,^^ to produce a set ofupdated meta parameters ^^^^ . We then use our mapping to map the updated meta parameters tonew MSE target values ^^ , and set these as the new set point in the first control loop.Fourth, we continue training for the next 250 steps using the updated ^^^^^^^^^^^^^^,^^ ^^ values. After250 steps we repeat the second and third steps, and so on.In this way, by dynamically updating the set point of the first control loop, based on a desiredcompression rate target, we directly provide a link between the regularisation in the lossfunction with both the target compression rate and the distortion amount, thereby facilitatinga "fire and forget" process of training whereby we only need to specify the desired bit ratetargets or even just the end points of the range we want and the network will learn to producenetworks that can achieve the desired targets. Thus the problem of needing to "guess" whatMSE scores will likely correspond to certain bitrates a priori is solved.In an alternative implementation of the above steps, it is also envisaged that the meta parametersmay be updated more frequently, for example every 4-5 steps, but that the forward pass isnot run on the entire holdout dataset, but only on one randomly selected video sequence.This approach results in more continuous control of the meta parameters and can reduce thecompute bottleneck of running a large number of forward passes at once if the second controlloop is only run after a larger number of steps e.g. 250. This approach may be facilitated by anAdam optimiser or other optimiser to produce an effect akin to amortisation of the updates overseveral iterations whereby the momentum helps to smooth the noise inherent to single-sampleupdates.It will also be appreciated that the above training schedule numbers of steps are exemplaryonly and other numbers of steps are also envisaged.Figure 5 illustratively shows an implementation 500 of the use of first and second controlloops to control regularisation of the rate distortion loss function when training the neuralnetworks of an AI-based compression pipeline. The regularisation parameter ^^^^ ^^ is illustratedas an array 501 of different values for different compression rate levels T-1 and frame indicesacross a sequence of frames from 0 to G-1. As described above, these values are updated bya controller 502, for example a PI controller that uses a target distortion amount (e.g. MSEscore or some other distortion metric) as a set point and an observed distortion amount as theprocess variable.The target distortion amounts 503, e.g. MSE targets, are linked by a mapping 504 to a set ofmeta parameters 505. This may be on a one-for-one basis, or by way of interpolation using thehighest and lowest values in the MSE target and meta-parameter ranges as the end points.The values of the meta parameters 505 are set by a meta controller 506 which, as with thecontroller 502 may be a PI controller that uses a target compression rate as a set point and anobserved compression rate on some validation data set as a process variable.Finally, the first control loop 502 and the second control loop 506 may be incorporated intoexisting training code or software through a targeting interface 507 that is able to access, forexample, the holdout data set and other parameters and values generated during training, suchas observed MSE scores and so on.It will be appreciated that the above approaches to loss function regularisation using first andsecond control loops may be generalised to more than just the use of the distortion amount andcompression rate as training metrics on which the control loops operate. Thus, for example,there may be other values or metrics that are calculated during training on which the controlloops may operate and thus on which the loss function may be regularised. One example maybe, for example, using a flow metric (e.g. an average flow amount output by the flow modulein Figure 3). Whereby if there is a flow term in the loss function and this term starts to growor shrink, a control loop that uses a difference between a target flow and an observed flowamount may be estimated in a PI controller to either directly regularise the flow loss termusing ^^ or some other regularisation parameter, or to indirectly control one or more of theseregularisation parameters using the same meta controller approach as described above. Forexample, controllers may be used to achieve automated smoothness (Lipschitz) control overone or more discriminators present in an adversarial training setup.Concept 2: Rate range wideningA bitrate ladder is a set of different video quality levels, each defined by a specific combinationof resolution and bitrate, that is used in adaptive streaming. These levels are created to ensurethat viewers with different network conditions can stream video content smoothly, withoutbuffering, by dynamically adjusting to the highest possible quality their connection can support.A bitrate ladder includes multiple tiers, with each step representing a different combinationof resolution (e.g., 144p, 360p, 720p, 1080p, etc.) and bitrate (e.g. measured in kilobits persecond, kbps, or megabits per second, Mbps). For instance, in a bitrate ladder, a lower tiermight offer 360p resolution at 500 kbps, while a higher tier might offer 1080p resolution at5 Mbps. Adaptive bitrate streaming (ABR) algorithms use this ladder to seamlessly switchbetween these levels based on the user’s internet bandwidth and device capabilities.Bitrate ladders in traditional compression accordingly rely on different encoding an input videostream using different encoder and / or decoder settings to achieve some desired set of steps onthe bitrate ladder.In AI-based compression, a plurality of sets of networks of the pipeline (each set being "amodel") may be trained, where each set of networks may converge towards being operationallyeffective at a given bitrate range, for example 0.1-1 Mbps, 1-3 Mbps, 3-6Mbps, 6+ Mbps andso on. To create a bitrate ladder, the plurality of models may be brought and used together inthe same way that encoding and decoding using different settings is used to create a bitrateladder in traditional compression.However, this approach may be problematic in AI-based compression because each modelmay have a not insignificant memory footprint and power consumption requirements. If abitrate ladder is implemented using the plurality of models, the memory footprint and powerconsumption may quickly become prohibitive for use on many devices.It is accordingly advantageous to train a single model that is operationally effective across amuch wider bitrate range than those described above, for example 0.1-9+ Mbps or 0.5-6 Mbpsto facilitate the implementation of functionality corresponding to a bitrate ladder but usingfewer models, for example one model, to reduce memory footprint and power consumptionand allow encoding and decoding to be run on resource constrained devices.However, the inventors have found that targeting wider bitrate ranges during the training ofAI-based compression pipelines, for example by exposing the model to different rate distortiontrade offs that correspond to a wide range of bitrates for example using the above-describedregularsiation controller approach, or any other rate-distortion trade off control method, resultsin a model that performs uniformly badly across the whole range. That is, models trainedto target wide bitrate ranges generally achieve poor distortion scores for a given bit rate i.e.poor bits per pixel performance across the entire, wide bitrate range despite that being theprimary target of training. Conversely, exposing the model to a narrow range of rate distortiontrade offs (i.e. targeting narrower bitrate ranges such as 1-3 Mbps) during training has no suchissues and the models are able to perform well within the targeted narrower range. In verygeneral terms, the networks of AI-based compression pipelines struggle to generalise to widebitrate ranges but have no difficulty on narrow bitrate ranges.Accordingly, one approach may be to train a plurality of said performant narrow bitrate rangemodels and to construct a bitrate ladder form them, accepting the undesirable consequencethat there may be burdensome memory and power limitations that prevent deployment of thepipeline on resource constrained devices.Alternatively, a solution to this problem may be found which allowing a single model to performeffectively across a wide bitrate range, and facilitating deployment of AI-based compressionpipelines on resource constrained devices. The present concept is directed to such a solution.More specifically, the inventors have found that whilst a randomly initialised model strugglesto learn to be performant across a wide bitrate range, a model that has already been subject totraining targeting a narrow bitrate range and is performant on that narrow range can have thebitrate range at which it is performant widened by further training where the (already narrowlytrained) model is exposed to a wider range of bitrate ranges than it was exposed to in theinitial training. At a very general level, a narrowly trained model can be coaxed into becomingperformant at a wider bitrate range where a freshly initialised model struggles to learn at all.Figure 6 illustrates a toy example distortion against rate curve 600 (in this case a VideoMultimethod Assessment Fusion VMAF score) against rate in Mbps of three separatelytrained models 601, 502, 603. That is, each model comprises the networks of an AI-basedcompression pipeline, such as that of Figure 3, trained by using a narrow range of rate distortiontrade offs (e.g. a different regularisation parameters ^^ in the rate distortion loss function isselected for each step or steps of training, for example as described above in connection withregularisation controllers). Also shown on Figure 6 is a dotted lined 604 which is indicative ofan ideal targeted rate distortion relationship or envelope that a given adaptive bitrate streamingalgorithm targets. That is, an adaptive bitrate streaming algorithm may switch between theavailable models based on a given connection speed to stay as close to the dotted line 604 aspossible for optimum end user viewing quality of the video stream. Combining the individualmodels 601, 602, 603 very roughly tracks the dotted line 604 and accordingly provides animplementation of a bitrate ladder for adaptive bitrate streaming using AI-based compression.In this case, the first model 601 may be used whenever the available connection speed is 0.1-1Mbps, the second model 602 may be used when the available connection speed is 1-3 Mbps,and the third model 603 may be used whenever the available connection speed is 3-6 Mbps.However, as described above, this toy example implementation requires three separate models,each having its own networks weights and other parameters and thus significantly increasingthe resource footprint needed to run the implementation.Figure 7 illustrates a similar toy example distortion against rate curve 700 with an optimal ratedistortion trade off line 704 of an arbitrary adaptive bitrate streaming algorithm. However,in Figure 7, a single trained model 702 which initially had a 1-3 Mbps effective operationalbitrate range has been subject to bitrate range widening 706 to 0.1-6 Mbps resulting in a newmodel 705 that approximately tracks the ideal adaptive bitrate streaming algorithm envelope704 from its lowest point to its highest point. The widened single model 705 is accordinglyable to produce the desired behaviour of an adaptive bitrate streaming alogrithm without theneed to rely on multiple underlying models to approximate the ideal distortion rate envelope704. In very general terms, widening the bitrate range of the a given model from an initiallytrained narrower range results in the effective operational bitrate range that that model is ableto work within to be extended. That means that the model is able to assign fewer bits to easilycompressible frames while also being free to use many more bits should doing so be useful forcompressing more complex frames without causing a large spike in distortion. A narrow rangemodel on the other hand does not know how to use these lower bitrate or higher bitrate ranges.That is, even if the frame might be more efficiently compressible to even smaller bit rate levels,a narrowly trained model is not able to take advantage of that possibility because of its narrowtraining.One non-limiting, exemplary implementation of the concept of rate range widening may makeuse of the above-described regularisation controllers methodology. This is illustrated in Figure8. Figure 8 illustratively shows an implementation 800 of the use of first and second controlloops to control regularisation of the rate distortion loss function when training the neuralnetworks of an AI-based compression pipeline. The regularisation parameter ^^^^ ^^ is illustratedas an array 801 of different values for different compression rate levels T-1 frame indicesacross a sequence of frames from 0 to G-1. As described above, these values are updated bya controller 802, for example a PI controller that uses a target distortion amount (e.g. MSEscore or some other distortion metric) as a set point and an observed distortion amount as theprocess variable.The target distortion amounts 803, e.g. MSE targets, are linked by a mapping 804 to a set ofmeta parameters 805. This may be on a one-for-one basis, or by way of interpolation using thehighest and lowest values in the MSE target and meta-parameter ranges as the end points.The values of the meta parameters 805 are set by a meta controller 806 which, as with thecontroller 802 may be a PI controller that uses a target compression rate as a set point and anobserved compression rate on some validation data set as a process variable.Finally, the first control loop 802 and the second control loop 806 may be incorporated intoexisting training code or software through a targeting interface 807 that is able to access, forexample, the holdout data set and other parameters and values generated during training, suchas observed MSE scores and so on.For a predetermined number of first training steps the above-described process is performedwith a first set of MSE targets between ^^0,^^, ^^ and ^^^^,^^, ^^ with associated metaparameters ^^0and ^^^^ which will result a model (i.e. AI-based compression pipeline such asthat of Figure 3) having some initial bitrate range at which it is able to operate effectively at.For the sake of example, let’s say this initial bitrate range is 1-3 Mbps.After these predetermined number of first training steps, the minimum and maximum MSEtargets become widened 808 to ^^′ , ^^′by widening the metaparameters ^^′ an′ 0,^^, ^^ ^^,^^, ^^ 0 d ^^^^ thatcontrol where the MSE targets settle. For a predetermined number of training steps,the above-described process is repeated but now with the MSE targets settled at wider valuesthrough the widening of the metaparameters. This results in a new array of ^^′^^, ^^ based onthe widened targets. Because the weights of the networks are already pretrained in the firsttraining steps, the continued training with the widened MSE targets and metaparameters iseffective the model’s bitrange range is widened. If we consider the toy example, the widenedbitrate range may now be 0.1-6 Mbps. Without the narrow bitrate range pretraining, the modelweights typically struggle to converge to a performant set of weights. That is, if training isstarted with a wide set of targets from the beginning, training is substantially unsuccessful andthe resulting models do not come close to a desired distortion rate performance of an idealenvelope such as 604 or 704 illustrated in Figures 6 and 7.An exemplary implementation of the above steps including pre-training and rate widening isillustrated in the pseudocode in algorithm 2 below.

[0004] Algorithm 2 Training with bitrate range wideningInputs: Training dataset X, learning rate ^^, regularization parameter ^^, number of narrow bitraterange epochs ^^1, number of wider bitrate range epochs ^^2, network architecture ^^^^ ,regularisation controllers with MSE target range [^^0,^^, ^^ , ^^^^,^^, ^^ ], metaparameter targetrange [^^0, ^^^^]Initialize network parameters ^^for epoch = 1 to ^^1 dofor each batch in X doWhile running regularisation controller with [^^0,^^, ^^ , ^^^^,^^, ^^ ] controlled by [^^0, ^^^^] to estimate ^^ for aselected level, select level ∈ (0, .., ^^) and perform pass to compute predictions ^^^^ = ^^^^ (^^^^−1, ^^^^ )Compute Loss = ^^ + ^^^^Backward pass to compute gradients ∇^^LossUpdate parameters with optimizer O: ^^ ← O(^^, ∇^^Loss, ^^)end forend forChange regularisation controller range to [^^′′′′ 0,^^, ^^ , ^^ ^^,^^, ^^] and [^^0, ^^^^] for epoch = 1 to ^^2 dofor each batch in X doWhile running regularisation controller with [^^′0,^^, ^^ , ^^′ ^^,^^, ^^] and [^^′0, ^^′ ^^] to estimate ^^ for a selectedlevel, select level ∈ (0, .., ^^) and perform forward pass to compute predictions ^^^^ = ^^^^ (^^^^−1, ^^^^ )Compute Loss = ^^ + ^^^^Backward pass to compute gradients ∇^^LossUpdate parameters with optimizer O: ^^ ← O(^^, ∇^^Loss, ^^)end forend forOptionally evaluate on validation setThat is, a training data set X, a learning rate ^^, regularisation parameter ^^, a first number oftraining steps or epochs ^^1 for the initial narrow bitrate range, and a second number of trainingsteps or epochs ^^2 for the bitrate range widening is selected. The network architecture of ^^^^ isdefined, for as shown in Figure 3. The network parameters ^^ are randomly initialisedand then the training loop is started. For each batch in the training data X, perform a forwardpass through ^^^^ while the regularsiation controller such as that of Figure 5 and / or 8 is runningwith the initial MSE target range [^^0,^^, ^^ , ^^^^,^^, ^^ ] controlled by the metaparameter target range[^^0, ^^^^] to estimate a ^^^^, ^^ for the pass. The total ^^^^^^^^ will be calculated by combining aterm ^^ and a rate term ^^, and any other loss terms (not shown). The backwards passis then performed to compute gradients based on the loss, and the parameters ^^ are optimisedusing the optimiser, such as stochastic gradient descent SGD, or some other known optimiser.After the first number of training steps or epochs ^^1, the MSE target range [^^0,^^, ^^ , ^^^^,^^, ^^ ] iswidened by widening the metaparameter target range [^^0, ^^^^ ] resulting in [^^′′ 0,^^, ^^ , ^^ ^^,^^, ^^] and[^^′0, ^^′ ^^]. The above steps are then repeated for the second number of or epochs^^2. This widening of the regularisation controller target ranges has the effect of widening thetarget compression rate range that the training directed to, effectively increasing the maximumand lowering the minimum of the range of bitrates that the model is trying to hit during training.As training progresses, the model’s effective bitrate range increases, getting as close to thewidened target compression rate range as the training allows it to get.Optionally, a validation loss can be calculated. The learning rate, batch size, and or numberof epochs may be optimised during training, for example using a learning rate scheduler orsome other hyperparameter optimisation method. More generally, the hyperparameters may beoptimised experimentally. Note that the first number of training steps or epochs ^^1 (i.e. whento extend the rate range during training) may be specified in an initial, empirically determinedtraining schedule based on a hyper parameter sweep. Alternatively it may be dynamicallydetermined based on an estimated difference between the observed compression rate and atarget compression rate for a training input image in a current training step or in aggregatefrom a number of previous training steps whereby this difference is indicative of how well thepre-training is progressing. If the difference is still above a threshold, it indicates pre-trainingat the narrow rate range still has work to do. Once the difference drops below the threshold,pre-training may be considered complete and the rate range can be extended for the secondnumber of training steps.A number of further optional modifications may be made to the above-described methodologyto further increase its effectiveness. The first of these is controlling the sampling of the targetlevels in the regularisation controller during the bitrate widening training. In very generalterms, for each forward pass of training, a single target level ∈ (0, ..., ^^) will be selected for thatforward pass. For example, in training step 1, the forward pass might be performed with theMSE target at ^^0, the next forward pass might be performed with an MSE target at ^^3, the nextforward pass might be performed with an MSE target at ^^1 and so on. This approach meansthe networks are exposed to the different bitrate range targets within between the minimumsand the maximums of the MSE target range [^^0,^^, ^^ , ^^^^,^^, ^^ ] and associated metaparameter targetrange [^^0, ^^^^]. This selection (or sampling) of be from a uniform distribution ofthat for each forward pass, the selection of each level is as equally likely as any otherlevel. But it may also be more explicitly controlled whereby the distribution of the levelsbeing sampled from may be biased towards a lower, middle, or higher end of MSE target range[^^0,^^, ^^ , ^^^^,^^, ^^ ] and associated metaparameter target range [^^0, ^^^^ ]. More specifically, the inventors have unexpectedly found that, during bitrate range widening,models struggle to learn to become performant at the lower bitrate ranges. That is, the bitraterange widening is harder to achieve in the lower bitrate range direction than it is for the higherbitrate range direction widening. To solve this problem, the target level sampling can be biasedtowards the lower target levels comnpared to the highest levels. In one illustrative example,the inventors have found that a distribution in which 90% or more of the forward passes areperformed at ^^ = 0, ^^ = 1, or ^^ = 2 (where ^^ ∈ (0, ..., 7)) results in bitrate range wideningfrom 1-3 Mbps to 0.1-6 Mbps with stable training and good performance across the wholerange after training even though this biasing means the higher target levels (^^ > 2) are rarely(<10% of the time) sampled in forward passes and the network is accordingly rarely evenexposed to these higher target levels. In more general terms, it is hypothesised that goodperformance at the lower bitrate ranges is a harder task to learn but once learned, that goodperformance ability appears to transfer well into the higher bitrate ranges for "free". It isaccordingly envisaged that bitrate widening training may focus on lower bitrate ranges toimprove overall perfomance compared to uniform target level sampling over the full bitraterange. Note that the different levels as defined by ^^ and the associated bitrate ranges areillustrative only and it will be appreciated that any number of levels and associated bitrateranges are envisaged and may be based on, for example, a desired ideal distortion rate curveshape such as those illustrated in Figures 6 and 7.A further optional modification to the above-described methods is the introduction of dynamicchanging of the regularisation controller targets for a given target compression rate range basedon how well (or not well) training is going. For example, consider a static training schedulethat trains first with a narrow target compression rate range for a predetermined number of firsttraining steps, then widens the target compression rate range , and then continues training withthe widened target compression rate range. It can be difficult to determine how many trainingsteps are appropriate to produce a pre-trained narrow bitrate range model. This exercise mayinvolve a large number of hyper parameter sweeps and other experimentation to determine asuitable training schedule. If the number of training steps at the narrow range are too few andthe pre-trained narrow bitrate range model is not performant at the targetd narrow range, thesubsequent widening of the bitrate range model is likely also not performant at the widenedrange. To reduce the burden of manually determining when to switch from narrow training tobitrate range widening in a given training schedule, a dynamic target compression rate rangemethod may be introduced that sets the minimum and maximum of the target compressionrate range every training step or every plurality of steps based on one or more training metricsestimated during training.For example, the weights before the start of narrow bitrate range training are unlikely to beable to consistently compress the input training videos to the target bitrate ranges as no traininghas yet occurred. The difference between the target compression rate sampled from the narrowtarget range for that training step and the observed compression rate at this point of trainingis likely to be very high. As training progresses, the networks of the model learn to operateeffectivley at the target narrow bitrate range and the difference between the target compressionrate and the observed compression rate in each step and in aggregate across training steps(e.g. in a rolling average) drops. This drop is indicative of the model learning to operateeffectively at the target bitrate range and indicative that the model may be receptive to bitraterange widening without causing a drop in performance. This relationship can be harnessed todynamically control or adjust the minimum and maximum of the target compression rate rangeby lowering the minimum and increasing the maximum of the range as the difference betweenthe observed compression rate and the target compression rate drops, while increasing theminimum and lowering the maximum of the range as the difference increases. The size ofany adjustment may be based on the size of the difference at a given training step or acrossmultiple training steps, for example proportional to the size of the difference. This dynamiccontrolling of the target compression rate range effectively operates as a controller on top ofthe above-described regularisation controller, dynamically adjusting when the range endpointsto control how fast or slow the bitrate widening part of training occurs. This in turn mayreduce the burden of training by removing the number of hyperparameter sweeps performedto determine an optimal training schedule. In more general terms, this modification has theeffect of converting a training schedule with discrete, hardcoded compression rate range targetchanges into a continuous, gradually changing training schedule that updates the range of targetcompression rates after each training step or after each plurality of steps. The training maythen continue in this way until some end condition is met, for example, when the overall loss orspecific loss term has converged, or after some predetermined overall number of training steps.Whilst a difference between an observed compression rate and a target compression rate areused as an example training metric, it is also envisaged that other metrics may be used tocontrol the target compression rate range minimum and maximum inlcuding but not limited toa difference between input and output image, a rate of change of the overall loss or specificloss terms (indicating a loss curve is plateauing), and so on.It will further be appreciated that, in an ideal scenario, a single model can be subject tobitrate range widening training to cover the entire range of an ideal distortion rate curve.However, for very large bitrate ranges, for example from 0.1 Mbps to 10+ Mbps, it may still bechallenging to train a single model to operate effectively across the entire range. In this case, itis envisaged that an adaptive bitrate ladder may be constructed using a plurality of models thathave each been subject to bitrate range widening training and that together approximately tracka desired ideal path in a distortion rate curve such as that of Figure 6 and 7. In this case eachmodel comprises its own set of neural networks and is trained separately with different initialcompression rate target ranges and different widened compression rate target ranges. In thiscase it is envisaged that, after widening the targets, the lowered minimum of the range of thehigher bitrate model will approximately match the increased maximum of the range of thelower bitrate model so that the two (or more) models may be combined into an adaptive bitrateladder using any suitable adaptive bitrate algorithm. That is, the first model may be used toperform lossy video compression at a first level of the adaptive bitrate ladder and the secondmodel may be used to perform lossy video compression at a second level of the adaptive bitrateladder. This approach allows even very large bitrate ranges to be covered with relatively fewermodels, for example two, three, four, or five models, than would otherwise be possible withoutusing bitrate range widening during training.It will also be appreciated from the description above that the presently described concept ofbitrate range widening may be used together with the regularisation controller concept or in astandalone manner wherein bitrate range widening may be performed in a different way. Thatis, the regularisation controller implementation of bitrate range widening is intended to beillustrative only and other implementations are also envisaged. For example, bitrate rangewidening may also be implemented using an empirically determined, hard coded regularisationparameter (e.g. ^^ in a rate distortion loss function) update schedule applied during training.Whilst such an approach is inferior to a regularisation controller implementation, it is providedas an example to illustrate that other ways to implement bitrate range widening are envisagedand viable, and that the regularisation controller implementation provided above is not intendedto be limiting.The subject matter and the functional operations described in this specification can beimplemented in digital electronic circuitry, in tangibly-embodied computer software orfirmware, in computer hardware, including the structures disclosed in this specification andtheir structural equivalents, or in combinations of one or more of them. The subject matterdescribed in this specification can be implemented as one or more computer programs, i.e.,one or more modules of computer program instructions encoded on a tangible non transitoryprogram carrier for execution by, or to control the operation of, data processing apparatus.Alternatively or in addition, the program instructions can be encoded on an artificially generatedpropagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, thatis generated to encode information for transmission to suitable receiver apparatus for executionby a data processing apparatus. The computer storage medium can be a machine-readablestorage device, a machine-readable storage substrate, a random or serial access memory device,or a combination of one or more of them. The computer storage medium is not, however, apropagated signal.The term “data processing apparatus” encompasses all kinds of apparatus, devices, andmachines for processing data, including by way of example a programmable processor, acomputer, or multiple processors or computers. The apparatus can include special purposelogic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specificintegrated circuit). The apparatus can also include, in addition to hardware, code that createsan execution environment for the computer program in question, e.g., code that constitutesprocessor firmware, a protocol stack, a database management system, an operating system, ora combination of one or more of them.A computer program (which may also be referred to or described as a program, software, asoftware application, a module, a software module, a script, or code) can be written in anyform of programming language, including compiled or interpreted languages, or declarative orprocedural languages, and it can be deployed in any form, including as a stand alone program oras a module, component, subroutine, or other unit suitable for use in a computing environment.A computer program may, but need not, correspond to a file in a file system. A program can bestored in a portion of a file that holds other programs or data, e.g., one or more scripts storedin a markup language document, in a single file dedicated to the program in question, or inmultiple coordinated files, e.g., files that store one or more modules, sub programs, or portionsof code. A computer program can be deployed to be executed on one computer or on multiplecomputers that are located at one site or distributed across multiple sites and interconnected bya communication network.The processes and logic flows described in this specification can be performed by one or moreprogrammable computers executing one or more computer programs to perform functionsby operating on input data and generating output. The processes and logic flows can also beperformed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g.,an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).Computers suitable for the execution of a computer program include, by way of example,can be based on general or special purpose microprocessors or both, or any other kind ofcentral processing unit. Generally, a central processing unit will receive instructions and datafrom a read only memory or a random access memory or both. The essential elements ofa computer are a central processing unit for performing or executing instructions and oneor more memory devices for storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer data to, or both, one or moremass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks.However, a computer need not have such devices. Moreover, a computer can be embedded inanother device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio orvideo player, a VR headset, a game console, a Global Positioning System (GPS) receiver, aserver, a mobile phones, a tablet computer, a notebook computer, a music player, an e-bookreader, a laptop or desktop computer, a PDAs, a smart phone, or other stationary or portabledevices, that includes one or more processors and computer readable media, or a portablestorage device, e.g., a universal serial bus (USB) flash drive, to name just a few.Computer readable media suitable for storing computer program instructions and data includeall forms of non-volatile memory, media and memory devices, including by way of examplesemiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magneticdisks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM andDVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in,special purpose logic circuitry.The subject matter described in this specification can be implemented in a computing systemthat includes a back end component, e.g., as a data server, or that includes a middlewarecomponent, e.g., an application server, or that includes a front end component, e.g., a clientcomputer having a graphical user interface or a Web browser through which a user can interactwith an implementation of the subject matter described in this specification, or any combinationof one or more such back end, middleware, or front end components. The components of thesystem can be interconnected by any form or medium of digital data communication, e.g., acommunication network. Examples of communication networks include a local area network(“LAN”) and a wide area network (“WAN”), e.g., the Internet.The computing system can include clients and servers. A client and server are generally remotefrom each other and typically interact through a communication network. The relationship ofclient and server arises by virtue of computer programs running on the respective computersand having a client-server relationship to each other.While this specification contains many specific implementation details, these should beconstrued as descriptions of features that may be specific to particular examples of particularinventions. Certain features that are described in this specification in the context of separateexamples can also be implemented in combination in a single example. Conversely, variousfeatures that are described in the context of a single example can also be implemented inmultiple examples separately or in any suitable sub-combination.Similarly, while operations are depicted in the drawings in a particular order, this should notbe understood as requiring that such operations be performed in the particular order shownor in sequential order, or that all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processing may be advantageous.Moreover, the separation of various system modules and components in the examples describedabove should not be understood as requiring such separation in all examples, and it should beunderstood that the described program components and systems can generally be integratedtogether in a single software product or packaged into multiple software products.Finally, it will be appreciated that whilst the term controller and meta controller are used hereinin the singular, it will be appreciated that these may comprise multiple control loops and soaccordingly be controllers or meta controllers, each able to control one or more values of thearray of ^^ or ^^ values individually but together providing the singular controller functionality.

Claims

CLAIMS1. A method of training one or more neural networks, the one or more neural networks beingfor use in lossy image or video encoding, transmission and decoding, the method comprisingthe steps of:receiving an input image at a first computer system;encoding the input image using a first neural network to produce a latent representation;decoding the latent representation using a second neural network to produce an outputimage, wherein the output image is an approximation of the input image;evaluating a function based on a difference between the output image and the inputimage; updating the parameters of the first neural network and the second neural network basedon the evaluated function; andrepeating the above steps using a first set of input images to produce a first trained neuralnetwork and a second trained neural network;wherein the method comprises regularising a term of the function using one or moreregularisation parameters based on a target compression rate and / or a target distortion amount.

2. The method of claim 1, wherein said regularising comprises:updating the one or more regularisation parameters using a difference between a targetdistortion amount and an observed distortion amount indicative of a difference between theinput image and the output image.

3. The method of claim 2, wherein said regularising comprises, after a number of said steps:updating the target distortion amount using a difference between a target compressionrate and an observed compression rate.

4. The method of claim 3, wherein said observed compression rate is obtained from encodingone or more images of a second set of input images using the first neural network.

5. The method of claim 1, wherein said regularising comprises:updating the regularisation parameter using a difference between a target compressionrate and an observed compression rate of the input image after said encoding.

6. The method of claim 5, wherein said regularising comprises, after a number of said steps:updating the target compression rate using a difference between a target distortion amountand an observed distortion amount indicative of a difference between the input image and theoutput image.

7. The method of claim 2 or 6, wherein using the difference between the target distortionamount and the observed distortion amount comprises:applying a proportional–integral control loop function to the target distortion amountand the observed distortion amount, whereby the target distortion amount comprises a setpoint variable of the proportional–integral control loop function and the observed distortionamount comprises a process variable of the proportional-integral control loop.

8. The method of any of claims 3 to 5, wherein using the difference between the targetcompression rate and the observed target compression rate comprises:applying a proportional–integral control loop function to the target compression rateand the observed compression rate, whereby the target compression rate comprises a set pointvariable of the proportional–integral control loop function and the observed compression ratecomprises a process variable of the proportional-integral control loop.

9. The method of claim 2 or 5, wherein said regularising comprises, after a number of saidsteps: updating the target distortion amount or target compression rate using a differencebetween an observed metric and a target metric associated with the first neural network orsecond neural network.

10. The method of any of claims 1 to 9, wherein the function is defined by a rate distortionequation L:L = ^^(^^) + ^^^^ (^^, ^^)where ^^ is said one or more regularisation parameters, ^^ (^^, ^^) is a distortion loss term basedon an observed difference between the input image ^^ and the output image ^^, and ^^(^^) is arate loss term based on an observed compression rate of the input image ^^ after said encoding;wherein the method comprises controlling ^^ with a first proportional-integral controlloop having a set point comprising a target distortion amount and a process variable comprisingan observed distortion amount; andwherein the method comprises controlling the target distortion amount with a secondproportional-integral control loop having a set point comprising a target compression rate anda process variable comprising an observed compression rate.

11. The method of claim 10, wherein the observed compression rate is estimated by encodingone or more images of a second set of input images using the first neural network, andestimating a compression rate of the one or more images after said encoding.

12. The method of claim 10 or 11, wherein the observed compression rate is estimated byencoding one or more images of a plurality of sets of input images using the first neuralnetwork, and estimating a compression rate of the one or more images after said encoding.

13. The method of any of claims 10 to 12, wherein said controlling by the first proportional-integral controller comprises updating^^ and wherein said controlling by the second proportional-integral controller comprises updating the target distortion amount.

14. The method of claim 13 wherein updating the target distortion amount by the secondproportional-integral control loop is performed less frequently than the updating of ^^.

15. The method of claim 13 or 14, mapping an output of the second proportional-integralcontrol loop to a target distortion amount.

16. The method of any of claims 13 to 15, wherein the output of the second proportional-integralcontrol loop comprises a maximum value and a minimum value, and wherein the methodcomprises:mapping the maximum value and the minimum value to a maximum target distortionamount and a minimum target distortion amount; andinterpolating intermediate target distortion amounts from the maximum target distortionamount and the minimum target distortion amount.

17. The method of claim 16, comprising updating the set point of the first proportional-integralcontrol loop using one of the intermediate target distortion amounts.

18. A method of training one or more neural networks, the one or more neural networks beingfor use in lossy image or video encoding, transmission and decoding, the method comprisingthe steps of:receiving an input image at a first computer system;encoding the input image using a first neural network to produce a latent representation;decoding the latent representation using a second neural network to produce an outputimage, wherein the output image is an approximation of the input image;evaluating a function based on a difference between the output image and the inputimage; updating the parameters of the first neural network and the second neural network basedon the evaluated function; andrepeating the above steps using a first set of input images to produce a first trained neuralnetwork and a second trained neural network;wherein the method comprises regularising a term of the function using one or moreregularisation parameters based on a first target training metric; andupdating the first target training metric based on a second target training metric.

19. The method of claim 18, wherein the first target training metric comprises a target distortionamount, and wherein the second target training metric comprises a target compression rate.

20. The method of claim 19, wherein the second target training metric comprises a plurality oftarget compression rates, and wherein the one or more regularisation parameters comprises aplurality of values, each value associated with a different target compression rate.

21. The method of claim 19 or 20, wherein the second target training metric comprises aplurality of target compression rates, and wherein the one or more regularisation parameterscomprises a plurality of values, each value associated with a different image of the first set ofinput images.

22. The method of claim 18, 19 or 20, wherein the first set of input images comprises an I-frameand one or more P- and / or B-frames, wherein the second target training metric comprises aplurality of target compression rates, and wherein the one or more regularisation parameterscomprises a plurality of values, each value associated with an I-frame, a P-frame or a B-frame.

23. A method for lossy image or video encoding, transmission and decoding, the methodcomprising the steps of:receiving an input image at a first computer system;encoding the input image using a first trained neural network to produce a latentrepresentation; transmitting the latent representation to a second computer system;decoding the latent representation using a second trained neural network to produce anoutput image, wherein the output image is an approximation of the input image;wherein the first neural network and the second neural network are trained according toany of the methods of claims 1 to 19.

24. A method for lossy image or video encoding and transmission, the method comprising thesteps of:receiving an input image at a first computer system;encoding the input image using a first trained neural network to produce a latentrepresentation; andtransmitting the latent representation;wherein the first neural network is trained according to any of the methods of claims 1 to19.

25. A method for lossy image or video receipt and decoding, the method comprising the stepsof: receiving a latent representation transmitted by a first computer system at a secondcomputer system, the latent representation corresponding to an input image; anddecoding the latent representation using a second trained neural network to produce anoutput image, wherein the output image is an approximation of the input image;wherein the second neural network is trained according to any of the methods of claims1 to 22.

26. A data processing system configured to perform the method of any one of claims 1 to 25.

27. A data processing apparatus configured to perform the method of claims 1 to 25.

28. A computer program comprising instructions which, when the program is executed by acomputer, cause the computer to carry out the method of claims 1 to 25.

29. A computer-readable storage medium comprising instructions which, when executed by acomputer, cause the computer to carry out the method of claims 1 to 25.

30. A method of training one or more neural networks, the one or more neural networks beingfor use in lossy image or video encoding, transmission and decoding, the method comprisingthe steps of:receiving an input image at a first computer system;encoding the input image using a first neural network to produce a latent representation;decoding the latent representation using a second neural network to produce an outputimage, wherein the output image is an approximation of the input image;evaluating a function based on a difference between the output image and the input imageand based on a difference between an observed compression rate and a target compression rateof the input image, the target compression rate sampled from a range of target compressionrates;updating the parameters of the first neural network and the second neural network basedon the evaluated function;repeating, using a set of input images, the receiving, encoding, decoding, evaluating andupdating steps a first number of times;changing the range of target compression rates; andrepeating, using the set of input images, the receiving, encoding, decoding, evaluatingand updating steps a second number of times to produce a first trained neural network and asecond trained neural network.

31. The method of claim 30, wherein changing the range of target compression rates comprisesincreasing a maximum of the range and lowering a minimum of the range.

32. The method of any of claims 30 to 31, comprising sampling the target compression ratefrom a distribution of target compression rates in the range of target compression rates.

33. The method of claim 32, comprising biasing said sampling towards an upper end and / orlower end of the distribution of target compression rates.

34. The method of claim 33, wherein said biasing is performed after changing the range oftarget compression rates.

35. The method of any of claims 30 to 34, comprising regularising a term of the function usingone or more regularisation parameters based on the target compression rate and / or a targetdistortion amount indicative of the difference between the output image and the input image.

36. The method of claim 35, comprising updating the one or more regularisation parametersusing a difference between an observed distortion amount and the target distortion amount.

37. The method of claim 36, comprising updating the target distortion amount using thedifference between the observed compression rate and the target compression rate.

38. The method of any of claims 30 to 37, wherein said changing the range of targetcompression rates is based on the difference between the observed compression rate and atarget compression rate.

39. The method of claim 38, comprising performing said changing the range of targetcompression rates when the difference between the observed compression rate and the targetcompression rate falls below a threshold for a predetermined number of consecutive trainingsteps.

40. The method of any of claims 30 to 39, wherein changing the range of target compressionrates comprises increasing a maximum of the range from 3 Mbps to 6 Mbps, and lowering aminimum of the range from 1 Mbps to 0.1 Mbps.

41. The method of any of claims 30-40, comprising:encoding the input image using a third neural network to produce a latent representation;decoding the latent representation using a fourth neural network to produce an outputimage, wherein the output image is an approximation of the input image;evaluating a function based on a difference between the output image and the input imageand based on a difference between an observed compression rate and a target compressionrate of the input image, the target compression rate sampled from a second range of targetcompression rates;updating the parameters of the third neural network and the fourth neural network basedon the evaluated function;repeating, using a set of input images, the receiving, encoding, decoding, evaluating andupdating steps a first number of times;changing the second range of target compression rates;repeating, using the set of input images, the receiving, encoding, decoding, evaluatingand updating steps a second number of times to produce a third trained neural network and afourth trained neural network;using the first and second trained neural networks to perform lossy video compression ata first level of an adaptive bitrate ladder; andusing the third and fourth trained neural networks for performing lossy video compressionat a second level of the adaptive bitrate ladder.

42. A method for lossy image or video encoding, transmission and decoding, the methodcomprising the steps of:receiving an input image at a first computer system;encoding the input image using a first trained neural network to produce a latentrepresentation; transmitting the latent representation to a second computer system;decoding the latent representation using a second trained neural network to produce anoutput image, wherein the output image is an approximation of the input image;wherein the first neural network and the second neural network are trained according toany of the methods of claims 30 to 41.

43. A method for lossy image or video encoding and transmission, the method comprising thesteps of:receiving an input image at a first computer system;encoding the input image using a first trained neural network to produce a latentrepresentation; andtransmitting the latent representation;wherein the first neural network is trained according to any of the methods of claims 30to 41.

44. A method for lossy image or video receipt and decoding, the method comprising the stepsof: receiving a latent representation transmitted by a first computer system at a secondcomputer system, the latent representation corresponding to an input image; anddecoding the latent representation using a second trained neural network to produce anoutput image, wherein the output image is an approximation of the input image;wherein the second neural network is trained according to any of the methods of claims30 to 41.

45. A data processing system configured to perform the method of any one of claims 30 to 44.

46. A data processing apparatus configured to perform the method of claims 30 to 44.

47. A computer program comprising instructions which, when the program is executed by acomputer, cause the computer to carry out the method of claims 30 to 44.

48. A computer-readable storage medium comprising instructions which, when executed by acomputer, cause the computer to carry out the method of claims 30 to 44.

Images