Method and data processing system for lossy image or video encoding, transmission and decoding

Abstract

A method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: using a first I-frame module to produce an output I-frame from a first input image, the I-frame module comprising one or more neural networks; using a P- and / or B-frame module to produce an output P- or B-frame from a second input image and the output I-frame, the P- and / or B-frame module comprising one or more neural networks; evaluating a function based on a difference between the second input image and the output P- or B-frame; updating the parameters of the one or more neural networks of the P- and / or B-frame module based on the evaluated function; swapping the first I-frame module with a second I-frame module.

Description

[0001] Method and data processing system for lossy image or videoencoding, transmission and decodingBACKGROUNDThis invention relates to a method and system for lossy image or video encoding, transmissionand decoding, a method, apparatus, computer program and computer readable storage mediumfor lossy image or video encoding and transmission, and a method, apparatus, computerprogram and computer readable storage medium for lossy image or video receipt and decoding.There is increasing demand from users of communications networks for images and videocontent. Demand is increasing not just for the number of images viewed, and for the playingtime of video; demand is also increasing for higher resolution content. This places increasingdemand on communications networks and increases their energy use because of the largeramount of data being transmitted.To reduce the impact of these issues, image and video content is compressed for transmissionacross the network. The compression of image and video content can be lossless or lossycompression. In lossless compression, the image or video is compressed such that all of theoriginal information in the content can be recovered on decompression. However, when usinglossless compression there is a limit to the reduction in data quantity that can be achieved. Inlossy compression, some information is lost from the image or video during the compressionprocess. Known compression techniques attempt to minimise the apparent loss of informationby the removal of information that results in changes to the decompressed image or video thatis not particularly noticeable to the human visual system. JPEG, JPEG2000, AVC, HEVC andAVI are examples of compression processes for image and / or video files.In general terms, known lossy image compression techniques use the spatial correlationsbetween pixels in images to remove redundant information during compression. For example,in an image of a blue sky, if a given pixel is blue, there is a high likelihood that the neighbouringpixels, and their neighbouring pixels, and so on, are also blue. There is accordingly no need toretain all the raw pixel data. Instead, we can retain only a subset of the pixels which take upfewer bits and infer the pixel values of the other pixels using information derived from spatialcorrelations.A similar approach is applied in known lossy video compression techniques. That is, spatialcorrelations between pixels allow the removal of redundant information during compression.However, in video compression, there is further information redundancy in the form of temporalcorrelations. For example, in a video of an aircraft flying across a blue-sky background, mostof the pixels of the blue sky do not change at all between frames of the video. The mostof the blue sky pixel data for the frame at position t = 0 in the video is identical to that atposition t = 10. Storing this identical, temporally correlated, information is inefficient. Instead,only the blue sky pixel data for a subset of the frames is stored and the rest are inferred frominformation derived from temporal correlations.In the realm of lossy video compression in particular, the removal of redundant temporallycorrelated information in a video sequence is known inter-frame redundancy.One technique using inter-frame redundancy that is widely used in standard video compressionalgorithms involves the categorization of video frames into three types: I-frames, P-frames, andB-frames. Each frame type carries distinct properties concerning their encoding and decodingprocess, playing different roles in achieving high compression ratios while maintainingacceptable visual quality.I-frames, or intra-coded frames, serve as the foundation of the video sequence. These framesare self-contained, each one encoding a complete image without reference to any other frame.In terms of compression, I-frames are least compressed among all frame types, thus carryingthe most data. However, their independence provides several benefits, including being thestarting point for decompression and enabling random access, crucial for functionalities likefast-forwarding or rewinding the video.P-frames, or predictive frames, utilize temporal redundancy in video sequences to achievegreater compression. Instead of encoding an entire image like an I-frame, a P-frame representsthe difference between itself and the closest preceding I- or P-frame. The process, known asmotion compensation, identifies and encodes only the changes that have occurred, therebysignificantly reducing the amount of data transmitted. Nonetheless, P-frames are dependent onprevious frames for decoding. Consequently, any error during the encoding or transmissionprocess may propagate to subsequent frames, impacting the overall video quality.B-frames, or bidirectionally predictive frames, represent the highest level of compression.Unlike P-frames, B-frames use both the preceding and following frames as references in theirencoding process. By predicting motion both forwards and backwards in time, B-framesencode only the differences that cannot be accurately anticipated from the previous and nextframes, leading to substantial data reduction. Although this bidirectional prediction makesB-frames more complex to generate and decode, it does not propagate decoding errors sincethey are not used as references for other frames. Artificial intelligence (AI) based compressiontechniques achieve compression and decompression of images and videos through the use oftrained neural networks in the compression and decompression process. Typically, duringtraining of the neutral networks, the difference between the original image and video and thecompressed and decompressed image and video is analyzed and the parameters of the neuralnetworks are modified to reduce this difference while minimizing the data required to transmitthe content. However, AI based compression methods may achieve poor compression resultsin terms of the appearance of the compressed image or video or the amount of informationrequired to be transmitted.An example of an AI based image compression process comprising a hyper-network is describedin Ballé, Johannes, et al. “Variational image compression with a scale hyperprior.” arXivpreprint arXiv:1802.01436 (2018), which is hereby incorporated by reference.An example of an AI based video compression approach is shown in Agustsson, E., Minnen, D.,Johnston, N., Balle, J., Hwang, S. J., and Toderici, G. (2020), Scale-space flow for end-to-endoptimized video compression. In Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition (pp. 8503-8512), which is hereby incorporated by reference.A further example of an AI based video compression approach is shown in Mentzer, F.,Agustsson, E., Ballé, J., Minnen, D., Johnston, N., and Toderici, G. (2022, November). Neuralvideo compression using gans for detail synthesis and propagation. In Computer Vision–ECCV2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, PartXXVI (pp. 562-578), which is hereby incorporated by reference.SUMMARYAccording to an aspect of the present disclsoure, there is provided a method of training one ormore neural networks, the one or more neural networks being for use in lossy image or videoencoding, transmission and decoding, the method comprising the steps of:using a first I-frame module to produce an output I-frame from a first input image, theI-frame module comprising one or more neural networks;using a P- and / or B-frame module to produce an output P- or B-frame from a secondinput image and the output I-frame, the P- and / or B-frame module comprising one or moreneural networks;evaluating a function based on a difference between the second input image and theoutput P- or B- frame;updating the parameters of the one or more neural networks of the P- and / or B-framemodule based on the evaluated function;repeating the above steps using a set of input images to produce trained neural networksof the P- and / or B-frame module; andswapping the first I-frame module with a second I-frame module.Optionally, the method comprises using the second I-frame module and the P- and / or B-framemodule to encode and decode a plurality of images.Optionally, the second I-frame module is configured to produce an output I-frame that is abetter approximation of the first input image than an I-frame produced by the first I-framemodule.Optionally, a quality of the approximation is defined by a difference score.Optionally, the difference score comprises a mean squared error, and wherein the betterapproximation is defined by a smaller mean squared error between an input image and anoutput I-frame.Optionally, the function comprises a rate term, a distortion term, and wherein at least one ofthe rate term or distortion term is weighted by a Lagrange multiplier.Optionally, the Lagrange multiplier comprises a learned Lagrange multiplier.Optionally, the set of input images comprises groups of pictures (GOPs) of fewer than 20frames.Optionally, the method comprises swapping the second I-frame module with a third I-framemodule.Optionally, the method comprises selecting the second I-frame module or the third I-framemodule based on a property of the first input image or a third input image.Optionally, said selecting comprises applying a classifier to the first input image and / or thethird input image to produce a classification output, and wherein said selecting is based on theclassification output.Optionally, the first input image comprises an input image of a first image type, and whereinthe third input image comprises an input image of a second type.Optionally, the first image type is a high-motion image type, and wherein the second imagetype is a low-motion image type.According to a further aspecf of the present disclosure, there is provided a method of trainingone or more neural networks, the one or more neural networks being for use in lossy image orvideo encoding, transmission and decoding, the method comprising the steps of:using a first I-frame module to produce an output I-frame from a first input image, theI-frame module comprising one or more neural networks;using a P- and / or B-frame module to produce an output P- or B-frame from a secondinput image and the output I-frame, the P- and / or B-frame module comprising one or moreneural networks;evaluating a function based on a difference between the first input image and the firstoutput I-frame and / or a difference between the second input image and the output P- or B-frame;updating the parameters of the one or more neural networks of the P- and / or B-framemodule based on the evaluated function;updating the parameters of the one or more neural networks of the I-frame module basedon the evaluated functionrepeating the above steps using a set of input images to produce trained neural networksof the P- and / or B-frame module and trained neural networks of the I-frame module.Optionally, the method comprises freezing the parameters of the one or more neural neuralnetworks of the P- and / or B-frame module after a number of said repeated steps.Optionally, the method comprises modifying the function after said freezing.Optionally, the function comprises a rate term, a distortion term, wherein at least one of therate term or distortion term is weighted by a Lagrange multiplier, and wherein modifying thefunction comprises modifying the Lagrange multiplier.Optionally, the method comprises modifying the Lagrange multiplier to increase the relativeweight of the distortion term relative to other terms of the function.According to a further aspect, there is provided a data processing apparatus configured toperform any of the above methods.According to a further aspect, there is provided a computer program comprising instructionswhich, when the program is executed by a computer, cause the computer to carry out any ofthe above methods.According to a further aspect, there is provided a computer-readable storage medium comprisinginstructions which, when executed by a computer, cause the computer to carry out any of theabove methods.According to a further aspect, there is provided, a method of lossy image or video encodingand decoding, the method comprising the steps of:receiving a first input image at a first computer system;using a first I-frame module to produce a first output I-frame from the first input image,the I-frame module comprising one or more neural networks;receiving a second input image at the first computer system;using a P- and / or B-frame module to produce a first output P- or B-frame using the firstoutput I-frame and the second input image, the P- and / or B-frame module comprising one ormore neural networks;swapping the first I-frame module with a second I-frame module;receiving a third input image at the first computer system;using the second I-frame module to produce a second output I-frame from the third inputimage, the I-frame module comprising one or more neural networks; andreceiving a fourth input image at the first computer system; andusing the P- and / or B-frame module to produce a second output P- or B-frame using thesecond I-frame and the fourth input image.Optionally, the first input image comprises an input image of a first image type, and whereinthe third input image comprises an input image of a second type.Optionally, the first image type is a high-motion image type, and wherein the second imagetype is a low-motion image type.Optionally, the method comprises selecting the first I-frame module and / or the second I-framemodule from a set of I-frame modules.Optionally, said selecting comprises applying a classifier to the first input image and / or thesecond input image to produce a classification result, and wherein said selecting is based onthe classification result.Optionally, said selecting comprises receiving meta data associated with the first input imageand / or the second input image, and wherein said selecting is based on the meta data.According to a further aspect, there is provided a data processing apparatus configured toperform any of the above methods.According to a further aspect, there is provided a computer program comprising instructionswhich, when the program is executed by a computer, cause the computer to carry out any ofthe above methods.According to a further aspect, there is provided a computer-readable storage medium comprisinginstructions which, when executed by a computer, cause the computer to carry out any of theabove methods.According to a further aspect of the present disclosure, there is provided a method of trainingone or more neural networks, the one or more neural networks being for use in lossy image orvideo encoding and decoding, the method comprising the steps of:receiving a first input image at a first computer system;encoding the first input image using a first neural network to produce a latent representa-tion; decoding the latent representation using a second neural network to produce a first outputimage, wherein the first output image is an approximation of the first input image;receiving a second input image at the first computer system;with a third neural network, producing a latent representation of optical flow informationand / or residual information, the optical flow information and / or residual information beingindicative of a difference between the first input image and the second input image;with a fourth neural network, producing a second output image using the optical flowinformation and / or residual information, wherein the second output image is an approximationof the second input image;evaluating a function based on a difference between the second input image and thesecond output image;updating the parameters of the third neural network and the fourth neural network basedon the evaluated function;repeating the above steps using a set of input images to produce a trained third neuralnetwork and a trained fourth neural network; andreplacing the first neural network and second neural network with a fifth neural networkand a sixth neural network.Optionally, the fifth neural network and the sixth neural network comprise pre-trained neuralnetworks configured to produce a better approximation of the first input image than the firstneural network and the second neural network.Optionally, a quality of the approximation is defined by a difference score.Optionally, the difference score comprises a mean squared error, and wherein the betterapproximation is defined by a smaller mean squared error between the first input image andthe first output image.According to a further aspect of the present disclosure, there is provided a method of trainingone or more neural networks, the one or more neural networks being for use in lossy image orvideo encoding and decoding, the method comprising the steps of:receiving a first input image at a first computer system;encoding the first input image using a first neural network to produce a latent representa-tion; decoding the latent representation using a second neural network to produce a first outputimage, wherein the first output image is an approximation of the first input image;receiving a second input image at the first computer system;with a third neural network, producing a latent representation of optical flow informationand / or residual information, the optical flow information and / or residual information beingindicative of a difference between the first input image and the second input image;with a fourth neural network, producing a second output image using the optical flowinformation and / or residual information, wherein the second output image is an approximationof the second input image;evaluating a function based on a difference between the second input image and thesecond output image, and / or the first input image and the first output image;updating the parameters of the first neural network, the second neural network, the thirdneural network and the fourth neural network based on the evaluated function;repeating the above steps using a set of input images for a first number of steps;modifying the function by changing a weighting of a rate term relative to a distortionterm of the function; andrepeating the above steps for a second number of steps.Optionally, the method comprises freezing the parameters of the third neural network and thefourth neural network after repeating the steps for the first number of times.Optionally, changing the weighting of the rate term relative to the distortion term comprisesincreasing the weighting of the distortion term relative to the rate term to specify a relativelylower distortion loss training objective.According to a further aspect, there is provided a data processing apparatus configured toperform any of the above methods.According to a further aspect, there is provided a computer program comprising instructionswhich, when the program is executed by a computer, cause the computer to carry out any ofthe above methods.According to a further aspect, there is provided a a computer-readable storage mediumcomprising instructions which, when executed by a computer, cause the computer to carry outany of the above methods.According to a further aspect, there is provided a method of training one or more neuralnetworks, the one or more neural networks being for use in lossy image or video encoding,transmission and decoding, the method comprising the steps of:using a P- and / or B-frame module to produce an output P- or B-frame from a firstimage and a second image, the P- and / or B-frame module having a first distortion target andcomprising one or more neural networks;evaluating a first function based on a difference between the second image and the outputP- or B- frame;updating the parameters of the one or more neural networks of the P- and / or B-framemodule based on the evaluated function, and updating the first distortion target;repeating the above steps using a set of input images to produce trained neural networksof the P- and / or B-frame module and a final first distortion target;using an I-frame module to produce an output I-frame from a third image, the I-framemodule having a second distortion target and comprising one or more neural networks, whereinthe second distortion target is based on the final first distortion target;evaluating a second function based on a difference between the third image and theoutput I-frame; andupdating the parameters of the one or more neural networks of the I-frame module basedon the evaluated function to produce trained neural networks of the I-frame module.Optionally, the method comprises setting a first bit rate target, and wherein said updating thefirst distortion target is based on the the first bit rate target.Optionally, said updating the first distortion target comprises evaluating a bit rate achievedwhen producing the output P- or B-frame and comparing said bit rate against the first bit ratetarget.Optionally, said updating the first distortion target comprises increasing the first distortiontarget when the bit rate achieved is higher than the first bit rate target, or decreasing the firstdistortion target when the bit rate achieved is lower than the first bit rate target.Optionally, an amount and direction of an update to the first distortion target when performingsaid updating is based on how much higher or lower the the bit rate achieved is to the first bitrate target.Optionally, wherein said updating the first distortion target comprises running a proportional,integral and / or derivative controller with a value of the achieved bit rate as a process variable,and the first distortion target as the control variable,Optionally, the method comprises using a placeholder I-frame module to to produce thefirst image, the placeholder I-frame module having a different distortion target to the seconddistortion target.Optionally, the method comprises discarding the placeholder I-frame module and combiningthe trained I-frame module with the trained P- and / or B-frame module to produce a videocompression pipeline.Optionally, the second distortion target is a predetermined fraction or multiple of the final firstdistortion target.Optionally, the second distortion target is substantially 1 / 3 of the final first distortion target.According to a further aspect, there is provided a method for lossy image or video encoding,transmission and decoding, the method comprising the steps of:receiving an input image at a first computer system;encoding the input image using a first trained neural network to produce a latentrepresentation; transmitting the latent representation to a second computer system;decoding the latent representation using a second trained neural network to produce anoutput image, wherein the output image is an approximation of the input image;wherein the first neural network and the second neural network are trained according toany of the above methods.According to a further aspect, there is provided a method for lossy image or video encodingand transmission, the method comprising the steps of:receiving an input image at a first computer system;encoding the input image using a first trained neural network to produce a latentrepresentation; andtransmitting the latent representation;wherein the first neural network is trained according to any of the above methods.According to a further aspect, there is provided a method for lossy image or video receipt anddecoding, the method comprising the steps of:receiving a latent representation transmitted by a first computer system at a secondcomputer system, the latent representation corresponding to an input image; anddecoding the latent representation using a second trained neural network to produce anoutput image, wherein the output image is an approximation of the input image;wherein the second neural network is trained according to any of the above methods.According to a further aspect, there is provided a data processing system configured to performany of the above methods.According to a further aspect, there is provided a data processing apparatus configured toperform any of the above methods.According to a further aspect, there is provided a computer program comprising instructionswhich, when the program is executed by a computer, cause the computer to carry out any ofthe above methods.According to a further aspect, there is provided a computer-readable storage medium comprisinginstructions which, when executed by a computer, cause the computer to carry out any of theabove methods.BRIEF DESCRIPTION OF THE DRAWINGSAspects of the invention will now be described by way of examples, with reference to thefollowing figures in which:Figure 1 illustrates an example of an image or video compression, transmission and decom-pression pipeline.Figure 2 illustrates a further example of an image or video compression, transmission anddecompression pipeline including a hyper-network.Figure 3 illustrates an example of a video compression, transmission and decompressionpipeline.Figure 4 illustrates an example of a video compression, transmission and decompressionsystem.Figure 5a illustrates an example of a video compression, transmission and decompressionpipeline.Figure 5b illustrates an example of a video compression, transmission and decompressionpipeline.Figure 5c illustrates an example of a video compression, transmission and decompressionpipeline.Figure 6 illustrates a classifier for selecting a module of a video compression, transmissionand decompression pipeline.Figure 7 illustrates an example of a video compression, transmission and decompressionpipeline.Figure 8 illustrates an example training flow chart of a I-, P- and / or B- frame modules of avideo compression, transmission and decompression pipeline.DETAILED DESCRIPTION OF THE DRAWINGSCompression processes may be applied to any form of information to reduce the amountof data, or file size, required to store that information. Image and video information is anexample of information that may be compressed. The file size required to store the information,particularly during a compression process when referring to the compressed file, may bereferred to as the rate. In general, compression can be lossless or lossy. In both forms ofcompression, the file size is reduced. However, in lossless compression, no information is lostwhen the information is compressed and subsequently decompressed. This means that theoriginal file storing the information is fully reconstructed during the decompression process.In contrast to this, in lossy compression information may be lost in the compression anddecompression process and the reconstructed file may differ from the original file. Image andvideo files containing image and video data are common targets for compression.In a compression process involving an image, the input image may be represented as ^^. Thedata representing the image may be stored in a tensor of dimensions ^^ × ^^ × ^^, where ^^represents the height of the image, ^^ represents the width of the image and ^^ represents thenumber of channels of the image. Each ^^ × ^^ data point of the image represents a pixel valueof the image at the corresponding location. Each channel ^^ of the image represents a differentcomponent of the image for each pixel which are combined when the image file is displayed bya device. For example, an image file may have 3 channels with the channels representing thered, green and blue component of the image respectively. In this case, the image informationis stored in the RGB colour space, which may also be referred to as a model or a format.Other examples of colour spaces or formats include the CMKY and the YCbCr colour models.However, the channels of an image file are not limited to storing colour information and otherinformation may be represented in the channels. As a video may be considered a series ofimages in sequence, any compression process that may be applied to an image may also beapplied to a video. Each image making up a video may be referred to as a frame of the video.The output image may differ from the input image and may be represented by ^^. The differencebetween the input image and the output image may be referred to as distortion or a differencein image quality. The distortion can be measured using any distortion function which receivesthe input image and the output image and provides an output which represents the differencebetween input image and the output image in a numerical way. An example of such a methodis using the mean square error (MSE) between the pixels of the input image and the outputimage, but there are many other ways of measuring distortion, as will be known to the personskilled in the art. The distortion function may comprise a trained neural network.Typically, the rate and distortion of a lossy compression process are related. An increase inthe rate may result in a decrease in the distortion, and a decrease in the rate may result in anincrease in the distortion. Changes to the distortion may affect the rate in a correspondingmanner. A relation between these quantities for a given compression technique may be definedby a rate-distortion equation.AI based compression processes may involve the use of neural networks. A neural network isan operation that can be performed on an input to produce an output. A neural network maybe made up of a plurality of layers. The first layer of the network receives the input. One ormore operations may be performed on the input by the layer to produce an output of the firstlayer. The output of the first layer is then passed to the next layer of the network which mayperform one or more operations in a similar way. The output of the final layer is the output ofthe neural network.Each layer of the neural network may be divided into nodes. Each node may receive at leastpart of the input from the previous layer and provide an output to one or more nodes in asubsequent layer. Each node of a layer may perform the one or more operations of the layer onat least part of the input to the layer. For example, a node may receive an input from one ormore nodes of the previous layer. The one or more operations may include a convolution, aweight, a bias and an activation function. Convolution operations are used in convolutionalneural networks. When a convolution operation is present, the convolution may be performedacross the entire input to a layer. Alternatively, the convolution may be performed on at leastpart of the input to the layer.Each of the one or more operations is defined by one or more parameters that are associatedwith each operation. For example, the weight operation may be defined by a weight matrixdefining the weight to be applied to each input from each node in the previous layer to eachnode in the present layer. In this example, each of the values in the weight matrix is a parameterof the neural network. The convolution may be defined by a convolution matrix, also knownas a kernel. In this example, one or more of the values in the convolution matrix may be aparameter of the neural network. The activation function may also be defined by values whichmay be parameters of the neural network. The parameters of the network may be varied duringtraining of the network.Other features of the neural network may be predetermined and therefore not varied duringtraining of the network. For example, the number of layers of the network, the number ofnodes of the network, the one or more operations performed in each layer and the connectionsbetween the layers may be predetermined and therefore fixed before the training process takesplace. These features that are predetermined may be referred to as the hyperparameters of thenetwork. These features are sometimes referred to as the architecture of the network.To train the neural network, a training set of inputs may be used for which the expected output,sometimes referred to as the ground truth, is known. The initial parameters of the neuralnetwork are randomized and the first training input is provided to the network. The output ofthe network is compared to the expected output, and based on a difference between the outputand the expected output the parameters of the network are varied such that the differencebetween the output of the network and the expected output is reduced. This process is thenrepeated for a plurality of training inputs to train the network. The difference between theoutput of the network and the expected output may be defined by a loss function. The result ofthe loss function may be calculated using the difference between the output of the networkand the expected output to determine the gradient of the loss function. Back-propagation ofthe gradient descent of the loss function may be used to update the parameters of the neuralnetwork using the gradients ^^^^ / ^^^^ of the loss function. A plurality of neural networks in asystem may be trained simultaneously through back-propagation of the gradient of the lossfunction to each network.In the context of image or video compression, this type of system, where simultaneous trainingwith back-propagation through each element or the whole network architecture may be referredto as end-to-end, learned image or video compression. Unlike in traditional compressionalgorithms that use primarily handcrafted, manually constructed steps, an end-to-end learnedsystem learns itself during training what combination of parameters best achieves the goal ofminimising the loss function. This approach is advantageous compared to systems that are notend-to-end learned because an end-to-end system has a greater flexibility to learn weights andparameters that might be counter-intuitive to someone handcrafting features.It will be appreciated that the term "training" or "learning" as used herein means the processof optimizing an artificial intelligence or machine learning model, based on a given set of data.This involves iteratively adjusting the parameters of the model to minimize the discrepancybetween the model’s predictions and the actual data, represented by the above-describedrate-distortion loss function.The training process may comprise multiple epochs. An epoch refers to one complete passof the entire training dataset through the machine learning algorithm. During an epoch, themodel’s parameters are updated in an effort to minimize the loss function. It is envisaged thatmultiple epochs may be used to train a model, with the exact number depending on variousfactors including the complexity of the model and the diversity of the training data.Within each epoch, the training data may be divided into smaller subsets known as batches.The size of a batch, referred to as the batch size, may influence the training process. A smallerbatch size can lead to more frequent updates to the model’s parameters, potentially leading tofaster convergence to the optimal solution, but at the cost of increased computational resources.Conversely, a larger batch size involves fewer updates, which can be more computationallyefficient but might converge slower or even fail to converge to the optimal solution.The learnable parameters are updated by a specified amount each time, determined by thelearning rate. The learning rate is a hyperparameter that decides how much the parametersare adjusted during the training process. A smaller learning rate implies smaller steps in theparameter space and a potentially more accurate solution, but it may require more epochs toreach that solution. On the other hand, a larger learning rate can expedite the training processbut may risk overshooting the optimal solution or causing the training process to diverge.The training described herein may involve use of a validation set, which is a portion of thedata not used in the initial training, which is used to evaluate the model’s performance and toprevent overfitting. Overfitting occurs when a model learns the training data too well, to thepoint that it fails to generalize to unseen data. Regularization techniques, such as dropout orL1 / L2 regularization, can also be used to mitigate overfitting.It will be appreciated that training a machine learning model is an iterative process thatmay comprise selection and tuning of various parameters and hyperparameters. As will beappreciated, the specific details, such as hyper parameters and so on, of the training processmay vary and it is envisaged that producing a trained model in this way may achieved in anumber of different ways with different epochs, batch sizes, learning rates, regularisations,and so on, the details of which are not essential to enabling the advantages and effects of thepresent disclosure, except where stated otherwise. The point at which an “untrained” neuralnetwork is considered be “trained” is envisaged to be case specific and depend on, for example,on a number of epochs, a plateauing of any further learning, or some other metric and is notconsidered to be essential in achieving the advantages described herein.More details of an end-to-end, learned compression process will now be described. It will beappreciated that in some cases, end-to-end, learned compression processes may be combinedwith one or more components that are handcrafted or trained separately.In the case of AI based image or video compression, the loss function may be defined by therate distortion equation. The rate distortion equation may be represented by ^^^^^^^^ = ^^ + ^^ ∗ ^^,where ^^ is the distortion function, ^^ is a weighting factor, and ^^ is the rate loss. ^^ may bereferred to as a lagrange multiplier. The langrange multiplier provides as weight for a particularterm of the loss function in relation to each other term and can be used to control which termsof the loss function are favoured when training the network.In the case of AI based image or video compression, a training set of input images maybe used. An example training set of input images is the KODAK image set (for exampleat www.cs.albany.edu / xypan / research / snr / Kodak.html). An example training set of inputimages is the IMAX image set. An example training set of input images is the Imagenetdataset (for example at www.image-net.org / download). An example training set of inputimages is the CLIC Training Dataset P (“professional”) and M (“mobile”) (for example athttp: / / challenge.compression.cc / tasks / ).An example of an AI based compression, transmission and decompression process 100 isshown in Figure 1. As a first step in the AI based compression process, an input image 5 isprovided. The input image 5 is provided to a trained neural network 110 characterized by afunction ^^^^ acting as an encoder. The encoder neural network 110 produces an output basedon the input image. This output is referred to as a latent representation of the input image 5. Ina second step, the latent representation is quantised in a quantisation process 140 characterisedby the operation ^^, resulting in a quantized latent. The quantisation process transforms thecontinuous latent representation into a discrete quantized latent. An example of a quantizationprocess is a rounding function.In a third step, the quantized latent is entropy encoded in an entropy encoding process 150 toproduce a bitstream 130. The entropy encoding process may be for example, range or arithmeticencoding. In a fourth step, the bitstream 130 may be transmitted across a communicationnetwork.In a fifth step, the bitstream is entropy decoded in an entropy decoding process 160. Thequantized latent is provided to another trained neural network 120 characterized by a function^^^^ acting as a decoder, which decodes the quantized latent. The trained neural network 120produces an output based on the quantized latent. The output may be the output image of theAI based compression process 100. The encoder-decoder system may be referred to as anautoencoder.Entropy encoding processes such as range or arithmetic encoding are typically able to losslesslycompress given input data up to close to the fundamental entropy limit of that data, as determinedby the total entropy of the distribution of that data. Accordingly, one way in which end-to-end,learned compression can minimise the rate loss term of the rate-distortion loss function andthereby increase compression effectiveness is to learn autoencoder parameter values thatproduce low entropy latent representation distributions. Producing latent representationsdistributed with as low an entropy as possible allows entropy encoding to compress the latentdistributions as close to or to the fundamental entropy limit for that distribution. The lowerthe entropy of the distribution, the more entropy encoding can losslessly compress it and thelower the amount of data in the corresponding bitstream. In some cases where the latentrepresentation is distributed according to a gaussian or Laplacian distribution, this learningmay comprise learning optimal location and scale parameters of the gaussian or Laplaciandistributions, in other cases, it allows the learning of more flexible latent representationdistributions which can further help to achieve the minimising of the rate-distortion lossfunction in ways that are not intuitive or possible to do with handcrafted features. Examples ofthese and other advantages are described in WO2021 / 220008A1, which is incorporated in itsentirety by reference.Something which is closely linked to the entropy encoding of the latent distribution and whichaccordingly also has an effect on the effectiveness of compression of end-to-end learnedapproaches is the quantisation step. During inference, a rounding function may be used toquantise a latent representation distribution into bins of given sizes, a rounding function isnot differentiable everywhere. Rather, a rounding function is effectively one or more stepfunctions whose gradient is either zero (at the top of the steps) or infinity (at the boundarybetween steps). Back propagating a gradient of a loss function through a rounding functionis challenging. Instead, during training, quantisation by rounding function is replaced byone or more other approaches. For example, the functions of a noise quantisation model aredifferentiable everywhere and accordingly do allow backpropagation of the gradient of theloss function through the quantisation parts of the end-to-end, learned system. Alternatively, astraight-through estimator (STE) quantisation model or one other quantisation models may beused. It is also envisaged that different quantisation models may be used for during evaluationof different term of the loss function. For example, noise quantisation may used to evaluate therate or entropy loss term of the rate-distortion loss function while STE quantisation may beused to evaluate the distortion term.In a similar manner to how learning parameters top produce certain distributions of the latentrepresentation facilitates achieving better rate loss term minimisation, end-to-end learning ofthe quantisation process achieves a similar effect. That is, learnable quantisation parametersprovide the architecture with a further degree of freedom to achieve the goal of minimising theloss function. For example, parameters corresponding to quantisation bin sizes may be learnedwhich is likely to result in an improved rate-distortion loss outcome compared to approachesusing hand-crafted quantisation bin sizes.Further, as the rate-distortion loss function constantly has to balance a rate loss term against adistortion loss term, it has been found that the more degrees of freedom the system has duringtraining, the better the architecture is at achieving optimal rate and distortion trade off.The system described above may be distributed across multiple locations and / or devices. Forexample, the encoder 110 may be located on a device such as a laptop computer, desktopcomputer, smart phone or server. The decoder 120 may be located on a separate device whichmay be referred to as a recipient device. The system used to encode, transmit and decode theinput image 5 to obtain the output image 6 may be referred to as a compression pipeline.The AI based compression process may further comprise a hyper-network 105 for thetransmission of meta-information that improves the compression process. The hyper-network105 comprises a trained neural network 115 acting as a hyper-encoder ^^ ℎ^^and a trained neuralnetwork 125 acting as a hyper-decoder ^^ℎ^^. An example of such a system is shown in Figure 2.Components of the system not further discussed may be assumed to be the same as discussedabove. The neural network 115 acting as a hyper-decoder receives the latent that is the output ofthe encoder 110. The hyper-encoder 115 produces an output based on the latent representationthat may be referred to as a hyper-latent representation. The hyper-latent is then quantizedin a quantization process 145 characterised by ^^ℎ to produce a quantized hyper-latent. Thequantization process 145 characterised by ^^ℎ may be the same as the quantisation process 140characterised by ^^ discussed above.In a similar manner as discussed above for the quantized latent, the quantized hyper-latent isthen entropy encoded in an entropy encoding process 155 to produce a bitstream 135. Thebitstream 135 may be entropy decoded in an entropy decoding process 165 to retrieve thequantized hyper-latent. The quantized hyper-latent is then used as an input to trained neuralnetwork 125 acting as a hyper-decoder. However, in contrast to the compression pipeline 100,the output of the hyper-decoder may not be an approximation of the input to the hyper-decoder115. Instead, the output of the hyper-decoder is used to provide parameters for use in theentropy encoding process 150 and entropy decoding process 160 in the main compressionprocess 100. For example, the output of the hyper-decoder 125 can include one or more ofthe mean, standard deviation, variance or any other parameter used to describe a probabilitymodel for the entropy encoding process 150 and entropy decoding process 160 of the latentrepresentation. In the example shown in Figure 2, only a single entropy decoding process 165and hyper-decoder 125 is shown for simplicity. However, in practice, as the decompressionprocess usually takes place on a separate device, duplicates of these processes will be presenton the device used for encoding to provide the parameters to be used in the entropy encodingprocess 150.Further transformations may be applied to at least one of the latent and the hyper-latent at anystage in the AI based compression process 100. For example, at least one of the latent and thehyper latent may be converted to a residual value before the entropy encoding process 150,155is performed. The residual value may be determined by subtracting the mean value of thedistribution of latents or hyper-latents from each latent or hyper latent. The residual valuesmay also be normalised.To perform training of the AI based compression process described above, a training set ofinput images may be used as described above. During the training process, the parameters ofboth the encoder 110 and the decoder 120 may be simultaneously updated in each trainingstep. If a hyper-network 105 is also present, the parameters of both the hyper-encoder 115 andthe hyper-decoder 125 may additionally be simultaneously updated in each training step.The training process may further include a generative adversarial network (GAN). Whenapplied to an AI based compression process, in addition to the compression pipeline describedabove, an additional neutral network acting as a discriminator is included in the system. Thediscriminator receives an input and outputs a score based on the input providing an indicationof whether the discriminator considers the input to be ground truth or fake. For example, theindicator may be a score, with a high score associated with a ground truth input and a lowscore associated with a fake input. For training of a discriminator, a loss function is used thatmaximizes the difference in the output indication between an input ground truth and input fake.When a GAN is incorporated into the training of the compression process, the output image 6may be provided to the discriminator. The output of the discriminator may then be used in theloss function of the compression process as a measure of the distortion of the compressionprocess. Alternatively, the discriminator may receive both the input image 5 and the outputimage 6 and the difference in output indication may then be used in the loss function of thecompression process as a measure of the distortion of the compression process. Training ofthe neural network acting as a discriminator and the other neutral networks in the compressionprocess may be performed simultaneously. During use of the trained compression pipelinefor the compression and transmission of images or video, the discriminator neural network isremoved from the system and the output of the compression pipeline is the output image 6.Incorporation of a GAN into the training process may cause the decoder 120 to performhallucination. Hallucination is the process of adding information in the output image 6 thatwas not present in the input image 5. In an example, hallucination may add fine detail tothe output image 6 that was not present in the input image 5 or received by the decoder 120.The hallucination performed may be based on information in the quantized latent received bydecoder 120.Details of a video compression process will now be described. As discussed above, a video ismade up of a series of images arranged in sequential order. AI based compression process100 described above may be applied multiple times to perform compression, transmissionand decompression of a video. For example, each frame of the video may be compressed,transmitted and decompressed individually. The received frames may then be grouped toobtain the original video.The frames in a video may be labelled based on the information from other frames that is usedto decode the frame in a video compression, transmission and decompression process. Asdescribed above, frames which are decoded using no information from other frames may bereferred to as I-frames. Frames which are decoded using information from past frames may bereferred to as P-frames. Frames which are decoded using information from past frames andfuture frames may be referred to as B-frames. Frames may not be encoded and / or decoded inthe order that they appear in the video. For example, a frame at a later time step in the videomay be decoded before a frame at an earlier time.The images represented by each frame of a video may be related. For example, a number offrames in a video may show the same scene. In this case, a number of different parts of thescene may be shown in more than one of the frames. For example, objects or people in a scenemay be shown in more than one of the frames. The background of the scene may also beshown in more than one of the frames. If an object or the perspective is in motion in the video,the position of the object or background in one frame may change relative to the position ofthe object or background in another frame. The transformation of a part of the image froma first position in a first frame to a second position in a second frame may be referred to asflow, warping or motion compensation. The flow may be represented by a vector. One or moreflows that represent the transformation of at least part of one frame to another frame may bereferred to as a flow map.An example AI based video compression, transmission, and decompression process 200 isshown in Figure 3. The process 200 shown in Figure 3 is divided into an I-frame part 201for decompressing I-frames, and a P-frame part 202 for decompressing P-frames. It will beunderstood that these divisions into different parts are arbitrary and the process 200 may bealso be considered as a single, end-to-end pipeline.As described above, I-frames do not rely on information from other frames so the I-frame part201 corresponds to the compression, transmission, and decompression process illustrated inFigures 1 or 2. The specific details will not be repeated here but, in summary, an input image^^0 is passed into an encoder neural network 203 producing a latent representation which isquantised and entropy encoded into a bitstream 204. The subscript 0 in ^^0 indicates the inputimage corresponds to a frame of a video stream at position t = 0. This may be the first frame ofan entire video stream or the first frame of a chunk of a video stream made up of, for example,an I-frame and a plurality of subsequent P-frames and / or B-frames. The bitstream 204 is thenentropy decoded and passed into a decoder neural network 205 to reproduce a reconstructedimage ^^0 which in this case is an I-frame. The decoding step may be performed both locallyat the same location as where the input image compression occurs as well as at the locationwhere the decompression occurs. This allows the reconstructed image ^^0 to be available forlater use by components of both the encoding and decoding sides of the pipeline.In contrast to I-frames, P-frames (and B-frames) do rely on information from other frames.Accordingly, the P-frame part 202 at the encoding side of the pipeline takes as input not onlythe input image ^^^^ that is to be compressed (corresponding to a frame of a video stream atposition t), but also one or more previously reconstructed images ^^^^−1 from an earlier framet-1. As described above, the previously reconstructed ^^^^−1 is available at both the encodeand decode side of the pipeline and can accordingly be used for various purposes at both theencode and decode sides.At the encode side, previously reconstructed images may be used for generating a flow mapscontaining information indicative of inter-frame movement of pixels between frames. In theexample of Figure 3, both the image being compressed ^^^^ and the previously reconstructedimage from an earlier frame ^^^^−1 are passed into a flow module part 206 of the pipeline. Theflow module part 206 comprises an autoencoder such as that of the autoencoder systems ofFigures 1 and 2 but where the encoder neural network 207 has been trained to produce alatent representation of a flow map from inputs ^^^^−1 and ^^^^ , which is indicative of inter-framemovement of pixels or pixel groups between ^^^^−1 and ^^^^ . The latent representation of the flowmap is quantised and entropy encoded to compress it and then transmitted as a bitstream 208.On the decode side, the bitstream is entropy decoded and passed to a decoder neural network209 to produce a reconstructed flow map ^^ .The reconstructed flow map ^^ is applied to the previously reconstructed image ^^^^−1 to generatea warped image ^^^^−1,^^. It is envisaged that any suitable warping technique may be used, forexample bi-linear or tri-linear warping, as is described in Agustsson, E., Minnen, D., Johnston,N., Balle, J., Hwang, S. J., and Toderici, G. (2020), Scale-space flow for end-to-end optimizedvideo compression. In Proceedings of the IEEE / CVF Conference on Computer Vision andPattern Recognition (pp. 8503-8512), which is hereby incorporated by reference. It is furtherenvisaged that a scale-space flow approach as described in the above paper may also optionallybe used. The warped image ^^^^−1,^^ is a prediction of how the previously reconstructed image^^^^−1 might have changed between frame positions t-1 and t, based on the output flow mapproduced by the flow module part 206 autoencoder system from the inputs of ^^^^ and ^^^^−1.As with the I-frame, the reconstructed flow map ^^ and corresponding warped image ^^^^−1,^^may be produced both on the encode side and the decode side of the pipeline so they areavailable for use by other components of the pipeline on both the encode and decode sides.In the example of Figure 3, both the image being compressed ^^^^ and the ^^^^−1,^^ are passedinto a residual module part 210 of the pipeline. The residual module part 210 comprises anautoencoder system such as that of the autoencoder systems of Figures 1 and 2 but where theencoder neural network 211 has been trained to produce a latent representation of a residualmap indicative of differences between the input mage ^^^^ and the warped image ^^^^−1,^^. Thelatent representation of the residual map is then quantised and entropy encoded into a bitstream212 and transmitted. The bitstream 212 is then entropy decoded and passed into a decoderneural network 213 which reconstructs a residual map ^^ from the decoded latent representation.Alternatively, a residual map may first be pre-calculated between ^^^^ and the ^^^^−1,^^ and thepre-calculated residual map may be passed into an autoencoder for compression only. Thishand-crafted residual map approach is computationally simpler, but reduces the degrees offreedom with which the architecture may learn weights and parameters to achieve its goalduring training of minimising the rate-distortion loss function.Finally, on the decode side, the residual map ^^ is applied (e.g. combined by addition, subtractionor a different operation) to the warped image to produce a reconstructed image ^^^^ which is areconstruction of image ^^^^ and accordingly corresponds to a P-frame at position t in a sequenceof frames of a video stream. It will be appreciated that the reconstructed image ^^^^ can then beused to process the next frame. That is, it can be used to compress, transmit and decompress^^^^+1, and so on until an entire video stream or chunk of a video stream has been processed.Thus, for a block of video frames comprising an I-frame and ^^ subsequent P-frames, thebitstream may contain (i) a quantised, entropy encoded latent representation of the I-frameimage, and (ii) a quantised, entropy encoded latent representation of a flow map and residualmap of each P-frame image. For completeness, whilst not illustrated in Figure 3, any of theautoencoder systems of Figure 3 may comprise hyper and hyper-hyper networks such as thosedescribed in connection with Figure 2. Accordingly, the bitstream may also contain hyper andhyper-hyper parameters, their latent quantised, entropy encoded latent representations and soon, of those networks as applicable.Finally, the above approach may generally also be extended to B-frames, for example as isdescribed in Pourreza, R., and Cohen, T. (2021). Extending neural p-frame codecs for b-framecoding. In Proceedings of the IEEE / CVF International Conference on Computer Vision (pp.6680-6689).The above-described flow and residual based approach is highly effective at reducing theamount of data that needs to be transmitted because, as long as at least one reconstructed frame(e.g. I-frame ^^^^−1) is available, the encode side only needs to compress and transmit a flowmap and a residual map (and any hyper or hyper-hyper parameter information, as applicable)to reconstruct a subsequent frame.Figure 4 shows an example of an AI image or video compression process such as that describedabove in connection with Figures 1-3 implemented in a video streaming system 400. Thesystem 400 comprises a first device 401 and a second device 402. The first and seconddevices 401, 402 may be user devices such as smartphones, tablets, AR / VR headsets or otherportable devices. In contrast to known systems which primarily perform inference on GPUssuch as Nvidia A100, Geforce 3090, Gefore 4090 GPU cards, the system 400 of Figure 4performs inference on a CPU of the first and second devices respectively. That is, computefor performing both encoding and decoding are performed by the respective CPUs of the firstand second devices 401, 402. This places very different power usage, memory and runtimeconstraints on the implementation of the above methods than when implementing AI-basedcompression methods on GPUs. In one example, the CPU of first and second devices 401, 402may comprise a Qualcomm Snapdragon CPU.The first device 401 comprises a media capture device 403, such as a camera, arranged tocapture a plurality of images, referred to hereafter as a video stream 404, of a scene 404. Thevideo stream 404 is passed to a pre-processing module 406 which splits the video stream intoblocks of frames, various frames of which will be designated as I-frames, P-frames, and / orB-frames. The blocks of frames are then compressed by an AI-compression module 407comprising the encode side of the AI-based video compression pipeline of Figure 3. Theoutput of the AI-compression module is accordingly a bitstream 408a which is transmittedfrom the first device 401, for example via a communications channel, for example over oneor more of a WiFi, 3G, 4G or 5G channel, which may comprise internet or cloud-based 409communications.The second device 402 receives the communicated bitstream 408b which is passed to anAI-decompression module 410 comprising the decode side of the AI-based video compressionpipeline of Figure 3. The output of the AI-decompression module 402 is the reconstructedI-frames, P-frames and / or B-frames which are passed to a post-processing module 411 wherethey can prepared, for example passed into a buffer, in preparation for streaming 412 to andrendering on a display device 413 of the second device 402.It is envisaged that the system 400 of Figure 4 may be used for live video streaming at 30fps ofa 1080p video stream, which means a cumulative latency of both the encode and decode sideis below substantially 50ms, for example substantially 30ms or less. Achieving this level ofruntime performance with only CPU compute on user devices presents challenges which arenot addressed by known methods and systems or in the wider AI-compression literature.For example, execution of different parts of the compression pipeline during inferencemay be optimized by adjusting the order in which operations are performed using one ormore known CPU scheduling methods. Efficient scheduling can allow for operations to beperformed in parallel, thereby reducing the total execution time. It is also envisaged thatefficient management of memory resources may be implemented, including optimising cachingmethods such as storing frequently-accessed data in faster memory locations, and memoryreuse, which minimizes memory allocation and deallocation operations.A number of concepts related to the AI compression processes and / or their implementationin a hardware system discussed above will now be described. Although each concept isdescribed separately, one or more of the concepts described below may be applied in an AIbased compression process as described above.Concept 1: Automatic I-frame model selectionAs has been described above in the general section, known video compression pipelines use anapproach that relies on encoding and decoding an I-frame, which may take up a lot of bits, andthen encoding and decoding P- and / or B- frames that rely on information from the I-frame.In AI-based compression there is an added complication to this approach because the neuralnetworks that make up the encoders and decoders of the pipeline (such as those shown inFigure 3) can behave unpredictably across different input types (e.g. different video sequences)and resolutions (e.g. 480p, 720p, 1080p and so on). For example, consider for example twoseparate AI-based compression pipelines that may have the same or similar architecture as thatshown in Figure 3. The first one is running at 720p, whereby a sequence of 720p input imagesare used as input and a sequence of 720p images are reconstructed as output. The pipeline isable to compress the video sequence at a down to some arbitrary bit rate (i.e. compressionrate). This pipeline was trained to maximise the reconstruction quality (e.g. by minimisingsome distortion loss term such as MSE, or some other metric, down to a target level) for theinput images at the 720p resolution while at the same time minimising the number of bitsneeded to achieve that reconstruction quality.The second pipeline is the same but has instead been trained to maximise the reconstructionquality to a higher level by setting an even lower distortion loss target for the same bit rate.The inventors have realised that mixing components of the two pipelines that have been trainedfor different reconstruction qualities results in a pipeline that can compress and reconstructimage sequences at better accuracy than either pipeline individually.Counter-intuitively, a number of the components of the pipeline trained with the objective ofthe lower target distortion loss turn out to be "lazy" in that they rely on other components of thepipeline to do most of the work of the compression and reconstruction to reach the targeted bitrates and distortion losses. Conversely, some of the components of the pipeline compensate forthese "lazy" components by contributing substantially more to reaching the targeted bit ratesand distortion losses. The inventors have found that extracting the components that contributethe most and combining one or more of these with a set of components that have been trainedto a different target quality and / or bit rate produces a hybrid compression pipeline where all ofthe components contribute more than the sum of their parts.Specifically, consider the encoder and decoder neural networks of an I-frame module and theencoder and decoder neural networks of a P- and / or B-frame module such as that shown inFigure 5a. These may be, for example, an AI based compression pipeline of the type shown inFigure 3.The flow encoder neural network takes a current image ^^^^ and a previous image ^^^^−1, encodesthese into a flow latent representation which is optionally quantised and entropy encoded andtransmitted as a bit stream. On the decode side, the bitstream is received, entropy decoded intothe flow latent representation that the flow decoder neural network uses as input to producea representation of flow ^^ , e.g. a flow map. The flow map ^^ is applied to a previouslydecoded image ^^^^−1 to generate a warped version of that previously decoded image ^^^^−1,^^ . Thewarped version of the previously decoded image ^^^^−1,^^ is then fed into the residual encodernetwork, together with the current image ^^^^ to produce a residual latent representation which isoptionally quantised and entropy encoded and transmitted as a bit stream. On the decode side,the bitstream is received and entropy decoded back into the residual latent representation ^^ andused by the residual decoder neural network, in combination with information from the warpedversion of the previously decoded image ^^^^−1,^^ , to produce the reconstructed image ^^^^ . In thecase of Figure 5a, the information associated with the warped previously decoded image ^^^^−1,^^is optionally first processed by module ^^ , referred to herein after as a composition adapter, forexample to downsample and / or pad it before it is fed into the residual decoder together withthe entropy decoded residual latent representation to produce the final reconstructed image^^^^ . This process may then be repeated for ^^^^+1 and so on to encode, transmit, and decode asequence of frames.In the example of Figure 5a, the I-frame module comprising encoder 203 and decoder 205 hasbeen trained with a target distortion loss to target a given reconstruction quality ^^^^ for a givenbit rate. For completeness, an exemplary method of targeting specific reconstruction qualitytargets is described later herein with respect to Figure 7.It turns out that when the I-frame neural networks are trained to produce a high quality imagereconstruction (e.g. a low MSE loss) at a low bit rate, the P- and / or B-frame modules whichrely on the I-frame as input don’t have to work as hard to maintain the same low MSE loss andlow bit rate across an image sequence because the high quality I-frame allows the P- and / orB-frame modules to be "lazy" as there is no need for them to further reduce the MSE loss orlow bit rates when the I-frame has already done most of the work. With reference to Figure 5a,this means the flow encoder 207 and decoder 209 networks, and the residual encoder 211 anddecoder 213 networks perform sub-optimally when targeting a reconstruction quality of ^^^^irrespective of how they have been trained.Accordingly, to get the better performance out of a given set of I-frame neural networks andtheir corresponding P- and / or B-frame counterparts, the present disclosure combines I-frameneural networks and P- and / or B-frame neural networks that have been trained to differenttraining objectives (e.g. different target distortion and / or rate losses). For example, the I-frameencoder / decoder of Figure 5a, ^^^^^^ , trained to target a reconstruction quality of ^^^^ might becombined with P- and / or B-frame encoders / decoders ^^^^^^−1 or ^^^^^^+1that have been trained totarget a different reconstruction quality e.g. ^^^^−1 or ^^^^+1, ^^^^ is indicative of a giventarget distortion and / or rate loss achieved during training of the neural networks. Or, if wedenote the P- and / or B-frame modules as having the target quality ^^^^, that is ^^^^^^ , the presentdisclosure envisages grafting ^^^^^^−1 or ^^^^^^+1 onto ^^^^^^This is illustrated in Figure 5b which corresponds to Figure 5a but the neural networks 203,205 of the I-frame module 201 have been swapped out and replaced by neural networks 203,205 trained to target a different, higher, target quality ^^^^+1. That is, ^^^^^^ has been replaced with^^^^^^+1 , giving:^^^^^^^^^^^^→ ^^^^^^+1^^^^^^ Similarly, Figure 5c corresponds to Figure 5a but the neural networks 203, 205 of the I-framemodule 201 have been swapped out and replaced by neural networks 203, 205 trained to targeta different, lower, target quality ^^^^+1. That is, ^^^^^^ has been replaced with ^^^^^^−1 giving:^^^^^^^^^^^^→ ^^^^^^−1^^^^^^The effect of these swaps will now be described in more detail below.In the case of providing a higher quality I-frame module, as in Figure 5b, this allows theflow and residual encoders and decoders to more efficiently identify and remove spatial andtemporal redundancies in the P- and B-frames because the reconstructed I-frame containsfewer artefacts and retains more of the original input image that has been compressed. Becausethe P- and / or B-frame neural networks are highly dependent on the quality of the their inputs,a higher quality input image lets them more effectively compress and reconstruct the P- and / orB-frames with fewer artefacts compared to when the starting point is a lower quality input.Further, the small relative increase in bits used to encode the higher quality or resolutionI-frame is more than made up for by the substantial drop in distortion (e.g. MSE scores). Thatis, the effect on bit rate is small but the effect on reconstruction quality is high. Indeed, theinventors have found that this advantageous effect is observed across several typical videoresolutions and qualities used in the known video compression domain, including, but notlimited to 480p, 720p, 1080p, 1440p, 2160p, 4320p, SD, HD, Full HD, QHD, 2K, 4k, 8k, aswell as qualities or resolutions higher and lower than these. The overall effect of this approach isaccordingly a "hybrid" compression pipeline with improved compression and / or reconstructionaccuracy than the non-hybrid pipeline, where the term "hybrid" refers to combining modulestrained at one target image reconstruction quality with modules trained for a different targetquality.In the case of providing a lower quality I-frame module, as in Figure 5c, this allows theP- and B-frame modules to let the I-frame module do most of the work and accordinglyprovides a way to control which parts of the pipeline are over-performing and under-performing.This is particularly advantageous in the context of analysing and troubleshooting under- orover-performance of a pipeline during and after training and accordingly provides a particularlyeffective diagnostics tool. It can also counter-intuitively be beneficial in high motion sceneswhere information in the I-frame does not persist for many frames.Consider next the further generalisation of the above approach whereby we can generaliseto an arbitrary set of set of I-frame modules. Let ^^^^^^ = {^^^^1 , ^^^^2 , ^^^^3 , .. .} denote the set ofI-frame modules trained to different target qualities where ^^^^ is a target output reconstructionquality. Let ^^^^^^ = {^^^^1 , ^^^^2 , ^^^^3 , . ..} denote the set of P- and / or B-frame modules trainedto different target qualities where ^^^^ is a target output reconstruction quality. In the case ofthis generalisation it is also envisaged that ^^^^ denotes not only quality generally, but alsoquality for a specific set of input images ^^. This is because the neural networks of the encodersand decoders of the I-frame and P- and / or B-frame modules can be sensitive to the type ofdata they are trained on. That is, if trained on only high-motion video game frames, theperformance on such video game frames will be high, but the performance on static naturedocumentary footage may be low. Conversely, if trained on static nature documentary footage,the performance on high-motion video game will be low and the performance on static naturedocumentary footage will be high. Accordingly, it is possible to finetune different I-framemodels for different input data types these may then be included in the set ^^^^^^ which may thuscomprise a set of specialist I-frame modules trained to perform well on different types of inputimages. Similarly, a set of specialist P- and / or B-frame modules may be provided trained ondifferent data types to perform well for those data types. The present disclosure envisagesidentifying a best performing I-frame module and P- and / or B-frame module combinations fora given set of input images ^^ which can be updated on a per input GOP (group of pictures)basis, a per frame basis, or on a longer term basis for an entire sequence of frames of a video,and so on. This generalised frame is set out in pseudocode below:

[0002] Algorithm 1 Automatic I-frame model selection and combination with and P-, B- frame modelsInputs: (i) Set of pre-trained I-frame encoder / decoder neural networks ^^^^^^ = {^^^^1 , ^^^^2 , ^^^^3 , ...}where ^^^^ is a target output reconstruction quality or other target(ii) Set of pre-trained P- and / or B-frame encoder / decoder neural networks^^^^^^ = {^^^^1 , ^^^^2 , ^^^^3 , ...} where ^^^^ is a target output reconstruction quality or othertarget (iii) Classifier ^^Outputs: ^^^^^^^^^^^^Variables: Input image(s) ^^Apply classifier ^^^^^^^^^^^^ ^^ ^^^^^^^^^^^^^^ = ^^ (^^).Select ^^^^^^ based on ^^^^^^^^^^^^ ^^ ^^^^^^^^^^^^^^.Select ^^^^^^ based on ^^^^^^^^^^^^ ^^ ^^^^^^^^^^^^^^.Attach ^^^^^^ to ^^^^^^ .

[0003] More specifically, a the set of pre-trained I-frame modules ^^^^^^ and the set of pre-trained P-and / or B-frame modules ^^^^^^ is provided. An input image or images ^^ is then received andprocessed by a classifier ^^. This classifier may itself be a neural network or a simple non-neuralmodel, for example a regression model. The output of the classifier ^^ may be a flag or someother classification result indicative of which of the set ^^^^^^ and / or ^^^^^^ is predicted to have thebest performance for that specific input image or images ^^. The classifier ^^ may be trained onlabelled data comprising images, an indication of an I-frame and / or P- and / or B-frame modelused to compress and reconstruct that data, and associated bit rates and / or distortion scores forthat compression and reconstruction. The classification result may then be used to select agiven pair of ^^^^^^ and / or ^^^^^^ modules which are then attached and used on the input image orimages ^^ during inference. The next input image or images ^^ is then received and the processis repeated. For example, a first I-frame module may be selected for a first input image orimages, a second I-frame module selected for a second input image or images, a third I-framemodule selected for a third input image or images, and so on.Figure 6 illustratively shows steps of using a classifier ^^ to select an I-frame module froma set of I-frame 201 modules {^^^^^^−1 , ^^^^^^ , ^^^^^^+1 , ...} , for example corresponding the methodsdescribed above. An input set of images ^^0, ^^^^−1, and ^^^^ are received at a computer system.The classifier ^^ is applied to the input set of images to produce a classification output onwhich a selection of one of the I-frame modules is based. The selected I-frame module is thengrafted onto the flow-residual modules (i.e. the P- and / or B-frame modules) to produce thehybrid compression pipeline which is then used to encode, transmit and decode the input set ofimages. A corresponding approach (not shown) may be used for selecting a P- and / or B-framemodule from a set of P- and / or B-frame modules.It will be appreciated that either of the sets ^^^^^^ and / or ^^^^^^ may only have a single module,for example only one P- and / or B-frame module may be provided in ^^^^^^ , in which case theclassifier ^^ selects only a suitable I-frame module to combine with the only choice of P- and / orB-frame modules. In this case, the P- and / or B-frame module can be trained on all types ofdata sets, to be a generalist module, while I-frame modules are trained to be specialists fordifferent types of data (e.g. high motion video game frames, static nature scene frames, videoconference footage frames and so on) and these can be hot-swapped into the compressionpipeline based on the input ^^. The above generalised framework of course still also appliesgenerally to the different reconstruction qualities hybrid pipelines regardless of data typedescribed at the beginning of concept 1.Finally, it is envisaged that instead of a classifier, a rules or euristic based approach may beused to select a given I-frame module ^^^^^^ and / or P- and / or B-frame module from the setsof said modules. For example responsive to receiving meta data associated with the inputimage or images ^^ that can be used to select the most suitable I-frame module and / or P- and / orB-frame module.

[0004] Concept 2: Training hybrid compression pipelinesExtending concept 1 further, consider the behaviour of the neural networks of the I-framemodule and P- and / or B-frame modules during training. One behaviour that is observed isthat the individual neural networks of the pipeline are able to learn to compensate for changesmade elsewhere in the pipeline. For example, if during training the neural networks of theflow encoder and / or decoder are simplified and made less expressive by for example removinglayers, the neural networks of the residual encoder and / or decoder can learn to become moreexpressive to compensate as best as possible within the confines of their given architecture. Theinventors have realised that this behaviour can be exploited to force individual components tobecome more expressive and better at their specific tasks by artificially introducing difficultiesduring training, and then removing these difficulties during inference.Specifically, the inventors have realised that combining a low quality I-frame module (e.g.a pre-trained I-frame encoder and decoder that is less expressive or has been pre-trained toemphasise compression rate over image reconstruction quality and which thus produces lowquality I-frames) with trainable P- and / or B-frame modules during training forces the neuralnetworks of the P- and / or B-frame modules to compensate for the weaker I-frame module.They do this by producing outputs that have higher reconstruction accuracy despite the lowerquality I-frame so that the overall loss at the end of training still converges to some low, targetlevel. In other words, giving the pipeline a worse I-frame module during training stops the P-and / or B-frame modules from "cheating" by offloading all the work on to the I-frame module.The overall result of this type of training is a set of trained P- and / or B-frame module neuralnetworks that produce more accurate image reconstructions for a given bit rate comparedto corresponding neural networks trained with a standard or high quality I-frame module.That is, the neural networks of the P- and / or B-frame module in the standard approach are"lazy" and let the I-frame module do more of the work than it needs to whereas making theI-frame encoder / decoder worse during training forces the P- and / or B-frame flow and residualencoders / decoders and other associated components to compensate to achieve comparableresults. The P- and / or B-frame flow and residual encoders / decoders trained in this way canthen be grafted back onto the original standard or high quality I-frame encoder / decoder toproduce a compression pipeline during inference with I-frame, P- and / or B-frame modulesthat are not compensating for each other but instead all contribute to produce higher imagereconstruction qualities for given bit rates. The following pseudocode algorithm provides aframework for the training approach of the present concept:

[0005] Algorithm 2 Hybrid quality I-frame and P-, B- frame encoder decoder training, with pre-trained I-frame module.Inputs: (i) Pre-trained I-frame encoder / decoder neural networks ^^^^1 with quality index ^^1(ii) Pre-trained I-frame encoder / decoder neural ^^^^2 with quality index ^^2,where ^^2 is lower quality than ^^1(iii) Trainable P- and / or B-frame encoder / decoder neural networks^^^^1 with target qualityindex ^^1Outputs: ^^^^1^^^^1Variables: Optimizer ^^, gradient computation flag ^^^^^^^^.Initialise: ^^^^^^^^ = ^^^^^^^^ for ^^^^2^^^^1 , and Training loop counter ^^ = 0.while ^^ < ^^ doForward pass through ^^^^2^^^^1.Compute loss and perform backward pass with ^^^^^^^^.Apply optimizer ^^ to update ^^^^1.^^ ← ^^ + 1end whileDetach ^^^^2 from ^^^^1.Attach ^^^^1 onto ^^^^1.

[0006] That is, a first I-frame module comprising encoder and decoder networks ^^^^1, is pre-trained onimages with a first loss function comprising at least a rate term and a distortion term. TheI-frame encoder and decoder may correspond to the encoder 203 and / or decoder 205 in Figures3 and 5. To target a higher quality reconstruction ^^1 during training, the distortion term isweighted more heavily. This may be achieved by way of one or more control units, for exampledescribed in Cui, Z., Wang, J., Gao, S., Guo, T., Feng, Y. and Bai, B., 2021. Asymmetric gaineddeep image compression with continuous rate adaptation. In Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition (pp. 10532-10541). Alternatively,the target rate may be set by using an augmented Lagrangian approach as described for examplein WO2022084702, which is incorporated by reference herein in its entirety, whereby therate term and / or distortion term of the loss function are weighted by a learned Lagrangianmultiplier. Other ways to modify the loss function to target different training objectives such asreconstruction quality are known to the skilled person. The image resolution and / or target bitrate are denoted here by a given quality index ^^1 associated with the trained I-frame encoderand decoder. The same process is repeated but for a lower target quality ^^2. Thus two differentquality pre-trained I-frame neural networks ^^^^1 and ^^^^2 are provided, where ^^2 is a lower targetreconstruction quality than ^^1.Alternatively, in a further generalisation of this concept, a larger set of different I-frame neuralnetworks may be provided and an I-frame module with a target ^^ value may be chosen randomlybased on a distribution centered around the ^^ target of the P- and / or B-frame quality beingtargeted. This approach attempts to instill in the P- and / or B-frame neural networks the abilityhandle both high and low quality I-frame inputs and controlling the center of the distributionallows the training of the P- and / or B-frame module to be controlled to attempt to converge toany target ^^ while at the same time retaining the ability to handle high and low quality inputI-frames of other ^^ values.Next, untrained P- and / or B-frame encoder and decoder neural networks ^^^^1, for examplecorresponding to encoders 207, 211 and decoders 209, 213 in Figures 3 and 5, are providedand joined to the lower quality I-frame neural networks ^^^^2 to give a compression pipelinecomprising the modules ^^^^2^^^^1. A control unit such as that described above for the I-frameencoder / decoder, or the augmented lagrangian approach of WO202208470, may also be appliedto the P- and / or B-frame encoder / decoder to target a given quality e.g. ^^1. The grad parameteris set to true for ^^^^2^^^^1 and the training loop for ^^ iterations is performed with correspondingforward passes, loss calculations and application of a suitable optimizer, for example stochasticgradient descent, or some other optimizer until and end training condition is met, such as after^^ steps, or after ^^ epochs, or after the loss plateaus, and so on. The now trained P- and / orB-frame networks are then detached from ^^^^2 and instead attached to the I-frame module ^^^^1which has a higher target reconstruction quality ^^1 to produce the hybrid pipeline comprising^^^^1^^^^1.The hybrid pipeline comprising networks ^^^^1^^^^1 accordingly comprises weights that are betterable to compress and reconstruct P- and / or B-frames compared to weights of a pipeline madeof components all trained to target a quality index e.g. ^^1 together. That is, the neural networksof ^^^^1 trained as described above with a lower quality index I-frame encoder / decoder ^^^^2 aresuperior to corresponding neural networks of ^^^^1 trained in a traditional end-to-end mannerwith a higher quality index I-frame encoder / decoder ^^^^1.In an alternative implementation, it is also possible to apply the above approach in an end-to-endmanner (that is, training the I-frame encoder / decoder at the same time, rather than pre-trainingit) by applying different control units, or by applying the augmented Lagrangian approach ofWO2022084702, that target different qualities ^^ for at different times during training. Thisalternative approach is set out in the following pseudocode algorithim.

[0007] Algorithm 3 Hybrid quality I-frame and P-, B- frame encoder decoder training, end-to-end with I-frame moduleInputs: (i) Trainable I-frame encoder / decoder neural networks ^^^^^^^^ with quality target index ^^^^^^(ii) Trainable P- and / or B-frame encoder / decoder neural networks ^^^^^^^^ with qualityindex ^^^^^^Outputs: ^^^^1^^^^1Variables: Optimizer ^^, gradient computation flag ^^^^^^^^, I-frame target quality ^^^^^^ , P-frame targetquality ^^^^^^ .Initialise: ^^^^^^^^ = ^^^^^^^^ for ^^^^^^^^ ^^^^^^^^ , Training loop counter ^^ = 0, and set target qualities ^^^^^^ = ^^2and ^^^^^^ = ^^1.while ^^ < ^^ doForward pass through ^^^^^^^^ ^^^^^^^^ .Compute loss and perform backward pass with ^^^^^^^^.Apply optimizer ^^ to update ^^^^^^^^ and ^^^^^^^^ .^^ ← ^^ + 1end whileReset training loop counter ^^ = 0.Freeze ^^^^^^^^ .Set I frame encoder decoder target quality ^^^^^^ = ^^1.while ^^ < ^^ doForward pass through ^^^^^^^^ ^^^^^^^^ .Compute loss and perform backward pass with ^^^^^^^^.Apply optimizer ^^ to update ^^^^^^^^ .^^ ← ^^ + 1end whileThat is, untrained I-frame encoder / decoder neural networks ^^^^^^^^ with quality target index ^^^^^^and untrained P- and / or B-frame encoder / decoder neural networks ^^^^^^^^ with quality index^^^^^^ are initialised. The variable ^^^^^^^^ is set to true for ^^^^^^^^^^^^^^^^ and different target qualitiesare set for each, that is ^^^^^^ = ^^2 and ^^^^^^ = ^^1, where ^^2 is a lower quality than ^^1. Thetraining loop is then started with forward and backward passes through ^^^^^^^^^^^^^^^^ , associatedloss estimations and weight optimizations using optimiser ^^. After a predetermined numberof training steps, the weights of ^^^^^^^^ are frozen, and the target quality for I-frame neuralnetworks ^^^^^^^^ are set higher e.g. to ^^1. A further ^^ number of training loops are performedbut now only updating the weights of ^^^^^^^^ to produce a final set of networks ^^^^1^^^^1 that cansubsequently be used in inference for better performance than an AI-based compresison piplinetrained in a non-hybrid manner.A further advantage of either of the above hybrid training approaches is that AI-basedcompression pipelines trained in this way are not only able to achieve reduced MSE scoresfor given bit rates but they also become substantially more robust to compression artefactsthat arise during the compression of long groups of pictures (GOPs). In video encoding anddecoding generally, a GOP is a collection of successive pictures within a coded video stream.Each coded video stream consists of successive GOPs, from which the displayed frames aregenerated. A long GOP may be understood as GOPs of greater than 20 frames, for example 30,40, 50 or 60 frames or more. Conversely, a short GOP may be understood as a GOP with <20frames. One problem with AI-based compression is that the encoders and decoders trained onshort GOPs (e.g. <20 frames) perform poorly when used to compress and decompress longGOPs during inference. One approach to overcome this issue is to increase the number of longGOP data in a given training data set. However, this can substantially increase training timesand in some cases results in non-convergence or instability during training. Further, long GOPtraining data sets are not widely available and are expensive to build from scratch.It turns out that the above described hybrid training approaches of training P- and / or B-frameencoders and decoders with an I-frame encoder / decoder of a different (worse) target qualityemulates the poorer quality of images that the P- and / or B-frame encoders / decoders would getas an input when performing inference on long GOP sequences. This is because artefacts anderrors in images towards the end of long GOP sequences are typically of poor quality becauseany artefacts and errors that arise in for the frames at the beginning and middle of the long GOPsequence propagate and degrade the image quality as the sequence progresses. By trainingthe P- and B-frame encoders and decoders with a poor quality I-frame encoder / decoder, theybecome used to working with poor quality images as input regardless of the actual length ofsequence being considered during inference. The overall effect of this is that their performanceon long GOP sequences during inference substantially improves despite not having seenany long GOP sequences in training data. The presently described hybrid training approachaccordingly acts as a proxy for long GOP training and provides the advantages of long GOPtraining without actually needing long GOP training data, without the long training times, andwithout the potential low training stability that can arise when training on long GOP data.More generally, the above-described ability of the P- and / or B-frame encoders and decoders tobetter handle longer sequences of images are observed across all types of image sequencesincluding, but not limited to, webcam image sequences, stereo image sequences, animationsequences, low-movement scenes, high-movement scenes, film, mobile-device filmed content,and so on.An example type of artefact that appears specifically in AI-based image and video compression,but not in traditional compression, that is reduced by the above-described approaches areable to solve is known as "unrolling". Unrolling is an artefact that typically appears in imagesequences where there is little to no movement between frames and manifests itself in thereconstructed images as halo-like rings around edges and objects in the images. These artefactstypically arise where small, barely noticeable changes in pixel values between frames suchas tiny lighting changes, block artefacts from traditional compression (if the input to the AIcompression pipeline came from a traditionally compressed source), noise, and so on becomeamplified over time by the AI based compression pipeline. This results in the networks ofthe compression pipeline treating what is effectively image noise as real movement and thenexaggerating across the sequence of frames. At the end of the sequence of frames, the artefactscan be quite severe and detrimental to the image reconstruction quality. In long GOP sequences,the effect can be particularly pronounced as errors propagate much further over the course oflong P- and / or B-frame sequences before the next I-frame provides "fresh" information withoutartefacts for the next set of P- and / or B-frames to be generated from. Unrolling artefactsoccur across mostly all types of image sequences but are particularly apparent when AI basedflow-residual compression models are used to encode and decode image sequences of internetvideo calls (whereby a foreground user who moves very little is surrounded by a largely staticbackground which also does not move much), screen sharing image sequences, security camerafeeds and others. If there is no movement after a period of time, objects and edges start toexhibit halo-like ringing around their edges. The artefacts also result in the compressionpipeline assigning a disproportionate number of bits to handle the hallucinated movementof the unrolling artefacts even though the corresponding parts of the actual images aren’tchanging or are hardly changing between frames so ought not to have many bits assigned tothem. An AI-based compression pipeline trained using the hybrid approach described hereinis able to mitigate unrolling and other artefacts across much longer GOP lengths because theimproved P- and / or B-frame module neural networks are better able to reconstruct inputs thatinclude the early stage formations of artefacts that cause unrolling before they can propagateand become worse as the sequence progresses.Other types of AI-based image compression artefacts, such as pixel shuffle artefacts, andothers are also reduced during inference when using networks trained according to the abovedescribed approaches.Concept 3: Setting Relative Quality Levels during TrainingOne issue that arises when training I-, P- and / or B- frame modules targeting a given qualitylevel is how to set the quality level targets to achieve some desired bitrate and distortion scores.Quality levels may be defined in one of any number of ways, including but not limited to thequality levels of each module being controlled with control units as described in Cui, Z., Wang,J., Gao, S., Guo, T., Feng, Y. and Bai, B., 2021. Asymmetric gained deep image compressionwith continuous rate adaptation. In Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition (pp. 10532-10541), or for example the control unit approachdescribed in more detail below with reference to Figure 7.To help with understanding quality levels and how they may be controlled, we first go intomore detail of of the non-limiting example control units shown in Figure 7 which illustrativelyshows a compression pipeline 700 corresponding to that of Figure 2, or the I-frame module200 of Figure 3, except now a control unit ^^ 701 and corresponding inverse control unit ^^−1702 are provided. On the encode side, the control unit 701 is positioned before the quantisationmodule. On the decode side, the inverse control unit 702 is positioned after the entropy decodermodule. Note the inverse control unit 702 need not be a strict mathematical inverse.The control unit 701 receives as input the latent representation and scales one or more channelsof the latent representation through processing it with with a learned matrix (or vector whenconsidering the latent on a per channel basis). Here processing may refer to multiplication,for example channel-wise multiplication. This has the effect of transforming the latentrepresentation (i.e. modifying its values) before they are quantised and accordingly provides adegree of control over what input the quantisation module receives in turn effects how well ornot well a given image will be compressed i.e. it facilitates rate control.More generally, the control unit 701 may apply a control matrix ^^ ∈ ^^^^×^^ where ^^ is thenumber of channels and ^^ is the number of control vectors ^^^^ = {^^^^,0, ^^^^,1, ..., ^^^^,^^−1} with ^^denoting the index of the control vectors in the control matrix ^^, and ^^^^, ^^ ∈ ^^ representsthe ^^ th control value in the control vector ^^^^ and ^^ ranges from 0 to ^^ − 1, and where eachchannel may be associated with its value or values. It follows that applying the control matrix^^ changes the latent representation by ^^^^, ^^ = ^^^^ × ^^^^, ^^ . More specifically, the operation appliedby the control unit 701 may be defined as ^^^^ = ^^^^ (^^, ^^) = ^^ ⊙ ^^^^ where ^^^^ (·) is the controlunit’s operation and ⊙ is the channel-wise multiplication.The modification of the latent representation by control matrix ^^ can be understood at a moregeneral level as corresponding to simulating better or worse compression rates by emphasisinghow many bits the network should assign to one or more of the input channels of the latentrepresentation.At one extreme, the control matrix ^^ blows all of the values of the latent representation toinfinity. That is, if the quantitation bin size stays fixed, then the quantisation error reduces asthe range of values that the latent can occupy increases. Although in practice, this extreme cannever be reached and runs into numerical problems.At the other extreme, the control matrix ^^ may completely transform all the channels of thelatent representation into 0 which may be perfectly losslessly compressed at minimal cost butfrom which it is practically impossible to reconstruct any meaningful image.Between these two extremes are a set of target reconstruction qualities (with associatedcompression rates) which are defined by the control vectors ^^^^ of the control matrix ^^ andwhich emphasise or de-emphasise higher rate or conversely higher distortion during trainingand which encourage the network to learn to assign more or fewer bits to the various channelsof the latent representation.On the decode side, a corresponding inverse control matrix ^^−1 is applied by the inversecontrol unit ^^−1 702 to the reconstructed latent representation output by the entropy decoderbefore it is passed into the decoder neural network to reconstruct the output image. Similarlyto the control matrix ^^, the inverse control matrix ^^−1 comprises a number of inverse controlvectors ^^−1^^= {^^−1^^,0 , ^^−1 ^^,1 , ..., ^^−1 ^^,^^−1}, and the operation of the inverse control unit ^^−1702can be = ^̂^ ⊙ ^^−1^^. Here ^^−1^^(·) and ⊙ corresponds to the samechannel-wise multiplication operations as on the encode side.As indicated above, the elements of the control matrix ^^ are learned jointly with the otherparameters of the compression pipeline (e.g. the weights of the encoders and decoders,and so on). Taking the control matrix ^^ and the inverse control matrix ^^−1 together, it isnoted that there will always be pairs of corresponding control vectors {^^^^, ^^−1^^ } bound by thecorresponding index ^^ whereby one part of the pair is associated with the the encode side andone part is associated with the decode side.The control matrix ^^ and inverse control matrix ^^−1 in general terms operate to simulatehigher or lower compression rates by modifying the latent representation. We can accordinglyassociate different Lagrange multipliers to be applied to the rate or distortion term of the lossfunction with each control vector pair during training of the compression pipeline.More specifically, for one training iteration, we select one Lagrange multiplier ^^^^ (which willemphasise rate or distortion more for that training iteration), together with its associated pair ofcontrol vectors {^^^^, ^^−1^^ }, evaluate the loss function with the rate or distortion term regularisedby that Lagrange multiplier and control vectors, and finally update the elements of the networkincluding the values of the elements of the control unit and inverse control unit, based on theevaluation of that loss function.For the next iteration, we may select a different Lagrange multiplier and its associated differentcontrol vector pairs and repeat the process but this time when we update the elements, it willbe for the different control vectors. We then repeat this process throughout training, selectingdifferent Lagrange multipliers and associated control vector pairs each iteration.At the start of training, the randomly initialised control vectors and network weights will havehigh losses, however after a few iterations loss decreases and the values of the elements of thecontrol vectors start to converge resulting in a set of control vectors {^^^^, ^^−1^^ } each associatedwith a different regularisation of the rate or distortion terms of the loss function, and thus eachemphasising a reconstruction of the target image emphasising a different rate and distortion.Given that rate effectively determines how many artefacts a reconstructed image will have (e.g.a measure of quality), the control vectors allow the network to target specific rates and thustarget reconstruction image qualities.Accordingly, during inference, a pair of the resulting, learned control vectors of the set canbe selected and applied respectively to the latent representation on the encode side and tothe output of the entropy decoder on the decode side to emphasise either more rate or moredistortion. If more distortion (i.e. a lower rate and corresponding to worse image quality) isdesired, the control vector pair associated with the higher distortion regularisation amountmay be selected.In this way, the compression pipeline can be controlled to vary the rate (and thus targetreconstruction quality) of the reconstructions it produces simply be selecting a different pairof the control vectors to apply to the latent representation and output of the entropy decoderrespectively. Further training details of the variable rate compression network will now be provided in moredetail. Consider the typical rate distortion loss function based on the steps of the pipeline ofFigure 1:^^ = ^^(^^( ^^^^ (^^))) + ^^ · ^^ (^^^^ (^^))where ^^ is the input image, ^^^^ is the encoder network that produces the latent representation fromthe input image, ^^ represents the quantisation operation applied to the latent representation,^^and ^^^^ is the decoder network that reconstructs the output image ^^ from the reconstructedlatent representation.In order to introduce a dependence of the loss on the control unit and the inverse control unit^^^^ and ^^−1^^ , the loss function is modified by the introduction of the regularisation term ^^^^with a pair of randomly initialised control vector pairs {^^^^, ^^−1^^ } that are linked tothe regularisation term by the index ^^, that is:^^ = ^^(^^(^^^^ ( ^^^^ (^^)))) + ^^^^ · ^^ (^^−1^^ (^^^^ (^^))) For the sake of example, assume we would like a set of five control vector pairs {^^1, ^^−11 },{^^2, ^^−1−1 −1 2}, {^^3, ^^3 }, {^^4, ^^4 },{^^5, ^^−1 5}, each pair associated with an associatedquality or rate ^^1, ^^2, ^^3, ^^4, ^^5. We set up a corresponding set of Lagrange multipliers^^1 = 0.5, ^^2 = 0.05, ^^3 = 0.005, ^^4 = 0.0005, ^^5 = 0.00005. The specific values here areillustrative only where a higher value will emphasise distortion more compared to rate whereasa lower value will emphases rate more over distortion.In the first training iteration, we initialise the control vector pairs (e.g. with random values)and weights of the networks of the compression pipeline and then randomly select an ^^ value,let’s say ^^ = 3, which sets ^^3 = 0.005, inserts the control vector pair {^^3, ^^−13 } into the pipelineat control unit ^^3 and inverse control unit ^^−13 , and thus results in the loss function for thisiteration as: ^^ = ^^(^^3(^^( ^^^^ (^^)))) + 0.005 × ·^^ (^^−13 (^^^^ (^^)))Backpropagation is performed, and the weights of the network and values of the control vectorpair {^^3, ^^−13 } are updated. For the next iteration, a different ^^ is randomly selected and the process is repeated for apredetermined number of training iterations or until some end of training condition is met(e.g. the training and / or validation loss stops decreasing, and so on). Whilst the initial trainingsteps will not produce good reconstructions at any target image quality, after a few iterations,the output reconstructions associated with each randomly selected ^^ will start to converge torespective positions along the rate distortion curve, such as those shown illustratively in Figure5.After training is completed, we have the set of set of five control vector pairs {^^1, ^^−11 },{^^2, ^^−12 }, {^^3, ^^−1 3}, {^^4, ^^−1 −1 4},{^^5, ^^5 } that, when applied during inference, result in the different target reconstruction qualities on the ratedistortion curve. Note for completeness that even though discrete control vector pairs arelearned, interpolating between the discrete values of ^^^^ and ^^^^+1 is possible, facilitatingintermediate quality levels in inference between the discrete values.It will be appreciated that one or more bits indicating which target reconstruction quality isbeing used may be included in the bitstream as metadata. This may be in the main bitstreamor, preferably, in the hypernetwork bitstream. In the latter case, the hypernetwork bitstream isdecoded first, allowing the the entropy decoded modified latent representation to be processedby the inverse control unit 702 using the correct control vectors before the processed modifiedlatent representation is fed into the decoder to reconstruct the image at the target image quality.More generally it is envisaged that the different control vectors of the control units once learnedwill from part of the pipeline’s weights and architecture installed on the encode and decodeside so it is not necessary to send all the values of the control vectors. Instead, a simpleindication of which vector pairs from the set of vector pairs has been selected may be sent, anda lookup table used to retrieve the correct control vectors during decode.In practice, the effect the control vector pairs have in inference is to modify the latentrepresentation before quantisation to make it more easily or less easily quantised into coarseor fine quantisation bins. That is, the learned values of the control vector pairs associatedwith the low rates typically end up with vector values that modify the distribution of the latentrepresentation to be easily quantised into coarse bins whereas the values of the control vectorpairs associated with higher rates typically result in a latent representation distribution thatrequires finer quantisation bins and which can accordingly not be compressed as easily. Asdescribed above, the exact values are learned parameters and may accordingly vary dependingon the number of vector pairs in the set of vector pairs, as well as the values of the regularisationparameters in the set of regularisation parameters.When implementing the above training methods in practice, a controller (e.g. a proportionalcontroller, a proportional integral controller, or proportional integral derivative controller orsimilar) may be used to target a predetermined set of rate and / or distortion targets, for examplemean square error (MSE) distortion targets and / or Mbit / sec bitrate targets. Whereby, whenevaluating the rate and / or distortion loss, the ^^ values associated with each control vector pairmay be increased or decreased based on how far off the rate and / or distortion score of a currenttraining step is from the target, thereby gradually causing convergence to a desired rate and / ordistortion target as training progresses. Whereby each quality level ^^^^ as described in concepts1 and 2 above each have their own set of control vector pairs. Further, a second controller (e.g.a proportional controller, a proportional integral controller, or proportional integral derivativecontroller or similar) may be used during training to adjust the targets of the first controllerbased on how far off the other of the rate and / or distortion loss of a current training step arefrom that target. The second controller may thus be said to be an auxiliary or meta controllerused for controlling the targets of the first controller. For example, if the starting distortiontargets of the first controller are MSE scores of 45, 60, 75, 90, 105 and the first controllercontrols the ^^ value of the control units based on how close the current step distortion score isto those MSE targets, then the second controller is used to adjust the values of those distortiontargets based on how close the current training step rate score, as measured on a hold out dataset, is to one or more rate targets (for example the targets 0.5 Mbit / s, 1 Mbit / s, 2 Mbit / s, 3Mbit / s, 6 Mbit / s, and so on) . For example, if the starting set of MSE targets 45, 60, 75, 90,105 are consistently resulting in rate of the current training step, as measured on a hold outdata set, being above the 0.5 Mbit / s, 1 Mbit / s, 2 Mbit / s, 3 Mbit / s, 6 Mbit / s targets, then thesecond controller might cause the MSE targets of the first controller to be adjusted higher toe.g. 50, 65, 90, 105, 120 (i.e. moving towards a worsening of reconstruction quality to lowerthe achieved bitrate), and so on, until the set of MSE targets is found that results in controlvector pairs that achieve the desired bitrates.In this way, the two controllers work together to converge to some set of control vector pairsbased on ^^ values controlled by a converging set of distortion targets whose own convergenceis based on one or more predetermined bitrate targets that the compression pipeline is trying tohit. This two-controller setup facilitates training where one of the only rate distortion targethyperparameters that has to be set is the predetermined bitrate targets. The MSE targets and^^ values (and associated control vector values) automatically converge to whatever valuesachieve those bitrate targets through operation of the controllers working together. In verygeneral terms, the two-controller setup facilitates a convenient "fire and forget" approach totraining. One or both controllers may be activated or deactivated during training according tosome predetermined training schedule, or based on some predetermined training criteria.With an illustrative rate distortion target control mechanism now described, we can now returnto the question of what the rate and distortion targets are that achieve some desired relativequality levels of I- and P- / B-frame modules. In the case of a control unit based quality levelcontrol mechanism, this question becomes: what should the relative rate and / or distortiontargets of the controller(s) should be when training the various sets of different quality I-framemodules and / or P- / B-frame modules both before and after they are grafted together as set outin concept 1 and 2 above. That is, in a single or two controller setup, how much higher orlower should the controller targets be when training the set ^^^^^^ compared to ^^^^^^? Or, moregenerally, what relative quality level targets between the I-frame and / or the P- / B-frame moduleachieves stable training and thus maximises the advantages of combining the different qualitylevel modules at inference as set out in detail in concepts 1 and 2 described above? For qualitylevel control mechanisms other than control units, a corresponding question arises based onwhatever control parameters such a mechanism uses.The inventors have realised that the values (or some fraction or multiple thereof) that thecontroller bitrate targets are set to and what the distortion targets converge as a result of thosebitrate targets to when training the P- / B-frame module can be used as a starting set of targetsfor the I-frame module controllers when training the (higher or lower) quality I-frame module.When the I-frame module targets are based on the final P- / B-frame module targets, trainingstability is improved and convergence to a global optimum of rate distortion performance isachieved not just for the I-frame module being trained but for the pipeline as a whole when thefinal I- and P- / B-frame modules are grafted together.To illustrate this, consider Figure 8 which shows a training framework in which an I-framemodule is trained using control units with ^^ values controlled by a set of distortion (MSE)target levels, and a P- / B-frame module is trained using control units with ^^ values controlledby a set of distortion (MSE) target levels. After training, the two modules are combined into asingle AI-based compression pipeline. The exact values of the MSE targets are not importantand depend on what kind of bit rate ranges the modules are being trained for (that is, higherMSE targets will typically produce control unit values that result in compression to low bitrateranges where the distortion is high whereas a lower MSE target will typically produce controlunit values that result in compression to high bitrate ranges). Using a non-limiting toy example,in order to cover a bitrate range of 0.5-6Mbit / sec with an architecture such as that of Figures3-5c , a set of MSE targets for a corresponding set of control vector pairs might be 45, 60, 75,90, 110, or some other range of values. If we consider this set of MSE targets to correspond toan P- / B-frame module having a baseline quality level ^^^^, then the question arises as to whatshould the starting MSE targets be for ^^^^±^^ for an I-frame module that is being grafted onto theP- / B-frame module that has been trained at quality level ^^^^.The inventors have found that the advantages of concepts 1 and 2 appear to be greatest, that isthe overall rate distortion performance of a pipeline with an I-frame module having a differentquality level to the P- / B-frame module it has been grafted to is best, when the MSE targets ofthe controller used to train the control units or control units of the I-frame module is basedon the MSE targets of the controller used to train the control units or control units of theP- / B-frame module that the I-frame module is going to be grafted to. To illustrate this, considerthe toy example shown in Figure 8 which shows a flowchart training framework of an I-framemodule and a P- / B-frame module with a two-controller quality level setup where a final stepcomprises combining modules trained at different target qualities.On the left side of Figure 8 is the box 801 representing the P- / B-frame module training. Onthe right side of Figure 8 is a box 802 representing the I-frame module training. Note that theP- / B-frame module training relies on a source of I-frames to produce the initial I-frame onwhich the GOP of P- / B-frames will be based and so to facilitate this a placeholder I-framemodule is used that may be trained jointly with the networks of the P- / B-frame module. Thisplaceholder I-frame is eventually replaced with the the main I-frame module trained in box802. Thus there are three sets of quality level targets to consider: the main I-frame moduletargets, the placeholder I-frame module targets, and the P- / B-frame module targets.As a first step, the networks of the placeholder I-frame module are initialised 803 and thestarting distortion targets of the first controller are set 804. For example, the distortion targetsof a toy example may be MSE targets having some set of values, for example 90, 120, 150,180, 220, or some other targets. Training 805 is then started with the first controller usingthose MSE targets which has the effect of controlling the ^^ values associated with the controlunits as described above. After some predetermined number of training steps, the secondcontroller (e.g. the meta controller as described above is run with some predetermined bitratetargets), which has the effect of updating 806 the distortion targets based on how much aboveor below the rate targets the placeholder I-frame module is performing. The training loop withthe first controller is then continued but now using 807 the new distortion targets set by themetacontroller.This training loop 805, 806, 807 is repeated for some predetermined number of steps until thetwo-controller set up causes the weights of the networks of the placeholder I-frame module andthe values of the associated control units to converge to the desired rate distortion performanceas set by the distortion and rate targets of the controllers. This convergence may be determinedempirically or be assumed to have occurred after some predetermined number of training steps.Once the initial training of the placeholder I-frame module is complete, the same process isrepeated for the P-B-frame module, using the placeholder I-frame module as the source ofI-frames. That is, in a first step, the networks of the P- / B-frame module are initialised 808 andthe initial distortion targets are set 809. As will be appreciated, following the teachings ofconcepts 1 and 2 above, the initial distortion targets may be higher or lower than the placeholderI-frame module distortion targets. For example, if we are training the P- / B-frame module usinga lower quality I-frame module as the source of I-frames for the GOP, then the initial distortiontargets of the P- / B-frame module may be lower than those of the placeholder I-frame module(e.g. lower than the initial distortion targets set in step 804, or lower than the distortion targetsthat the controller converged to in the repeated training loop of steps 806, 807). For example,the initial MSE targets of the P- / B-frame module may be half those of the initial targets of theplaceholder I-frame module, or some other fraction thereof, such as MSE targets of 45, 60, 75,90, 110 (half of 90, 120, 150, 180, 220).Training 810 is then started with the first controller using those MSE targets which has theeffect of controlling the ^^ values associated with the control units as described above. Aftersome predetermined number of training steps, the second controller (e.g. the meta controller asdescribed above is run with some predetermined rate targets), which has the effect of updating811 the distortion targets based on how much above or below the rate targets the P- / B-frame isperforming. The training loop with the first controller is then continued but now using 812 thenew distortion targets set by the metacontroller.This training loop 810, 811, 812 is repeated for some predetermined number of steps until thetwo-controller set up causes the weights of the networks of the P- / B-frame module and thevalues of the associated control units to converge to the desired rate distortion performance asset by the distortion and rate targets of the controllers. This convergence may be determinedempirically or be assumed to have occurred after some predetermined number of training steps.It is envisaged that the training of the placeholder I-frame module may be performed jointlywith the P- / B-frame module starts, rather than separately. This allows the weights and controlunit values to be updated using the respective control loops based on the pipeline as a whole,for example with the controllers updating the ^^ values and distortion targets based on how faraway the distortion and rate scores of the pipeline as a whole are to those targets. Finally, as thepurpose of the placeholder I-frame module is primarily to manipulate how the P-frame / B-framemodule responds in training, its controller’s rate and / or distortion targets can be changedaccording to some predetermined schedule away from whatever values they have convergedto or have been set at at that point in training. For example, when the metacontroller ofthe P- / B-frame module is run 811, the MSE targets of the placeholder I-frame module maybe reduced, for example back down to 45, 60. 75, 90, 110. This approach results in twotraining phases. The first phase is based on the methods of concepts 1 and 2 above wherethe placeholder I-frame module is of lower quality. Then, in the second phase, when themetacontroller is run to change the distortion targets based on rate focussed training objective,the distortion targets of the placeholder I-frame module targets may be reduced, making itproduce higher quality I-frames again while the training is focussed on bitrate. The inventorshave found that this two phase training process helps to improve training stability.Once training is complete, for example when the rate and distortion scores have converged tosome stable level as determined empirically or assumed to have happened after a predeterminednumber of steps, the final distortion targets that the metacontroller set, and the rate targetscorresponding to this are stored for use later when training the main I-frame module whichwill eventually be grafted onto the now-trained P- / B-frame module 813. Indeed, it is thesefinal P- / B-frame module targets from which an optimal relative quality level between theI-frame module and P- / B-frame module can be derived to maximise the advantageous effect ofconcepts 1 and 2 above.To illustrate this, consider the main I-frame module training steps as illustrated in box 802. Themain I-frame module is what will be grafted onto the P- / B-frame module and can accordinglybe trained separately as it has no dependence on other frames of a GOP. As a first step, thenetworks of the main I-frame module are initialised 814 and the starting distortion targetsof the first controller are set 815. The inventors have found that, generally, the first and / orsecond controllers are highly effective at tuning the rate and distortion performance whenthe starting targets are approximately in line with where convergence is going to end up, butstruggle to do this when the initial distortion targets are very far removed from where themetacontroller is trying to push them for a given bitrate target. Accordingly, the setting ofthe initial distortion target can have significant effects on training stability and the ability toconverge to the desired rate and distortion targets. Empirically determining the starting targetswith sweeps of different combinations of rate and distortion targets is not practically feasiblegiven the compute requirements needed to do this.It is at this point where the final distortion and bitrate target values of the P- / B-frame modulebecome useful. In particular, the inventors have found that their values, or some fraction ormultiple thereof, provide a starting set of targets for the main I-frame module that have goodtraining stability and generally end up facilitating good convergence to a global optimumrate distortion performance during training. That is, the inventors have found that the finaldistortion targets (or a fraction or multiple thereof) of the trained P- / B-frame module ontowhich the main I-frame module is to be grafted are a good approximation of where the mainI-frame module (with metacontroller, or without) will try to push the distortion targets duringtraining. Accordingly, using these, or a set of targets derived from these (for example, a fractionor multiple thereof based on a desired bitrate ratio of how many bits are used by the P- / B-framemodule relative to the main I-frame module), means a good set of starting distortion targets forthe I-frame module controllers can be obtained without needing to perform very large numbersof starting target sweeps. For example, if it is desired that the I-frame module is to use aroundthree times as many bits as the P- / B-frame module, then the P- / B-frame module can be trainedfirst, the final distortion targets the metacontroller converged to for a given set of rate targetsduring that training loop can be stored, and, when setting the initial distortion targets for themain I-frame module controller, those final targets from the P- / B-frame module or a fraction ormultiple thereof can be used as a starting target to almost always achieve good training stabilityand convergence to a global optimum. In very high level terms, the final distortion targets (ora fraction or multiple thereof) of the P- / B-frame module controllers are good starting targetsfor the main I-frame module controller to increase the chance of training being successful.Thus, returning to figure 8, the final targets 815 of the P- / B-frame training are used to set816 the initial distortion targets of the controller for the main I-frame module training. Then,as with the P- / B-frame module training, training 817 with the first controller is commencedwhich has the effect of controlling the ^^ values associated with the control units as describedabove. After some predetermined number of training steps, the second controller (e.g. themeta controller as described above) is run with some predetermined rate targets, which hasthe effect of updating 819 the distortion targets based on how much above or below the ratetargets the main I-frame module is performing. The training loop with the first controller isthen continued but now using 818 the new distortion targets set by the metacontroller.This training loop 817, 818, 819 is repeated for some predetermined number of steps until thetwo-controller set up causes the weights of the networks of the main I-frame module and thevalues of the associated control units to converge to the desired rate distortion performance asset by the distortion and rate targets of the controllers. This convergence may be determinedempirically or be assumed to have occurred after some predetermined number of training steps.Finally, the trained main I-frame module 820 is combined with or grafted onto 803 withthe trained P- / B-frame module 813 to produce the final pipeline where the quality of themain I-frame module is higher or lower than that of the P- / B-frame module by virtue of thedistortion targets of the main I-frame module being based on whatever the distortion targetsthe P- / B-frame module converged to. The relative quality difference is thus linked directly towhatever fraction or multiple of those final distortion targets the main I-frame module wastrained with, in a non-linear relationship that may be experimentally determined.To help further illustrate this, consider the following toy example. A placeholder I-framemodule and P- / B-frame module are initialised and starting MSE targets of 45, 60, 75, 90, 110and bitrate targets of 1, 2, 3, 4, 5 Mbit / sec are set for the two controllers. The initial training ofboth modules jointly with the first controller changes the ^^ values associated with the controlunits based on how far away the MSE scores are from the targets. When the metacontrolleris run, it may turn out that the bitrate that the networks are achieving with the starting MSEtargets is far too high, this large deviation causes the controller to increases the MSE targets,for example to 65, 80, 95, 110, 130. The training loop proceeds and the ^^ results in a differentset of control unit values and so on. For the sake of this toy example, lets assume that this iswhere training of the P- / B-frame module stops. We are left with a set of final MSE targets thatthe metacontroller caused the first controller to settle on. We then take a fraction or multipleof the values of these final targets where the multiple is based on how much higher or lowerthe quality level of the I-frame module being trained is intended to be, for example this mightbe one half of the values of the targets, or some other fraction, and then these are used as thestarting distortion targets for main I-frame module training. That is, we now set the MSEtargets for the main I-frame module to 32.5, 40, 47.5, 55, 65, and proceed with the trainingloop in the usual way. Eventually the training of the main I-frame module will complete andwe combine the trained main I-frame module and trained P- / B-frame module to produce apipeline where the main I-frame module is of higher quality, thereby achieving the benefits ofconcepts 1 and 2 described above, while having training stability due to the guess work ofthe starting distortion targets of the main I-frame module controller being based on the finaltargets of the P- / B-frame module.With this approach, the guess work is taken out of setting the starting distortion targets for thetraining of the I-frame module that will be grafted onto the P- / B-frame module as a replacementfor the placeholder I-frame module used for training the P- / B-frame modules, thereby ensuringtraining stability and convergence to weights and control unit values that achieve the desiredI-frame to P- / B-frame combination performance.The subject matter and the functional operations described in this specification can beimplemented in digital electronic circuitry, in tangibly-embodied computer software orfirmware, in computer hardware, including the structures disclosed in this specification andtheir structural equivalents, or in combinations of one or more of them. The subject matterdescribed in this specification can be implemented as one or more computer programs, i.e.,one or more modules of computer program instructions encoded on a tangible non transitoryprogram carrier for execution by, or to control the operation of, data processing apparatus.Alternatively or in addition, the program instructions can be encoded on an artificially generatedpropagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, thatis generated to encode information for transmission to suitable receiver apparatus for executionby a data processing apparatus. The computer storage medium can be a machine-readablestorage device, a machine-readable storage substrate, a random or serial access memory device,or a combination of one or more of them. The computer storage medium is not, however, apropagated signal.The term “data processing apparatus” encompasses all kinds of apparatus, devices, andmachines for processing data, including by way of example a programmable processor, acomputer, or multiple processors or computers. The apparatus can include special purposelogic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specificintegrated circuit). The apparatus can also include, in addition to hardware, code that createsan execution environment for the computer program in question, e.g., code that constitutesprocessor firmware, a protocol stack, a database management system, an operating system, ora combination of one or more of them.A computer program (which may also be referred to or described as a program, software, asoftware application, a module, a software module, a script, or code) can be written in anyform of programming language, including compiled or interpreted languages, or declarative orprocedural languages, and it can be deployed in any form, including as a stand alone program oras a module, component, subroutine, or other unit suitable for use in a computing environment.A computer program may, but need not, correspond to a file in a file system. A program can bestored in a portion of a file that holds other programs or data, e.g., one or more scripts storedin a markup language document, in a single file dedicated to the program in question, or inmultiple coordinated files, e.g., files that store one or more modules, sub programs, or portionsof code. A computer program can be deployed to be executed on one computer or on multiplecomputers that are located at one site or distributed across multiple sites and interconnected bya communication network.The processes and logic flows described in this specification can be performed by one or moreprogrammable computers executing one or more computer programs to perform functionsby operating on input data and generating output. The processes and logic flows can also beperformed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g.,an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).Computers suitable for the execution of a computer program include, by way of example,can be based on general or special purpose microprocessors or both, or any other kind ofcentral processing unit. Generally, a central processing unit will receive instructions and datafrom a read only memory or a random access memory or both. The essential elements ofa computer are a central processing unit for performing or executing instructions and oneor more memory devices for storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer data to, or both, one or moremass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks.However, a computer need not have such devices. Moreover, a computer can be embedded inanother device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio orvideo player, a VR headset, a game console, a Global Positioning System (GPS) receiver, aserver, a mobile phones, a tablet computer, a notebook computer, a music player, an e-bookreader, a laptop or desktop computer, a PDAs, a smart phone, or other stationary or portabledevices, that includes one or more processors and computer readable media, or a portablestorage device, e.g., a universal serial bus (USB) flash drive, to name just a few.Computer readable media suitable for storing computer program instructions and data includeall forms of non-volatile memory, media and memory devices, including by way of examplesemiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magneticdisks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM andDVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in,special purpose logic circuitry.The subject matter described in this specification can be implemented in a computing systemthat includes a back end component, e.g., as a data server, or that includes a middlewarecomponent, e.g., an application server, or that includes a front end component, e.g., a clientcomputer having a graphical user interface or a Web browser through which a user can interactwith an implementation of the subject matter described in this specification, or any combinationof one or more such back end, middleware, or front end components. The components of thesystem can be interconnected by any form or medium of digital data communication, e.g., acommunication network. Examples of communication networks include a local area network(“LAN”) and a wide area network (“WAN”), e.g., the Internet.The computing system can include clients and servers. A client and server are generally remotefrom each other and typically interact through a communication network. The relationship ofclient and server arises by virtue of computer programs running on the respective computersand having a client-server relationship to each other.While this specification contains many specific implementation details, these should beconstrued as descriptions of features that may be specific to particular examples of particularinventions. Certain features that are described in this specification in the context of separateexamples can also be implemented in combination in a single example. Conversely, variousfeatures that are described in the context of a single example can also be implemented inmultiple examples separately or in any suitable subcombination.Similarly, while operations are depicted in the drawings in a particular order, this should notbe understood as requiring that such operations be performed in the particular order shownor in sequential order, or that all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processing may be advantageous.Moreover, the separation of various system modules and components in the examples describedabove should not be understood as requiring such separation in all examples, and it should beunderstood that the described program components and systems can generally be integratedtogether in a single software product or packaged into multiple software products.Finally, whilst both concept 1 and concept 2 have been described in the context of I-, P- andB-frames. It is envisaged that the above principals may apply generally to any image and / orvideo compression scheme irrespective of whether the input image(s) are I-, P- and / or B-framesof some input image sequence. Accordingly, the present disclosure may further be definedmore generally by the following clauses, capturing the ideas of algorithms 2 and 3 in a moregeneral context than just I-, P- and / or B-frames:A method of training one or more neural networks, the one or more neural networks being foruse in lossy image or video encoding, transmission and decoding, the method comprising thesteps of:receiving a first input image at a first computer system;encoding the first input image using a first neural network to produce a latent representa-tion; decoding the latent representation using a second neural network to produce a first outputimage, wherein the first output image is an approximation of the first input image;receiving a second input image at the first computer system;with a third neural network, producing a latent representation of optical flow informationand / or residual information, the optical flow information and / or residual information beingindicative of a difference between the first input image and the second input image;with a fourth neural network, producing a second output image using the optical flowinformation and / or residual information, wherein the second output image is an approximationof the second input image;evaluating a function based on a difference between the second input image and thesecond output image;updating the parameters of the third neural network and the fourth neural network basedon the evaluated function;repeating the above steps using a set of input images to produce a trained third neuralnetwork and a trained fourth neural network; andreplacing the first neural network and second neural network with a fifth neural networkand a sixth neural network.The method described above, wherein the fifth neural network and the sixth neural networkcomprise pre-trained neural networks configured to produce a better approximation of the firstinput image than the first neural network and the second neural network.The method described above, wherein a quality of the approximation is defined by a differencescore.The method described above, wherein the difference score comprises a mean squared error,and wherein the better approximation is defined by a smaller mean squared error between thefirst input image and the first output image.A method of training one or more neural networks, the one or more neural networks being foruse in lossy image or video encoding, transmission and decoding, the method comprising thesteps of:receiving a first input image at a first computer system;encoding the first input image using a first neural network to produce a latent representa-tion; decoding the latent representation using a second neural network to produce a first outputimage, wherein the first output image is an approximation of the first input image;receiving a second input image at the first computer system;with a third neural network, producing a latent representation of optical flow informationand / or residual information, the optical flow information and / or residual information beingindicative of a difference between the first input image and the second input image;with a fourth neural network, producing a second output image using the optical flowinformation and / or residual information, wherein the second output image is an approximationof the second input image;evaluating a function based on a difference between the second input image and thesecond output image, and / or the first input image and the first output image;updating the parameters of the first neural network, the second neural network, the thirdneural network and the fourth neural network based on the evaluated function;repeating the above steps using a set of input images for a first number of steps;modifying the function by changing a weighting of a rate term relative to a distortionterm of the function; andrepeating the above steps for a second number of steps.The method descrobed above, comprising freezing the parameters of the third neural networkand the fourth neural network after repeating the steps for the first number of times.The method described above, wherein changing the weighting of the rate term relative to thedistortion term comprises increasing the weighting of the distortion term relative to the rateterm to specify a relatively lower distortion loss training objective.Finally, for completeness, when setting the I-frame module starting targets based on the finalP- / B-frame module targets, the inventors have found that a fraction of substantially 1 / 3 of thefinal P- / B-frame module distortion target achieves good performance in the bitrate range of0.5-6Mbit.sec. Although other fractions and multiples are also envisaged for different bit rateranges.

Claims

CLAIMS1. A method of training one or more neural networks, the one or more neural networks beingfor use in lossy image or video encoding, transmission and decoding, the method comprisingthe steps of:using a first I-frame module to produce an output I-frame from a first input image, theI-frame module comprising one or more neural networks;using a P- and / or B-frame module to produce an output P- or B-frame from a secondinput image and the output I-frame, the P- and / or B-frame module comprising one or moreneural networks;evaluating a function based on a difference between the second input image and theoutput P- or B- frame;updating the parameters of the one or more neural networks of the P- and / or B-framemodule based on the evaluated function;repeating the above steps using a set of input images to produce trained neural networksof the P- and / or B-frame module; andswapping the first I-frame module with a second I-frame module.

2. The method of claim 1 comprising using the second I-frame module and the P- and / orB-frame module to encode and decode a plurality of images.

3. The method of claims 1 or 2, wherein the second I-frame module is configured to produce anoutput I-frame that is a better approximation of the first input image than an I-frame producedby the first I-frame module.

4. The method of claim 3, wherein a quality of the approximation is defined by a differencescore.

5. The method of claim 4, wherein the difference score comprises a mean squared error, andwherein the better approximation is defined by a smaller mean squared error between an inputimage and an output I-frame.

6. The method of any of claims 1 to 5, wherein the function comprises a rate term, a distortionterm, and wherein at least one of the rate term or distortion term is weighted by a Lagrangemultiplier.

7. The method of claim 6, wherein the Lagrange multiplier comprises a learned Lagrangemultiplier.

8. The method of any of claims 1 to 7, wherein the set of input images comprises groups ofpictures (GOPs) of fewer than 20 frames.

9. The method of any of claims 1 to 8, comprising swapping the second I-frame module witha third I-frame module.

10. The method of claim 9, comprising selecting the second I-frame module or the thirdI-frame module based on a property of the first input image or a third input image.

11. The method of claim 10, wherein selecting comprises applying a classifier to the firstinput image and / or the third input image to produce a classification output, and wherein saidselecting is based on the classification output.

12. The method of any of claims 9 to 11, wherein the first input image comprises an inputimage of a first image type, and wherein the third input image comprises an input image of asecond type.

13. The method of claim 12, wherein the first image type is a high-motion image type, andwherein the second image type is a low-motion image type.

14. A method of training one or more neural networks, the one or more neural networks beingfor use in lossy image or video encoding, transmission and decoding, the method comprisingthe steps of:using a first I-frame module to produce an output I-frame from a first input image, theI-frame module comprising one or more neural networks;using a P- and / or B-frame module to produce an output P- or B-frame from a secondinput image and the output I-frame, the P- and / or B-frame module comprising one or moreneural networks;evaluating a function based on a difference between the first input image and the firstoutput I-frame and / or a difference between the second input image and the output P- or B-frame; updating the parameters of the one or more neural networks of the P- and / or B-framemodule based on the evaluated function;updating the parameters of the one or more neural networks of the I-frame module basedon the evaluated functionrepeating the above steps using a set of input images to produce trained neural networksof the P- and / or B-frame module and trained neural networks of the I-frame module.

15. The method of claims 14, comprising freezing the parameters of the one or more neuralneural networks of the P- and / or B-frame module after a number of said repeated steps.

16. The method of claim 15, comprising modifying the function after said freezing.

17. The method of claim 16, wherein the function comprises a rate term, a distortion term,wherein at least one of the rate term or distortion term is weighted by a Lagrange multiplier,and wherein modifying the function comprises modifying the Lagrange multiplier.

18. The method of claim 17, comprising modifying the Lagrange multiplier to increase therelative weight of the distortion term relative to other terms of the function.

19. A data processing apparatus configured to perform the method of any of claims 1 to 18.

20. A computer program comprising instructions which, when the program is executed by acomputer, cause the computer to carry out the method of any of claims 1 to 18.

21. A computer-readable storage medium comprising instructions which, when executed by acomputer, cause the computer to carry out the method of any of claims 1 to 18.

22. A method of lossy image or video encoding and decoding, the method comprising thesteps of:receiving a first input image at a first computer system;using a first I-frame module to produce a first output I-frame from the first input image,the I-frame module comprising one or more neural networks;receiving a second input image at the first computer system;using a P- and / or B-frame module to produce a first output P- or B-frame using the firstoutput I-frame and the second input image, the P- and / or B-frame module comprising one ormore neural networks;swapping the first I-frame module with a second I-frame module;receiving a third input image at the first computer system;using the second I-frame module to produce a second output I-frame from the third inputimage, the I-frame module comprising one or more neural networks; andreceiving a fourth input image at the first computer system; andusing the P- and / or B-frame module to produce a second output P- or B-frame using thesecond I-frame and the fourth input image.

23. The method of claim 22, wherein the first input image comprises an input image of a firstimage type, and wherein the third input image comprises an input image of a second type.

24. The method of claim 23, wherein the first image type is a high-motion image type, andwherein the second image type is a low-motion image type.

25. The method of any of claims 22 to 24, comprising selecting the first I-frame module and / orthe second I-frame module from a set of I-frame modules.

26. The method of claim 25, wherein selecting comprises applying a classifier to the firstinput image and / or the second input image to produce a classification result, and wherein saidselecting is based on the classification result.

27. The method of claim 25, wherein selecting comprises receiving meta data associated withthe first input image and / or the second input image, and wherein said selecting is based on themeta data.

28. A data processing apparatus configured to perform the method of any of claims 22 to 27.

29. A computer program comprising instructions which, when the program is executed by acomputer, cause the computer to carry out the method of any of claims 22 to 27.

30. A computer-readable storage medium comprising instructions which, when executed by acomputer, cause the computer to carry out the method of any of claims 22 to 27.

31. A method of training one or more neural networks, the one or more neural networks beingfor use in lossy image or video encoding, transmission and decoding, the method comprisingthe steps of:receiving a first input image at a first computer system;encoding the first input image using a first neural network to produce a latent representa-tion;decoding the latent representation using a second neural network to produce a first outputimage, wherein the first output image is an approximation of the first input image;receiving a second input image at the first computer system;with a third neural network, producing a latent representation of optical flow informationand / or residual information, the optical flow information and / or residual information beingindicative of a difference between the first input image and the second input image;with a fourth neural network, producing a second output image using the optical flowinformation and / or residual information, wherein the second output image is an approximationof the second input image;evaluating a function based on a difference between the second input image and thesecond output image;updating the parameters of the third neural network and the fourth neural network basedon the evaluated function;repeating the above steps using a set of input images to produce a trained third neuralnetwork and a trained fourth neural network; andreplacing the first neural network and second neural network with a fifth neural networkand a sixth neural network.

32. The method of claim 31, wherein the fifth neural network and the sixth neural networkcomprise pre-trained neural networks configured to produce a better approximation of the firstinput image than the first neural network and the second neural network.

33. The method of claim 32, wherein a quality of the approximation is defined by a differencescore.

34. The method of claim 33, wherein the difference score comprises a mean squared error,and wherein the better approximation is defined by a smaller mean squared error between thefirst input image and the first output image.

35. A method of training one or more neural networks, the one or more neural networks beingfor use in lossy image or video encoding, transmission and decoding, the method comprisingthe steps of:receiving a first input image at a first computer system;encoding the first input image using a first neural network to produce a latent representa-tion; decoding the latent representation using a second neural network to produce a first outputimage, wherein the first output image is an approximation of the first input image;receiving a second input image at the first computer system;with a third neural network, producing a latent representation of optical flow informationand / or residual information, the optical flow information and / or residual information beingindicative of a difference between the first input image and the second input image;with a fourth neural network, producing a second output image using the optical flowinformation and / or residual information, wherein the second output image is an approximationof the second input image;evaluating a function based on a difference between the second input image and thesecond output image, and / or the first input image and the first output image;updating the parameters of the first neural network, the second neural network, the thirdneural network and the fourth neural network based on the evaluated function;repeating the above steps using a set of input images for a first number of steps;modifying the function by changing a weighting of a rate term relative to a distortionterm of the function; andrepeating the above steps for a second number of steps.

36. The method of claim 35 comprising freezing the parameters of the third neural networkand the fourth neural network after repeating the steps for the first number of times.

37. The method of claim 35 or 36, wherein changing the weighting of the rate term relative tothe distortion term comprises increasing the weighting of the distortion term relative to therate term to specify a relatively lower distortion loss training objective.

38. A data processing apparatus configured to perform the method of any of claims 31 to 37.

39. A computer program comprising instructions which, when the program is executed by acomputer, cause the computer to carry out the method of any of claims 31 to 37.

40. A computer-readable storage medium comprising instructions which, when executed by acomputer, cause the computer to carry out the method of any of claims 31 to 37.

41. A method of training one or more neural networks, the one or more neural networks beingfor use in lossy image or video encoding, transmission and decoding, the method comprisingthe steps of:using a P- and / or B-frame module to produce an output P- or B-frame from a firstimage and a second image, the P- and / or B-frame module having a first distortion target andcomprising one or more neural networks;evaluating a first function based on a difference between the second image and the outputP- or B- frame;updating the parameters of the one or more neural networks of the P- and / or B-framemodule based on the evaluated function, and updating the first distortion target;repeating the above steps using a set of input images to produce trained neural networksof the P- and / or B-frame module and a final first distortion target;using an I-frame module to produce an output I-frame from a third image, the I-framemodule having a second distortion target and comprising one or more neural networks, whereinthe second distortion target is based on the final first distortion target;evaluating a second function based on a difference between the third image and theoutput I-frame; andupdating the parameters of the one or more neural networks of the I-frame module basedon the evaluated function to produce trained neural networks of the I-frame module.

42. The method of claim 41, comprising setting a first bit rate target, and wherein said updatingthe first distortion target is based on the the first bit rate target.

43. The method of claim 42, wherein said updating the first distortion target comprisesevaluating a bit rate achieved when producing the output P- or B-frame and comparing said bitrate against the first bit rate target.

44. The method of claim 43, wherein said updating the first distortion target comprisesincreasing the first distortion target when the bit rate achieved is higher than the first bit ratetarget, or decreasing the first distortion target when the bit rate achieved is lower than the firstbit rate target.

45. The method of claim 44, wherein an amount and direction of an update to the first distortiontarget when performing said updating is based on how much higher or lower the the bit rateachieved is to the first bit rate target.

46. The method of any of claims 43 to 44, wherein said updating the first distortion targetcomprises running a proportional, integral and / or derivative controller with a value of theachieved bit rate as a process variable, and the first distortion target as the control variable,47. The method of any of claims 41 to 46, comprising using a placeholder I-frame module toto produce the first image, the placeholder I-frame module having a different distortion targetto the second distortion target.

48. The method of claim 47, comprising discarding the placeholder I-frame module andcombining the trained I-frame module with the trained P- and / or B-frame module to produce avideo compression pipeline.

49. The method of any of claims 41 to 48, wherein the second distortion target is a predeterminedfraction or multiple of the final first distortion target.

50. The method of claim 49, wherein the second distortion target is substantially 1 / 3 of thefinal first distortion target.

51. A method for lossy image or video encoding, transmission and decoding, the methodcomprising the steps of:receiving an input image at a first computer system;encoding the input image using a first trained neural network to produce a latentrepresentation; transmitting the latent representation to a second computer system;decoding the latent representation using a second trained neural network to produce anoutput image, wherein the output image is an approximation of the input image;wherein the first neural network and the second neural network are trained according toany of the methods of claims 41 to 50.

52. A method for lossy image or video encoding and transmission, the method comprising thesteps of:receiving an input image at a first computer system;encoding the input image using a first trained neural network to produce a latentrepresentation; andtransmitting the latent representation;wherein the first neural network is trained according to any of the methods of claims 41to 50.

53. A method for lossy image or video receipt and decoding, the method comprising the stepsof: receiving a latent representation transmitted by a first computer system at a secondcomputer system, the latent representation corresponding to an input image; anddecoding the latent representation using a second trained neural network to produce anoutput image, wherein the output image is an approximation of the input image;wherein the second neural network is trained according to any of the methods of claims41 to 50.

54. A data processing system configured to perform the method of any one of claims 41 to 53.

55. A data processing apparatus configured to perform the method of claims 41 to 53.

56. A computer program comprising instructions which, when the program is executed by acomputer, cause the computer to carry out the method of claims 41 to 53.

57. A computer-readable storage medium comprising instructions which, when executed by acomputer, cause the computer to carry out the method of claims 41 to 53.

Images