Method and data processing system for lossy image or video encoding, transmission and decoding

Abstract

A method for lossy video encoding, transmission and decoding, the method comprising the steps of: receiving an input frame and a previous frame at a first computer system; encoding the input frame and an input based on the previous frame using a first trained neural network to produce a first latent representation; encoding the input frame using a second trained neural network to produce a second latent representation; transmitting the first and the second latent representation to a second computer system; decoding the first latent representation using a third trained neural network to obtain an output flow map between the input frame and the previous frame; and decoding the second latent representation and an input based on the output flow map using a fourth trained neural network to produce an output frame, wherein the output frame is an approximation of the input frame.

Description

[0001] Method and data processing system for lossy image or video encoding, transmission and decoding BACKGROUND This invention relates to a method and system for lossy image or video encoding, transmission and decoding, a method, apparatus, computer program and computer readable storage medium for lossy image or video encoding and transmission, and a method, apparatus, computer program and computer readable storage medium for lossy image or video receipt and decoding. There is increasing demand from users of communications networks for images and video content. Demand is increasing not just for the number of images viewed, and for the playing time of video; demand is also increasing for higher resolution content. This places increasing demand on communications networks and increases their energy use because of the larger amount of data being transmitted. To reduce the impact of these issues, image and video content is compressed for transmission across the network. The compression of image and video content can be lossless or lossy compression. In lossless compression, the image or video is compressed such that all of the original information in the content can be recovered on decompression. However, when using lossless compression there is a limit to the reduction in data quantity that can be achieved. In lossy compression, some information is lost from the image or video during the compression process. Known compression techniques attempt to minimise the apparent loss of information by the removal of information that results in changes to the decompressed image or video that is not particularly noticeable to the human visual system. JPEG, JPEG2000, AVC, HEVC and AVI are examples of compression processes for image and / or video files. In general terms, known lossy image compression techniques use the spatial correlations between pixels in images to remove redundant information during compression. For example, in an image of a blue sky, if a given pixel is blue, there is a high likelihood that the neighbouring pixels, and their neighbouring pixels, and so on, are also blue. There is accordingly no need to retain all the raw pixel data. Instead, we can retain only a subset of the pixels which take up fewer bits and infer the pixel values of the other pixels using information derived from spatial correlations. A similar approach is applied in known lossy video compression techniques. That is, spatial correlations between pixels allow the removal of redundant information during compression. However, in video compression, there is further information redundancy in the form of temporal correlations. For example, in a video of an aircraft flying across a blue-sky background, most of the pixels of the blue sky do not change at all between frames of the video. The most of the blue sky pixel data for the frame at position t = 0 in the video is identical to that at position t = 10. Storing this identical, temporally correlated, information is inefficient. Instead, only the blue sky pixel data for a subset of the frames is stored and the rest are inferred from information derived from temporal correlations. In the realm of lossy video compression in particular, the removal of redundant temporally correlated information in a video sequence is known inter-frame redundancy. One technique using inter-frame redundancy that is widely used in standard video compression algorithms involves the categorization of video frames into three types: I-frames, P-frames, and B-frames. Each frame type carries distinct properties concerning their encoding and decoding process, playing different roles in achieving high compression ratios while maintaining acceptable visual quality. I-frames, or intra-coded frames, serve as the foundation of the video sequence. These frames are self-contained, each one encoding a complete image without reference to any other frame. In terms of compression, I-frames are least compressed among all frame types, thus carrying the most data. However, their independence provides several benefits, including being the starting point for decompression and enabling random access, crucial for functionalities like fast-forwarding or rewinding the video. P-frames, or predictive frames, utilize temporal redundancy in video sequences to achieve greater compression. Instead of encoding an entire image like an I-frame, a P-frame represents the difference between itself and the closest preceding I- or P-frame. The process, known as motion compensation, identifies and encodes only the changes that have occurred, thereby significantly reducing the amount of data transmitted. Nonetheless, P-frames are dependent on previous frames for decoding. Consequently, any error during the encoding or transmission process may propagate to subsequent frames, impacting the overall video quality. B-frames, or bidirectionally predictive frames, represent the highest level of compression. Unlike P-frames, B-frames use both the preceding and following frames as references in their encoding process. By predicting motion both forwards and backwards in time, B-frames encode only the differences that cannot be accurately anticipated from the previous and next frames, leading to substantial data reduction. Although this bidirectional prediction makes B-frames more complex to generate and decode, it does not propagate decoding errors since they are not used as references for other frames. Artificial intelligence (AI) based compression techniques achieve compression and decompression of images and videos through the use of trained neural networks in the compression and decompression process. Typically, during training of the neutral networks, the difference between the original image and video and the compressed and decompressed image and video is analyzed and the parameters of the neural networks are modified to reduce this difference while minimizing the data required to transmit the content. However, AI based compression methods may achieve poor compression results in terms of the appearance of the compressed image or video or the amount of information required to be transmitted. An example of an AI based image compression process comprising a hyper-network is described in Ballé, Johannes, et al. “Variational image compression with a scale hyperprior.” arXiv preprint arXiv:1802.01436 (2018), which is hereby incorporated by reference. An example of an AI based video compression approach is shown in Agustsson, E., Minnen, D., Johnston, N., Balle, J., Hwang, S. J., and Toderici, G. (2020), Scale-space flow for end-to-end optimized video compression. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 8503-8512), which is hereby incorporated by reference. A further example of an AI based video compression approach is shown in Mentzer, F., Agustsson, E., Ballé, J., Minnen, D., Johnston, N., and Toderici, G. (2022, November). Neural video compression using gans for detail synthesis and propagation. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVI (pp. 562-578), which is hereby incorporated by reference. SUMMARY According to the present invention there is provided a method for lossy video encoding, transmission and decoding, the method comprising the steps of: receiving an input frame and a previous frame at a first computer system; encoding the input frame and an input based on the previous frame using a first trained neural network to produce a first latent representation; encoding the input frame using a second trained neural network to produce a second latent representation; transmitting the first and the second latent representation to a second computer system; decoding the first latent representation using a third trained neural network to obtain an output flow map between the input frame and the previous frame; and decoding the second latent representation and an input based on the output flow map using a fourth trained neural network to produce an output frame, wherein the output frame is an approximation of the input frame. The output flow map may be used to obtain an output warped frame from a previously decoded frame corresponding to the previous frame and the input based on the output flow map is based on the output warped frame. The output warped frame is transformed to obtain the input based on the output warped frame, wherein the transformation is at least one of the following: down-sampling, a convolution, a space-to-depth operation, a decomposition, a haar transform and a neural network. The method may further comprise the step of, at the first computer system, decoding the first latent representation using a fifth trained neural network to obtain a predicted flow map between the input frame and the previous frame; wherein the second trained neural network additionally encodes an input based on the predicted flow map to produce the second latent representation. The predicted flow map may be used to obtain an input warped frame from the previously decoded frame corresponding to the previous frame and the input based on the predicted flow map is based on the input warped frame. The input based on the previous frame may comprise a flow map between the input frame and the previous frame. The flow map between the input frame and the previous frame may be used to obtain a first warped frame from the previously decoded frame corresponding to the previous frame; and the input based on the previous frame comprises the first warped frame. The previously decoded frame corresponding to the previous frame may be additionally encoded by the first trained neural network to produce the first latent representation. A further previous frame may be received at the first computer system; and a further previously decoded frame corresponding to the further previous frame is additionally encoded by the first trained neural network to produce the first latent representation. A predicted flow map between the further previous frame and the previous frame may be additionally encoded by the first trained neural network to produce the first latent representation. At least two of the previously decoded frame corresponding to the previous frame, further previously decoded frame corresponding to the further previous frame and the predicted flow map between the further previous frame and the previous frame may be concatenated before being encoded by the first trained neural network. The previously decoded frame corresponding to the previous frame may be additionally decoded by the third trained neural network to produce the output flow map. The further previously decoded frame corresponding to the further previous frame may be additionally decoded by the third trained neural network to produce the output flow map. The predicted flow map between the further previous frame and the previous frame may be additionally decoded by the third trained neural network to produce the output flow map. At least two of the previously decoded frame corresponding to the previous frame, further previously decoded frame corresponding to the further previous frame and the predicted flow map between the further previous frame and the previous frame may be concatenated before being decoded by the third trained neural network. At least one of the previously decoded frame corresponding to the previous frame, further previously decoded frame corresponding to the further previous frame and the predicted flow map between the further previous frame and the previous frame may be transformed before being decoded by the third neural network, wherein the transformation is at least one of the following: down-sampling, a convolution, a space-to-depth operation, a decomposition, a haar transform and a neural network. A separate transformation may be used for each of the at least one of the previously decoded frame corresponding to the previous frame, further previously decoded frame corresponding to the further previous frame and the predicted flow map between the further previous frame and the previous frame. A future frame may be received at the first computer system; and an input based on the future frame may be additionally encoded by the first trained neural network to produce the first latent representation. The input based on the future frame may comprise a flow map between the input frame and the future frame. The flow map between the input frame and the future frame may be used to obtain a second warped frame from a previously decoded future frame corresponding to the future frame; and the input based on the future frame comprises the second warped frame. The previously decoded future frame corresponding to the future frame may be additionally encoded by the first trained neural network to produce the first latent representation. The previously decoded future frame corresponding to the future frame may be additionally decoded by the third trained neural network to produce the output flow map. A further future frame may be received at the first computer system; and a further previously decoded future frame corresponding to the further future frame may be additionally encoded by the first trained neural network to produce the first latent representation. A predicted flowmap between the further future frame and the future frame may be additionally encoded by the first trained neural network to produce the first latent representation. At least two of the previously decoded future frame corresponding to the future frame, further previously decoded future frame corresponding to the further future frame and the predicted flow map between the further future frame and the future frame may be concatenated before being encoded by the first trained neural network. The previously decoded future frame corresponding to the future frame may be additionally decoded by the third trained neural network to produce the output flow map. The further previously decoded future frame corresponding to the further future frame may be additionally decoded by the third trained neural network to produce the output flow map. The predicted flow map between the further future frame and the future frame may be additionally decoded by the third trained neural network to produce the output flow map. At least two of the previously decoded future frame corresponding to the future frame, further previously decoded future frame corresponding to the further future frame and the predicted flow map between the further future frame and the future frame may be concatenated before being decoded by the third trained neural network. At least one of the previously decoded future frame corresponding to the future frame, further previously decoded future frame corresponding to the further future frame and the predicted flow map between the further future frame and the future frame may be transformed before being decoded by the third neural network, wherein the transformation is at least one of the following: down-sampling, a convolution, a space-to-depth operation, a decomposition, a haar transform and a neural network. A separate transformation may be used for each of the at least one of the previously decoded future frame corresponding to the future frame, further previously decoded future frame corresponding to the further future frame and the predicted flow map between the further future frame and the future frame. The method may further comprise the steps of: encoding the second latent representation using a sixth trained neural network to produce a hyper-latent representation; additionally transmitting the hyper-latent representation to the second computer system; and decoding the hyper-latent representation using a seventh trained neural network to obtain an output latent representation wherein the output latent representation is used to obtain the output frame; and an input based on the output flow map is additionally encoded by the sixth trained neural network to produce the hyper-latent representation. The output flow map may be used to obtain an output warped frame from a previously decoded frame corresponding to the previous frame and the input based on the output flow map additionally encoded by the sixth trained neural network may be based on the output warped frame. The output warped frame may be transformed to obtain the input based on the output flow map additionally encoded by the sixth trained neural network, wherein the transformation is at least one of the following: down-sampling, a convolution, a space-to-depth operation, a decomposition, a haar transform and a neural network. The transformation may be applied a plurality of times. The input based on the output flow map may be additionally decoded by the seventh trained neural network to produce the output latent representation. The output flow map may be used to obtain an output warped frame from a previously decoded frame corresponding to the previous frame and the input based on the output flow map additionally decoded by the seventh trained neural network may be based on the output warped frame. The output warped frame may be transformed to obtain the input based on the output flow map additionally decoded by the seventh trained neural network, wherein the transformation is at least one of the following: down-sampling, a convolution, a space-to-depth operation, a decomposition, a haar transform and a neural network. The transformation may be applied a plurality of times. The method may further comprise the steps of: encoding the hyper-latent representation using an eighth trained neural network to produce a hyper-hyper-latent representation; additionally transmitting the hyper-hyper-latent representation to the second computer system; and decoding the hyper-hyper-latent representation using a ninth trained neural network to obtain an output hyper-latent representation wherein the output hyper-latent representation is used to obtain the output frame; and an input based on the output flow map is additionally encoded is additionally encoded by the eighth trained neural network to produce the hyper-latent representation. The output flow map may be used to obtain an output warped frame from a previously decoded frame corresponding to the previous frame and the input based on the output flow map additionally encoded by the eighth trained neural network may be based on the output warped frame. The output warped frame may be transformed to obtain the input based on the output flow map additionally encoded by the eighth trained neural network, wherein the transformation is at least one of the following: down-sampling, a convolution, a space-to-depth operation, a decomposition, a haar transform and a neural network. The transformation may be applied a plurality of times. The input based on the output flow map may be additionally decoded by the ninth trained neural network to produce the output latent representation. The output flow map may be used to obtain an output warped frame from a previously decoded frame corresponding to the previous frame and the input based on the output flow map additionally decoded by the ninth trained neural network is based on the output warped frame. The output warped frame may be transformed to obtain the input based on the output flow map additionally decoded by the ninth trained neural network, wherein the transformation is at least one of the following: down-sampling, a convolution, a space-to-depth operation, a decomposition, a haar transform and a neural network. The transformation may be applied a plurality of times. According to the present invention there is provided a method for lossy video encoding, transmission and decoding, the method comprising the steps of: receiving an input frame and a previous frame at a first computer system; encoding the input frame and an input based on the previous frame using a first trained neural network to produce a first latent representation; transmitting the first latent representation to a second computer system; decoding the first latent representation using a second trained neural network to obtain an output flow map between the input frame and the previous frame; and using the output flow map to obtain an output frame, wherein the output frame is an approximation of the input frame. According to the present invention there is provided a method of training one or more neural networks, the one or more neural networks being for use in lossy video encoding, transmission and decoding, the method comprising the steps of: receiving an input frame and a previous frame at a first computer system; encoding the input frame and an input based on the previous frame using a first neural network to produce a first latent representation; encoding the input frame using a second neural network to produce a second latent representation; decoding the first latent representation using a third neural network to obtain an output flow map between the input frame and the previous frame; and decoding the second latent representation and an input based on the output flow map using a fourth neural network to produce an output frame, wherein the output frame is an approximation of the input frame; evaluating a function based on a difference between the output frame and the input frame; updating the parameters of at least one of the first, second, third and fourth neural network based on the evaluated difference; and repeating the above steps using a first set of input images to produce a first, second, third and fourth trained neural network. The function may be additionally based on a difference between the output flow map and a flow map between the input frame and the previous frame. The output flow map may be used to obtain an output warped frame from a previously decoded frame corresponding to the previous frame; and the function may be additionally based on a difference between the output warped frame and the input frame. A first weighting may be applied to the contribution of the difference between the output flow map and the latent flow map to the function and a second weighting may be applied to the contribution of the difference between the output warped frame and the input frame. At least one of the first weighting and the second weighting may be changed between at least one of the repeated steps. The second weighting may be increased between at least one of the repeated steps. According to the present invention there is provided a method for lossy video encoding and transmission, the method comprising the steps of: receiving an input frame and a previous frame at a first computer system; encoding the input frame and an input based on the previous frame using a first trained neural network to produce a first latent representation; encoding the input frame using a second trained neural network to produce a second latent representation; transmitting the first and the second latent representation. According to the present invention there is provided a method for lossy video receipt and decoding, the method comprising the steps of: receiving a first and the second latent representation at a second computer system; decoding the first latent representation using a third trained neural network to obtain an output flow map between the input frame and the previous frame; and decoding the second latent representation and an input based on the output flow map using a fourth trained neural network to produce an output frame, wherein the output frame is an approximation of the input frame. According to the present invention there is provided a method for lossy video encoding, transmission and decoding, the method comprising the steps of: receiving a first frame, a second frame and a third frame at a first computer system; determining a first flow between the first frame and the second frame; determining a second flow between the second frame and the third frame; encoding the first flow and the second flow using a first trained neural network to produce a latent representation; transmitting the latent representation to a second computer system; decoding the latent representation using a second trained neural network to produce a first output flow and a second output flow; and using the first output flow and the second output flow to obtain an output frame, wherein the output frame is an approximation of the second frame. The method may further comprise the steps of: encoding the second frame using a third trained neural network to produce a latent frame representation; additionally transmitting the latent frame representation to the second computer system; and decoding the latent frame representation using a fourth trained neural network to produce the output frame. The first flow may be used to warp a previously decoded first frame corresponding to the first frame to obtain a first warped frame; the second flow may be used to warp a previously decoded third frame corresponding to the third frame to obtain a third warped frame; and the first warped frame and the third warped frame may be additionally encoded by the third trained neural network to produce the latent frame representation. The first warped frame and the third warped frame may be additionally decoded by the fourth trained neural network to produce the output frame. The method may further comprise the step of obtaining a combined flow; wherein the combined flow is based on the first output flow and the second output flow; and the combined flow is used to obtain the output frame. A first weighting may be applied to the first output flow and a second weighting may be applied to the second output flow to obtain the combined flow. The method may further comprise the step of obtaining a combined warped frame; wherein the combined warped frame is based on the first warped frame and the third warped frame; and the combined warped frame is used to obtain the output frame. A first weighting may be applied to the first warped frame and a second weighting may be applied to the third warped frame to obtain the combined warped frame. The method may further comprise the step of: determining a third flow between the first frame and the third frame; wherein an input based on the third flow is additionally encoded by the first trained neural network to produce the latent representation. The input based on the third flow may be additionally encoded by the third trained neural network to obtain the latent frame representation. The input based on the third flow may be additionally decoded by the fourth trained neural network to produce the output frame. The input based on third flow may be a third output flow corresponding to the third flow, the third output flow being previously decoded by the second trained neural network. According to the present invention there is provided a method of training one or more neural networks, the one or more neural networks being for use in lossy video encoding, transmission and decoding, the method comprising the steps of: receiving a first frame, a second frame and a third frame at a first computer system; determining a first flow between the first frame and the second frame; determining a second flow between the second frame and the third frame; encoding the first flow and the second flow using a first trained neural network to produce a latent representation; transmitting the latent representation to a second computer system; decoding the latent representation using a second trained neural network to produce a first output flow and a second output flow; and using the first output flow and the second output flow to obtain an output frame, wherein the output frame is an approximation of the second frame; evaluating a function based on a difference between the output frame and the second frame; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps using a first set of input frames to produce a first trained neural network and a second trained neural network. The method may further comprise the step of obtaining a combined flow; wherein the combined flow is based on the first output flow and the second output flow; a first weighting is applied to the first output flow and a second weighting is applied to the second output flow to obtain the combined flow; and the combined flow is used to obtain the output frame; wherein the first weighting and the second weighting are additionally updated based on the evaluated difference. According to the present invention there is provided a method for lossy video encoding and transmission, the method comprising the steps of: receiving a first frame, a second frame and a third frame at a first computer system; determining a first flow between the first frame and the second frame; determining a second flow between the second frame and the third frame; encoding the first flow and the second flow using a first trained neural network to produce a latent representation; and transmitting the latent representation. According to the present invention there is provided a method for lossy video receipt and decoding, the method comprising the steps of: receiving a latent representation at a second computer system; decoding the latent representation using a second trained neural network to produce a first output flow and a second output flow; and using the first output flow and the second output flow to obtain an output frame, wherein the output frame is an approximation of the second frame. According to the present invention there is provided a method for lossy video encoding, transmission and decoding, the method comprising the steps of: receiving a first frame and a second frame at a first computer system; determining a first flow between the first frame and the second frame; determining a second flow based on the first frame and the second frame; encoding an input based on the first flow and the second flow using a first trained neural network to produce a latent representation; transmitting the latent representation to a second computer system; decoding the latent representation using a second trained neural network to produce an output flow; and using the output flow to obtain an output frame, wherein the output frame is an approximation of the second frame. The method may further comprise the step of: transforming the first frame using the first flow to obtain a warped frame; wherein the second flow is determined between the warped frame and the second frame. The input based on the first flow and the second flow may be the second flow. The step of obtaining the output frame may comprise applying the output flow to a frame corresponding to the first frame to obtain an intermediate frame. The step of obtaining the output frame may comprise applying the first flow to the intermediate frame to obtain the output frame. The second flow may be a point-wise flow; and the input based on the first flow and the second flow may be the first flow subtracted from the second flow. The step of obtaining the output frame may comprise adding the first flow to the output flow to obtain a reconstructed flow. The step of obtaining the output frame may comprise applying the reconstructed flow to a frame corresponding to the first frame to obtain the output frame. The input based on the first flow and the second flow may be the first flow and the second flow. The step of obtaining the output frame may comprise applying the output flow to a frame corresponding to the first frame to obtain the output frame. The method may further comprise the step of dividing each of the first frame and the second frame into a plurality of corresponding sub-frames; and a first flow is determined for each of corresponding sub-frames. The first flow may be a linear function. The first flow may be determined by one of the following methods: a neural network that receives the first frame and the second frame as input and outputs one or more parameters of a matrix defining the first flow, a linear-least squares fit, a Fourier-Mellin transformation and an optimisation that minimizes the photometric error between the first frame and the second frame. The first flow may be a non-linear function. The first flow may comprise a frame transformation. The non-linear function may be a trained neural network. The first flow may be a global flow. The second flow may be a local flow. According to the present invention there is provided a method of training one or more neural networks, the one or more neural networks being for use in lossy video encoding, transmission and decoding, the method comprising the steps of: receiving a first frame and a second frame at a first computer system; determining a first flow between the first frame and the second frame; determining a second flow based on the first frame and the second frame; encoding an input based on the first flow and the second flow using a first neural network to produce a latent representation; transmitting the latent representation to a second computer system; decoding the latent representation using a second neural network to produce an output flow; using the output flow to obtain an output frame, wherein the output frame is an approximation of the second frame; evaluating a function based on a difference between the output frame and the second frame; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps using a first set of input frames to produce a first trained neural network and a second trained neural network. According to the present invention there is provided a method for lossy video encoding and transmission, the method comprising the steps of: receiving a first frame and a second frame at a first computer system; determining a first flow between the first frame and the second frame; determining a second flow based on the first frame and the second frame; encoding an input based on the first flow and the second flow using a first trained neural network to produce a latent representation; transmitting the latent representation. According to the present invention there is provided a method for lossy video receipt and decoding, the method comprising the steps of: receiving a latent representation at a second computer system; decoding the latent representation using a second trained neural network to produce an output flow; and using the output flow to obtain an output frame, wherein the output frame is an approximation of the second frame. According to the present invention there is provided a method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image or video at a first computer system; converting an input based on the input image or video using a first trained neural network to produce a plurality of parameter modifications; encoding the input image or video using a second trained neural network to produce a latent representation; transmitting the parameter modifications and the latent representation to a second computer system; modifying at least one of the parameters of a third trained neural network using the plurality of parameter modifications and decoding the latent representation using the third trained neural network to produce an output image or video, wherein the output image or video is an approximation of the input image or video. The method may further comprise the step of modifying at least one of the parameters of the second trained neural network using the plurality of parameter modifications prior to encoding the input image or video. The plurality of parameter modifications may comprise parameter modifications corresponding to the second neural network and parameter modifications corresponding to the third neural network. The input based on the input image or video converted by the first trained neural network may be the output of a fourth trained neural network that receives the input image or video as an input. The fourth trained neural network may comprise a global average pooling layer. The output of the fourth trained neural network may be downsampled with respect to the input. The first trained neural network may comprise at least one of the following: a dense linear transformation, a sparse linear transformation, a low-rank linear transformation and amulti-layer perceptron network. The output of the first trained neural network may have a larger number of elements than the input. The modification of at least one of the parameters of the third trained neural network using the plurality of parameter modifications may comprise a perturbation of the at least one of the parameters using one of the plurality of parameter modifications. The modification of at least one of the parameters of the third trained neural network using the plurality of parameter modifications may comprise replacing at least one of the parameters with one of the plurality of parameter modifications. The method may further comprise converting the input based on the input image or video using a further trained neural network to produce the plurality of parameter modifications; wherein the first trained neural network outputs a first subset of the plurality of parameter modifications and the further trained neural network outputs a second subset of the plurality of parameter modifications. When the input image or video is an input video, the input based on the input video may be based on at least two frames of the input video. The fourth trained neural network may comprise a convolution across the at least two frames of the input video. The input based on the input video may be obtained by combining the at least two frames of the input video by at least one of the following: an averaging, a recurrent neural network, a convolutional neural network and a transformer network. When the input image or video is an input video, separate outputs of the fourth trained neural network may be obtained for at least two frames of the input video, each output corresponding to a frame of the at least two frames of the input video; and the input based on the input video may be based on the separate outputs of the fourth trained neural network. The input based on the input video may be obtained by combining the separate outputs of the fourth trained neural network by at least one of the following: an averaging, a recurrent neural network, a convolutional neural network and a transformer network. When the input image or video is an input video, separate pluralities of parameter modifications may be obtained for at least two frames of the input video, each separate plurality of parameter modifications corresponding to a frame of the at least two frames of the input video; and the separate pluralities of parameter modifications may be combined before being transmitted to the second computer system; and the combined parameter modifications are the plurality of parameter modification used to modify the third neural network prior to decoding the at least two frames of the input video. The separate pluralities of parameter modifications may be combined by at least one of the following: an averaging, a recurrent neural network, a convolutional neural network and a transformer network. The method may further comprise the steps of: at the first computer system, entropy encoding the parameter modifications and the latent representation to obtain a bitstream; and at the second computer system, entropy decoding the bitstream to retrieve the parameter modifications and the latent representation; wherein the parameter modifications are entropy encoded and decoded using a first probability distribution and the latent representation is entropy encoded and decoded using a second probability distribution; and the first probability distribution is different to the second distribution. The first probability distribution may be a Laplace distribution or a spike and slab distribution. Themethodmay further comprise the steps of: encoding the plurality of parametermodifications using a fifth trained neural network to produce a hyper-latent representation; transmitting the hyper-latent representation to the second computer system; and decoding the hyper-latent representation using a sixth trained neural network and using the output of the sixth trained neural network during the entropy decoding to retrieve the parameter modifications. The first probability distribution may comprise a plurality of probability distributions; and each of the plurality of probability distributions may correspond to a group of parameter modifications corresponding to a group of parameters of the third trained neural network. According to the present invention there is provided a method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image or video at a first computer system; converting an input based on the input image or video using a first neural network to produce a plurality of parameter modifications; encoding the input image or video using a second neural network to produce a latent representation; transmitting the parameter modifications and the latent representation to a second computer system; modifying at least one of the parameters of a third neural network using the plurality of parameter modifications and decoding the latent representation using the third neural network to produce an output image or video, wherein the output image or video is an approximation of the input image or video; evaluating a function based on a difference between the output image or video and the input image or video; updating the parameters of the first neural network based on the evaluated function; and repeating the above steps using a first set of input images or videos to produce a first trained neural network. The function may be additionally based on the rate of the plurality of parameter modifications. The method may further comprise the steps of: at the first computer system, entropy encoding the parameter modifications and the latent representation to obtain a bitstream; and at the second computer system, entropy decoding the bitstream to retrieve the parameter modifications and the latent representation; wherein the parameter modifications are entropy encoded and decoded using a first probability distribution; and the parameters of the first probability distribution are additionally updated based on the evaluated function to produce learned parameters. The second neural network and the third neural network may be trained neural networks. The parameters of the second and third neural network may be additionally updated based on the evaluated functions to produce a second trained neural network and a third trained neural network. According to the present invention there is provided a method for lossy image or video encoding and transmission, the method comprising the steps of: receiving an input image or video at a first computer system; converting an input based on the input image or video using a first trained neural network to produce a plurality of parameter modifications; encoding the input image or video using a second trained neural network to produce a latent representation; transmitting the parameter modifications and the latent representation. According to the present invention there is provided a method for lossy image or video receipt and decoding, the method comprising the steps of: receiving a plurality of parameter modifications and a latent representation at a second computer system; and modifying at least one of the parameters of a third trained neural network using the plurality of parameter modifications and decoding the latent representation using the third trained neural network to produce an output image or video, wherein the output image or video is an approximation of the input image or video. According to the present invention there is provided a method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first neural network to produce a latent representation; decoding the latent representation using a second neural network to produce an output image, wherein the output image is an approximation of the input image; evaluating a function based on a difference between the output image and the input image; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps using a first set of input images to produce a first trained neural network and a second trained neural network; wherein the difference between the output image and the input image is determined based on the output of a neural network acting as a discriminator; the parameters of the neural network acting as a discriminator are additionally updated based on the evaluated function; and the parameters of the neural network acting as a discriminator are updated at a first learning rate; wherein, after at least one of the updates of the parameters of the neural network acting as a discriminator, the first learning rate is updated; and the update to the first learning rate is based on an error of the output of the neural network acting as a discriminator. The error may be based on a difference between the the output of the neural network acting as a discriminator and a target value. The target value may be predetermined. The target value may be modified in at least one of the repeats of the training steps. The update to the first learning rate may be based on a calculation comprising the error. The update to the first learning rate may be based on a calculation comprising the integral of the error. The update to the first learning rate may be based on a calculation comprising the derivative of the error. The contribution of at least one of the error, the integral of the error and the derivative of the error may be scaled by a predetermined value. The update to the first learning rate may be additionally based on an update to the first learning rate in at least one previous repeat of the training steps. The update to the first learning ratemay comprise the step of: determining a currentmodification of the first learning rate based on an error of the output of the neural network acting as a discriminator; and the update to the first learning rate is based on an average of the current modification of the first learning rate and the update to the first learning rate in at least one previous repeat of the training steps. The average may be an exponential moving average. The update to the first learning rate may be based on the output of a gradient descent algorithm which receives the error of the output of the neural network acting as a discriminator as an input. The gradient descent algorithm may be one of the following algorithms: SGD, Adagrand, RMSProp or Adam. According to the present invention there is provided a method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first neural network to produce a latent representation; decoding the latent representation using a second neural network to produce an output image, wherein the output image is an approximation of the input image; evaluating a function based on a difference between the output image and the input image and the rate of the latent representation; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps using a first set of input images to produce a first trained neural network and a second trained neural network; wherein at the contribution of at least one of the difference between the output image and the input image and the rate of the latent representation to the function is scaled by a first parameter; after at least one of the updates of the parameters of the the first neural network and the second neural network, the first parameter is updated; and the update to the first parameter is based on an error of the at least one of the difference between the output image and the input image and the rate of the latent representation. According to the present invention there is provided a method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first trained neural network to produce a latent representation; transmitting the latent representation to a second computer system; decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image; wherein at least one of the first trained neural network and the second trained neural network has been trained according to the method above. According to the present invention there is provided a method for lossy image or video encoding and transmission, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first trained neural network to produce a latent representation; and transmitting the latent representation; wherein the first trained neural network has been trained according to the method above. According to the present invention there is provided a method for lossy image or video receipt and decoding, the method comprising the steps of: receiving a latent representation at a second computer system; and decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image; wherein the second trained neural network has been trained according to the method above. According to the present invention there is provided a method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first trained neural network to produce a latent representation, wherein the latent representation has a probability distribution described by at least one probability parameter; dividing the latent representation into a plurality of sub-latent representations, wherein each sub-latent representation has a sub- probability distribution described by at least one sub-probability parameter; entropy encoding the plurality of sub-latent representations using the plurality of at least one sub-probability parameters to produce a bitstream; transmitting the bitstream to a second computer system; entropy decoding the bitstream using the plurality of at least one sub-probability parameters to retrieve the plurality of sub-latent representations and combining the plurality of sub-latent representations to retrieve the latent representation; and decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image. At least one of the plurality of sub-latent representations may have a different number of elements to another of the plurality of sub-latent representations. The number of elements in each of the plurality of sub-latent representations may be selected such that the entropy encoding of each of the plurality of sub-latent representations has equivalent computational complexity. The method may further comprise the step of: after dividing the latent representation into a plurality of sub-latent representations, adding a tag to each sub-latent representation, the tag indicating the position of the sub-latent representation within the latent representation; and using the tag when combining the plurality of sub-latent representations to retrieve the latent representation. The bitstream may comprise a plurality of sub-bitstreams, each sub-bitstream corresponding to one of the plurality of sub-latent representations, and the method further comprises the step of: adding a tag to each sub-bitstream, the tag indicating the start of each sub-bitstream; wherein, after receipt of the bitstream at the second computer system, the bitstream is divided to retrieve each sub-bitstream using the tag indicating the start of each sub-bitstream. At least two of the plurality of sub-latent representations may be encoded in parallel. At least two of the plurality of sub-latent representations may be retrieved by the decoding of the bitstream in parallel. The step of entropy encoding the plurality of sub-latent representations using the plurality of at least one sub-probability parameters to produce a bitstream may comprise the steps of: entropy encoding each of the plurality of sub-latent representations to obtain a plurality of states and a plurality of sub-bitstreams; entropy encoding the plurality of states; and combining the plurality of sub-bitstreams and the entropy encoded plurality of states to obtain the bitstream. The may further comprise the steps of: encoding the latent representation using a third trained neural network to produce a hyper-latent representation; entropy encoding the hyper-latent representation and including the entropy encoded hyper-latent representation in the bitstream; at the second computer system, entropy decoding the entropy encoded hyper-latent representation to retrieve the hyper-latent representation; and decoding the hyper-latent representation using a fourth trained neural network, wherein the output of the fourth trained neural network is used to obtain the output image. The step of entropy encoding the hyper-latent representation may comprise dividing the hyper-latent representation into a plurality of sub-hyper-latent representations, wherein each sub-hyper-latent representation has a sub-probability distribution described by at least one sub- probability parameter; and entropy encoding the plurality of sub-hyper-latent representations using the plurality of at least one sub-probability parameters; and the step of entropy decoding the entropy encoded hyper-latent representation may comprise entropy decoding the the entropy encoded hyper-latent representation using the plurality of at least one sub-probability parameters to retrieve the plurality of sub-hyper-latent representations and combining the plurality of sub-hyper-latent representations to retrieve the hyper-latent representation. The step of entropy encoding the plurality of sub-latent representations using the plurality of at least one sub-probability parameters to produce a bitstream may comprise the steps of: entropy encoding each of the plurality of sub-latent representations to obtain a plurality of states and a plurality of sub-bitstreams; entropy decoding the plurality of states to obtain a state latent; entropy encoding the state latent to obtain a state bitstream; and combining the plurality of sub-bitstreams and the state bitstream to obtain the bitstream. A lossy or lossless compression process amy be performed on the state latent prior to the entropy encoding of the state latent to obtain the state bitstream. The step of entropy encoding the plurality of sub-hyper-latent representations using the plurality of at least one sub-probability parameters may comprise the steps of: entropy encoding each of the plurality of sub-hyper-latent representations to obtain a plurality of hyper-states and a plurality of sub-hyper-bitstreams; entropy decoding the plurality of hyper-states to obtain a hyper-state latent; entropy encoding the hyper-state latent to obtain a hyper-state bitstream; and combining the plurality of sub-bitstreams and the hyper-state bitstream to obtain the bitstream. A lossy or lossless compression process may be performed on the hyper-state latent prior to the entropy encoding of the hyper-state latent to obtain the bitstream. The step of entropy encoding the plurality of sub-latent representations using the plurality of at least one sub-probability parameters to produce a bitstream may comprise the steps of: entropy encoding each of the plurality of sub-latent representations to obtain a plurality of states and a plurality of sub-bitstreams; and entropy decoding the plurality of states to obtain a state latent; wherein the step of entropy encoding the plurality of sub-hyper-latent representations using the plurality of at least one sub-probability parameters may comprise the steps of: entropy encoding each of the plurality of sub-hyper-latent representations to obtain a plurality of hyper-states and a plurality of sub-hyper-bitstreams; and entropy decoding the plurality of hyper-states to obtain a hyper-state latent; wherein the method may further comprises the steps of: combining the state latent and the hyper-state latent to obtain a combined state latent; entropy encoding the combined state latent to obtain a combined state bitstream; and combining the plurality of sub-bitstreams and the combined state bitstream to obtain the bitstream. A lossy or lossless compression process may be performed on the combined state latent prior to the entropy encoding of the combined state latent to obtain the bitstream. The method may further comprise the steps of: receiving a further input image at the first computer system, wherein the input image and the further input image are frames of an input video; encoding the further input image using the first trained neural network to produce a further latent representation, wherein the further latent representation has a probability distribution described by at least one probability parameter; dividing the further latent representation into a plurality of further sub-latent representations, wherein each further sub-latent representation has a sub-probability distribution described by at least one sub-probability parameter; entropy encoding the plurality of further sub-latent representations using the plurality of at least one sub-probability parameters to produce a further bitstream; transmitting the further bitstream to a second computer system; entropy decoding the further bitstream using the plurality of at least one sub-probability parameters to retrieve the plurality of further sub-latent representations and combining the plurality of further sub-latent representations to retrieve the further latent representation; and decoding the further latent representation using the second trained neural network to produce a further output image, wherein the further output image is an approximation of the further input image. The step of entropy encoding the further plurality of sub-latent representations using the plurality of at least one sub-probability parameters to produce a further bitstream may comprise the steps of: entropy encoding each of the plurality of further sub-latent representations to obtain a plurality of further states and a plurality of further sub-bitstreams; entropy decoding the plurality of further states to obtain a further state latent; combining the state latent and the further latent to obtain a combined state latent; entropy encoding the combined state latent to obtain a combined state bitstream; and combining the plurality of sub-bitstreams, the plurality of further sub-bitstreams and the combined state bitstream to obtain the bitstream. According to the present invention there is provided a method for bitstream encoding, transmission and decoding, the method comprising the steps of: receiving an input at a first computer system, wherein the input has a probability distribution described by at least one probability parameter; dividing the input into a plurality of sub-inputs, wherein each sub-input has a sub-probability distribution described by at least one sub-probability parameter; entropy encoding the plurality of sub-inputs using the plurality of at least one sub-probability parameters to produce a bitstream; transmitting the bitstream to a second computer system; and entropy decoding the bitstream using the plurality of at least one sub-probability parameters to retrieve the plurality of sub-inputs and combining the plurality of sub-latent inputs to retrieve the input. According to the present invention there is provided a method for lossy image or video encoding and transmission, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first trained neural network to produce a latent representation, wherein the latent representation has a probability distribution described by at least one probability parameter; dividing the latent representation into a plurality of sub-latent representations, wherein each sub-latent representation has a sub-probability distribution described by at least one sub-probability parameter; entropy encoding the plurality of sub-latent representations using the plurality of at least one sub-probability parameters to produce a bitstream; transmitting the bitstream. According to the present invention there is provided a method for lossy image or video receipt and decoding, the method comprising the steps of: receiving a bitstream at a second computer system; entropy decoding the bitstream using the plurality of at least one sub-probability parameters to retrieve the plurality of sub-latent representations and combining the plurality of sub-latent representations to retrieve the latent representation; and decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image. According to an aspect of the present disclosure, there is provided a method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first trained neural network to produce a latent representation, wherein the latent representation has a probability distribution described by at least one probability parameter; generating a mask based on at least one of the latent representation or the at least one probability parameter; applying the mask to the latent representation to produce a masked latent representation; entropy encoding the masked latent representations using the at least one probability parameter to produce a bitstream; transmitting the bitstream to a second computer system; entropy decoding the bitstream using the at least one probability parameter and the mask to retrieve the latent representation; and decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image. Optionally, the method comprises: dividing the latent representation into a plurality of partitions, estimating at least one statistical value associated the plurality of partitions, and using the at least one estimated statistical value to produce the mask. Optionally, the statistical value comprises a mean. Optionally, producing the mask comprises applying a threshold to thge at least one statistical value to generate a mask value for at least one partition of the plurality of partitions Optionally, applying the mask comprises setting one or more elements of the latent representa- tion to zero. Optionally, the method comprises using the at least one probability parameter to estimate a value of at least one element of the latent representation, and using the estimated value to generate a mask value for the at least one element. Optionally, entropy encoding the masked latent representation comprises skipping masked elements of the masked latent representation. Optionally, entropy decoding the bitstream comprises skipping elements corresponding to masked elements of the masked latent representation. Optionally, skipping elements comprises initialising a buffer with zeros and skipping a predetermined number of zeros based on a size of a masked section of the masked latent representation. Optionally, skipping elements comprises inserting a predetermined number of zeros in a buffer based on a size of a masked section of the masked latent representation. Optionally, the method comprises retrieving the mask from the bitstream. Optionally, entropy decoding comprises using the at least one probability parameter to estimate a value of at least one element of the latent representation, and using the estimated value to generate a mask value for the at least one element. According to an aspect of the present disclosure, there is provided, a method for lossy image or video encoding and transmission, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first trained neural network to produce a latent representation, wherein the latent representation has a probability distribution described by at least one probability parameter; generating a mask based on at least one of the latent representation or the at least one probability parameter; applying the mask to the latent representation to produce a masked latent representation; entropy encoding the masked latent representations using the at least one probability parameter to produce a bitstream; and transmitting the bitstream to a second computer system. According to an aspect of the present disclosure there is provided, a method for lossy image or video receiving and decoding, the method comprising the steps of: receiving a bitstream a a second computer system, the bitstream produced by entropy encoding a masked latent representation associated with an input image using at least one entropy parameter; entropy decoding the bitstream using the at least one probability parameter and a mask to retrieve the latent representation; and decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image. According to the present invention there is provided a data processing system or apparatus configured to perform any of the methods above. According to the present invention there is provided a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out any of the methods above. According to the present invention there is provided a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out any of the methods above. Aspects of the invention will now be described by way of examples, with reference to the following figures in which: Aspects of the invention will now be described by way of examples, with reference to the following figures in which: Figure 1 illustrates an example of an image or video compression, transmission and decom- pression pipeline. Figure 2 illustrates a further example of an image or video compression, transmission and decompression pipeline including a hyper-network. Figure 3 illustrates an example of a video compression, transmission and decompression pipeline. Figure 4 illustrates an example of a video compression, transmission and decompression system. Figure 5 illustrates an example of a flow composition network. Figure 6 illustrates an example of a video compression pipeline with a flow composition network Figure 7 illustrates an example of a flow composition network applied to B-frames. Figure 8 illustrates a schematic example of a P-frame module in the standard flow-residual framework Figure 9 illustrates a schematic example representation of the P-frame module with the residual composition network. Figure 10 illustrates a schematic example of a video compression pipeline with a residual composition network. Figure 11 illustrates a schematic example of a video compression pipeline including a hyper-encoder and hyper-decoder with a flow composition network. Figure 12 illustrates a schematic example of a video compression pipeline including a hyper-hyper-encoder and hyper-hyper-decoder with a flow composition network. Figure 13 illustrates a schematic example of a modified video compression pipeline including a hyper-hyper-encoder and hyper-hyper-decoder with a flow composition network. Figure 14 illustrates a schematic example of a further modified video compression pipeline including a hyper-hyper-encoder and hyper-hyper-decoder with a flow composition network. Figure 15 illustrates a schematic example of losses in the P-framemodule of a video compression pipeline. Figure 16 illustrates a visualisation of encode order with and without B-frames enabled. Figure 17 illustrates a visualisation of relationships between different flows in a video compression process. Figure 18 illustrates a visualization of different linear global flows. Figure 19 illustrates a visualization of block-based global flows. Figure 20 illustrates a schematic overview of a finetuning architecture. Figure 21 illustrates a schematic overview of a metanetwork architecture. Figure 22 illustrates a schematic example of a serial RANS pipeline. Figure 23 illustrates a schematic example of a parallel RANS encoder and decoder pipeline. Figure 24 illustrates a schematic example of state decoding, compression and re-encoding. Figure 25 illustrates a masked rANS encoder and decoder implementation according to the present disclosure Figure 26 illustrates a masked rANS approach where the mask is estimated from the entropy parameters ^^ alone. DETAILED DESCRIPTION OF THE DRAWINGS Compression processes may be applied to any form of information to reduce the amount of data, or file size, required to store that information. Image and video information is an example of information that may be compressed. The file size required to store the information, particularly during a compression process when referring to the compressed file, may be referred to as the rate. In general, compression can be lossless or lossy. In both forms of compression, the file size is reduced. However, in lossless compression, no information is lost when the information is compressed and subsequently decompressed. This means that the original file storing the information is fully reconstructed during the decompression process. In contrast to this, in lossy compression information may be lost in the compression and decompression process and the reconstructed file may differ from the original file. Image and video files containing image and video data are common targets for compression. In a compression process involving an image, the input image may be represented as ^^. The data representing the image may be stored in a tensor of dimensions ^^ × ^^ × ^^, where ^^ represents the height of the image, ^^ represents the width of the image and ^^ represents the number of channels of the image. Each ^^ × ^^ data point of the image represents a pixel value of the image at the corresponding location. Each channel ^^ of the image represents a different component of the image for each pixel which are combined when the image file is displayed by a device. For example, an image file may have 3 channels with the channels representing the red, green and blue component of the image respectively. In this case, the image information is stored in the RGB colour space, which may also be referred to as a model or a format. Other examples of colour spaces or formats include the CMKY and the YCbCr colour models. However, the channels of an image file are not limited to storing colour information and other information may be represented in the channels. As a video may be considered a series of images in sequence, any compression process that may be applied to an image may also be applied to a video. Each image making up a video may be referred to as a frame of the video. The output image may differ from the input image and may be represented by ˆ. The difference between the input image and the output image may be referred to as distortion or a difference in image quality. The distortion can be measured using any distortion function which receives the input image and the output image and provides an output which represents the difference between input image and the output image in a numerical way. An example of such a method is using the mean square error (MSE) between the pixels of the input image and the output image, but there are many other ways of measuring distortion, as will be known to the person skilled in the art. The distortion function may comprise a trained neural network. Typically, the rate and distortion of a lossy compression process are related. An increase in the rate may result in a decrease in the distortion, and a decrease in the rate may result in an increase in the distortion. Changes to the distortion may affect the rate in a corresponding manner. A relation between these quantities for a given compression technique may be defined by a rate-distortion equation. AI based compression processes may involve the use of neural networks. A neural network is an operation that can be performed on an input to produce an output. A neural network may be made up of a plurality of layers. The first layer of the network receives the input. One or more operations may be performed on the input by the layer to produce an output of the first layer. The output of the first layer is then passed to the next layer of the network which may perform one or more operations in a similar way. The output of the final layer is the output of the neural network. Each layer of the neural network may be divided into nodes. Each node may receive at least part of the input from the previous layer and provide an output to one or more nodes in a subsequent layer. Each node of a layer may perform the one or more operations of the layer on at least part of the input to the layer. For example, a node may receive an input from one or more nodes of the previous layer. The one or more operations may include a convolution, a weight, a bias and an activation function. Convolution operations are used in convolutional neural networks. When a convolution operation is present, the convolution may be performed across the entire input to a layer. Alternatively, the convolution may be performed on at least part of the input to the layer. Each of the one or more operations is defined by one or more parameters that are associated with each operation. For example, the weight operation may be defined by a weight matrix defining the weight to be applied to each input from each node in the previous layer to each node in the present layer. In this example, each of the values in the weight matrix is a parameter of the neural network. The convolution may be defined by a convolution matrix, also known as a kernel. In this example, one or more of the values in the convolution matrix may be a parameter of the neural network. The activation function may also be defined by values which may be parameters of the neural network. The parameters of the network may be varied during training of the network. Other features of the neural network may be predetermined and therefore not varied during training of the network. For example, the number of layers of the network, the number of nodes of the network, the one or more operations performed in each layer and the connections between the layers may be predetermined and therefore fixed before the training process takes place. These features that are predetermined may be referred to as the hyperparameters of the network. These features are sometimes referred to as the architecture of the network. To train the neural network, a training set of inputs may be used for which the expected output, sometimes referred to as the ground truth, is known. The initial parameters of the neural network are randomized and the first training input is provided to the network. The output of the network is compared to the expected output, and based on a difference between the output and the expected output the parameters of the network are varied such that the difference between the output of the network and the expected output is reduced. This process is then repeated for a plurality of training inputs to train the network. The difference between the output of the network and the expected output may be defined by a loss function. The result of the loss function may be calculated using the difference between the output of the network and the expected output to determine the gradient of the loss function. Back-propagation of the gradient descent of the loss function may be used to update the parameters of the neural network using the gradients ^^ ^^ / ^^ ^^ of the loss function. A plurality of neural networks in a system may be trained simultaneously through back-propagation of the gradient of the loss function to each network. In the context of image or video compression, this type of system, where simultaneous training with back-propagation through each element or the whole network architecture may be referred to as end-to-end, learned image or video compression. Unlike in traditional compression algorithms that use primarily handcrafted, manually constructed steps, an end-to-end learned system learns itself during training what combination of parameters best achieves the goal of minimising the loss function. This approach is advantageous compared to systems that are not end-to-end learned because an end-to-end system has a greater flexibility to learn weights and parameters that might be counter-intuitive to someone handcrafting features. It will be appreciated that the term "training" or "learning" as used herein means the process of optimizing an artificial intelligence or machine learning model, based on a given set of data. This involves iteratively adjusting the parameters of the model to minimize the discrepancy between the model’s predictions and the actual data, represented by the above-described rate-distortion loss function. The training process may comprise multiple epochs. An epoch refers to one complete pass of the entire training dataset through the machine learning algorithm. During an epoch, the model’s parameters are updated in an effort to minimize the loss function. It is envisaged that multiple epochs may be used to train a model, with the exact number depending on various factors including the complexity of the model and the diversity of the training data. Within each epoch, the training data may be divided into smaller subsets known as batches. The size of a batch, referred to as the batch size, may influence the training process. A smaller batch size can lead to more frequent updates to the model’s parameters, potentially leading to faster convergence to the optimal solution, but at the cost of increased computational resources. Conversely, a larger batch size involves fewer updates, which can be more computationally efficient but might converge slower or even fail to converge to the optimal solution. The learnable parameters are updated by a specified amount each time, determined by the learning rate. The learning rate is a hyperparameter that decides how much the parameters are adjusted during the training process. A smaller learning rate implies smaller steps in the parameter space and a potentially more accurate solution, but it may require more epochs to reach that solution. On the other hand, a larger learning rate can expedite the training process but may risk overshooting the optimal solution or causing the training process to diverge. The training described herein may involve use of a validation set, which is a portion of the data not used in the initial training, which is used to evaluate the model’s performance and to prevent overfitting. Overfitting occurs when a model learns the training data too well, to the point that it fails to generalize to unseen data. Regularization techniques, such as dropout or L1 / L2 regularization, can also be used to mitigate overfitting. It will be appreciated that training a machine learning model is an iterative process that may comprise selection and tuning of various parameters and hyperparameters. As will be appreciated, the specific details, such as hyper parameters and so on, of the training process may vary and it is envisaged that producing a trained model in this way may achieved in a number of different ways with different epochs, batch sizes, learning rates, regularisations, and so on, the details of which are not essential to enabling the advantages and effects of the present disclosure, except where stated otherwise. The point at which an “untrained” neural network is considered be “trained” is envisaged to be case specific and depend on, for example, on a number of epochs, a plateauing of any further learning, or some other metric and is not considered to be essential in achieving the advantages described herein. More details of an end-to-end, learned compression process will now be described. It will be appreciated that in some cases, end-to-end, learned compression processes may be combined with one or more components that are handcrafted or trained separately. In the case of AI based image or video compression, the loss function may be defined by the rate distortion equation. The rate distortion equation may be represented by ^^ ^^ ^^ ^^ = ^^ + ^^ ∗ ^^, where ^^ is the distortion function, ^^ is a weighting factor, and ^^ is the rate loss. ^^ may be referred to as a lagrange multiplier. The langrange multiplier provides as weight for a particular term of the loss function in relation to each other term and can be used to control which terms of the loss function are favoured when training the network. In the case of AI based image or video compression, a training set of input images may be used. An example training set of input images is the KODAK image set (for example at www.cs.albany.edu / xypan / research / snr / Kodak.html). An example training set of input images is the IMAX image set. An example training set of input images is the Imagenet dataset (for example at www.image-net.org / download). An example training set of input images is the CLIC Training Dataset P (“professional”) and M (“mobile”) (for example at http: / / challenge.compression.cc / tasks / ). An example of an AI based compression, transmission and decompression process 100 is shown in Figure 1. As a first step in the AI based compression process, an input image 5 is provided. The input image 5 is provided to a trained neural network 110 characterized by a function ^^^^acting as an encoder. The encoder neural network 110 produces an output based on the input image. This output is referred to as a latent representation of the input image 5. In a second step, the latent representation is quantised in a quantisation process 140 characterised by the operation ^^, resulting in a quantized latent. The quantisation process transforms the continuous latent representation into a discrete quantized latent. An example of a quantization process is a rounding function. In a third step, the quantized latent is entropy encoded in an entropy encoding process 150 to produce a bitstream 130. The entropy encoding process may be for example, range or arithmetic encoding. In a fourth step, the bitstream 130 may be transmitted across a communication network. In a fifth step, the bitstream is entropy decoded in an entropy decoding process 160. The quantized latent is provided to another trained neural network 120 characterized by a function ^^^^acting as a decoder, which decodes the quantized latent. The trained neural network 120 produces an output based on the quantized latent. The output may be the output image of the AI based compression process 100. The encoder-decoder system may be referred to as an autoencoder. Entropy encoding processes such as range or arithmetic encoding are typically able to losslessly compress given input data up to close to the fundamental entropy limit of that data, as determined by the total entropy of the distribution of that data. Accordingly, one way in which end-to-end, learned compression can minimise the rate loss term of the rate-distortion loss function and thereby increase compression effectiveness is to learn autoencoder parameter values that produce low entropy latent representation distributions. Producing latent representations distributed with as low an entropy as possible allows entropy encoding to compress the latent distributions as close to or to the fundamental entropy limit for that distribution. The lower the entropy of the distribution, the more entropy encoding can losslessly compress it and the lower the amount of data in the corresponding bitstream. In some cases where the latent representation is distributed according to a gaussian or Laplacian distribution, this learning may comprise learning optimal location and scale parameters of the gaussian or Laplacian distributions, in other cases, it allows the learning of more flexible latent representation distributions which can further help to achieve the minimising of the rate-distortion loss function in ways that are not intuitive or possible to do with handcrafted features. Examples of these and other advantages are described in WO2021 / 220008A1, which is incorporated in its entirety by reference. Something which is closely linked to the entropy encoding of the latent distribution and which accordingly also has an effect on the effectiveness of compression of end-to-end learned approaches is the quantisation step. During inference, a rounding function may be used to quantise a latent representation distribution into bins of given sizes, a rounding function is not differentiable everywhere. Rather, a rounding function is effectively one or more step functions whose gradient is either zero (at the top of the steps) or infinity (at the boundary between steps). Back propagating a gradient of a loss function through a rounding function is challenging. Instead, during training, quantisation by rounding function is replaced by one or more other approaches. For example, the functions of a noise quantisation model are differentiable everywhere and accordingly do allow backpropagation of the gradient of the loss function through the quantisation parts of the end-to-end, learned system. Alternatively, a straight-through estimator (STE) quantisation model or one other quantisation models may be used. It is also envisaged that different quantisation models may be used for during evaluation of different term of the loss function. For example, noise quantisation may used to evaluate the rate or entropy loss term of the rate-distortion loss function while STE quantisation may be used to evaluate the distortion term. In a similar manner to how learning parameters top produce certain distributions of the latent representation facilitates achieving better rate loss term minimisation, end-to-end learning of the quantisation process achieves a similar effect. That is, learnable quantisation parameters provide the architecture with a further degree of freedom to achieve the goal of minimising the loss function. For example, parameters corresponding to quantisation bin sizes may be learned which is likely to result in an improved rate-distortion loss outcome compared to approaches using hand-crafted quantisation bin sizes. Further, as the rate-distortion loss function constantly has to balance a rate loss term against a distortion loss term, it has been found that the more degrees of freedom the system has during training, the better the architecture is at achieving optimal rate and distortion trade off. The system described above may be distributed across multiple locations and / or devices. For example, the encoder 110 may be located on a device such as a laptop computer, desktop computer, smart phone or server. The decoder 120 may be located on a separate device which may be referred to as a recipient device. The system used to encode, transmit and decode the input image 5 to obtain the output image 6 may be referred to as a compression pipeline. The AI based compression process may further comprise a hyper-network 105 for the transmission of meta-information that improves the compression process. The hyper-network 105 comprises a trained neural network 115 acting as a hyper-encoder ^^^^ℎand a trained neural network 125 acting as a hyper-decoder ^^^ℎ^ . An example of such a is shown in Figure 2. Components of the system not further discussed may be assumed to the same as discussed above. The neural network 115 acting as a hyper-decoder receives the latent that is the output of the encoder 110. The hyper-encoder 115 produces an output based on the latent representation that may be referred to as a hyper-latent representation. The hyper-latent is then quantized in a quantization process 145 characterised by ^^ℎto produce a quantized hyper-latent. The quantization process 145 characterised by ^^ℎmay be the same as the quantisation process 140 characterised by ^^ discussed above. In a similar manner as discussed above for the quantized latent, the quantized hyper-latent is then entropy encoded in an entropy encoding process 155 to produce a bitstream 135. The bitstream 135 may be entropy decoded in an entropy decoding process 165 to retrieve the quantized hyper-latent. The quantized hyper-latent is then used as an input to trained neural network 125 acting as a hyper-decoder. However, in contrast to the compression pipeline 100, the output of the hyper-decoder may not be an approximation of the input to the hyper-decoder 115. Instead, the output of the hyper-decoder is used to provide parameters for use in the entropy encoding process 150 and entropy decoding process 160 in the main compression process 100. For example, the output of the hyper-decoder 125 can include one or more of the mean, standard deviation, variance or any other parameter used to describe a probability model for the entropy encoding process 150 and entropy decoding process 160 of the latent representation. In the example shown in Figure 2, only a single entropy decoding process 165 and hyper-decoder 125 is shown for simplicity. However, in practice, as the decompression process usually takes place on a separate device, duplicates of these processes will be present on the device used for encoding to provide the parameters to be used in the entropy encoding process 150. Further transformations may be applied to at least one of the latent and the hyper-latent at any stage in the AI based compression process 100. For example, at least one of the latent and the hyper latent may be converted to a residual value before the entropy encoding process 150,155 is performed. The residual value may be determined by subtracting the mean value of the distribution of latents or hyper-latents from each latent or hyper latent. The residual values may also be normalised. To perform training of the AI based compression process described above, a training set of input images may be used as described above. During the training process, the parameters of both the encoder 110 and the decoder 120 may be simultaneously updated in each training step. If a hyper-network 105 is also present, the parameters of both the hyper-encoder 115 and the hyper-decoder 125 may additionally be simultaneously updated in each training step. The training process may further include a generative adversarial network (GAN). When applied to an AI based compression process, in addition to the compression pipeline described above, an additional neutral network acting as a discriminator is included in the system. The discriminator receives an input and outputs a score based on the input providing an indication of whether the discriminator considers the input to be ground truth or fake. For example, the indicator may be a score, with a high score associated with a ground truth input and a low score associated with a fake input. For training of a discriminator, a loss function is used that maximizes the difference in the output indication between an input ground truth and input fake. When a GAN is incorporated into the training of the compression process, the output image 6 may be provided to the discriminator. The output of the discriminator may then be used in the loss function of the compression process as a measure of the distortion of the compression process. Alternatively, the discriminator may receive both the input image 5 and the output image 6 and the difference in output indication may then be used in the loss function of the compression process as a measure of the distortion of the compression process. Training of the neural network acting as a discriminator and the other neutral networks in the compression process may be performed simultaneously. During use of the trained compression pipeline for the compression and transmission of images or video, the discriminator neural network is removed from the system and the output of the compression pipeline is the output image 6. Incorporation of a GAN into the training process may cause the decoder 120 to perform hallucination. Hallucination is the process of adding information in the output image 6 that was not present in the input image 5. In an example, hallucination may add fine detail to the output image 6 that was not present in the input image 5 or received by the decoder 120. The hallucination performed may be based on information in the quantized latent received by decoder 120. Details of a video compression process will now be described. As discussed above, a video is made up of a series of images arranged in sequential order. AI based compression process 100 described above may be applied multiple times to perform compression, transmission and decompression of a video. For example, each frame of the video may be compressed, transmitted and decompressed individually. The received frames may then be grouped to obtain the original video. The frames in a video may be labelled based on the information from other frames that is used to decode the frame in a video compression, transmission and decompression process. As described above, frames which are decoded using no information from other frames may be referred to as I-frames. Frames which are decoded using information from past frames may be referred to as P-frames. Frames which are decoded using information from past frames and future frames may be referred to as B-frames. Frames may not be encoded and / or decoded in the order that they appear in the video. For example, a frame at a later time step in the video may be decoded before a frame at an earlier time. The images represented by each frame of a video may be related. For example, a number of frames in a video may show the same scene. In this case, a number of different parts of the scene may be shown in more than one of the frames. For example, objects or people in a scene may be shown in more than one of the frames. The background of the scene may also be shown in more than one of the frames. If an object or the perspective is in motion in the video, the position of the object or background in one frame may change relative to the position of the object or background in another frame. The transformation of a part of the image from a first position in a first frame to a second position in a second frame may be referred to as flow, warping or motion compensation. The flow may be represented by a vector. One or more flows that represent the transformation of at least part of one frame to another frame may be referred to as a flow map. An example AI based video compression, transmission, and decompression process 200 is shown in Figure 3. The process 200 shown in Figure 3 is divided into an I-frame part 201 for decompressing I-frames, and a P-frame part 202 for decompressing P-frames. It will be understood that these divisions into different parts are arbitrary and the process 200 may be also be considered as a single, end-to-end pipeline. As described above, I-frames do not rely on information from other frames so the I-frame part 201 corresponds to the compression, transmission, and decompression process illustrated in Figures 1 or 2. The specific details will not be repeated here but, in summary, an input image ^^0is passed into an encoder neural network 203 producing a latent representation which is quantised and entropy encoded into a bitstream 204. The subscript 0 in ^^0indicates the input image corresponds to a frame of a video stream at position t = 0. This may be the first frame of an entire video stream or the first frame of a chunk of a video stream made up of, for example, an I-frame and a plurality of subsequent P-frames and / or B-frames. The bitstream 204 is then entropy decoded and passed into a decoder neural network 205 to reproduce a reconstructed image ˆ0which in this case is an I-frame. The decoding step may be performed both locally at the same location as where the input image compression occurs as well as at the location where the decompression occurs. This allows the reconstructed image ˆ0to be available for later use by components of both the encoding and decoding sides of the pipeline. In contrast to I-frames, P-frames (and B-frames) do rely on information from other frames. Accordingly, the P-frame part 202 at the encoding side of the pipeline takes as input not only the input image ^^^^that is to be compressed (corresponding to a frame of a video stream at position t), but also one or more previously reconstructed images ˆ^^−1from an earlier frame t-1. As described above, the previously reconstructed ˆ^^−1is available at both the encode and decode side of the pipeline and can accordingly be used for various purposes at both the encode and decode sides. At the encode side, previously reconstructed images may be used for generating a flow maps containing information indicative of inter-frame movement of pixels between frames. In the example of Figure 3, both the image being compressed ^^^^and the previously reconstructed image from an earlier frame ˆ^^−1are passed into a flow module part 206 of the pipeline. The flow module part 206 comprises an autoencoder such as that of the autoencoder systems of Figures 1 and 2 but where the encoder neural network 207 has been trained to produce a latent representation of a flow map from inputs ˆ^^−1and ^^^^, which is indicative of inter-frame movement of pixels or pixel groups between ˆ^^−1and ^^^^. The latent representation of the flow map is quantised and entropy encoded to compress it and then transmitted as a bitstream 208. On the decode side, the bitstream is entropy decoded and passed to a decoder neural network 209 to produce a reconstructed flow map ^^ . The reconstructed flow map ^^ is applied to the previously reconstructed image ˆ^^−1to generate a warped image ˆ^^−1, ^^. It is envisaged that any suitable warping technique may be used, for example bi-linear or tri-linear warping, as is described in Agustsson, E., Minnen, D., Johnston, N., Balle, J., Hwang, S. J., and Toderici, G. (2020), Scale-space flow for end-to-end optimized video compression. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 8503-8512), which is hereby incorporated by reference. It is further envisaged that a scale-space flow approach as described in the above paper may also optionally be used. The warped image ˆ^^−1, ^^is a prediction of how the previously reconstructed image ˆ^^−1might have changed between frame positions t-1 and t, based on the output flow map produced by the flow module part 206 autoencoder system from the inputs of ^^^^and ˆ^^−1. As with the I-frame, the reconstructed flow map ^^ and corresponding warped image ˆ^^−1, ^^may be produced both on the encode side and the decode side of the pipeline so they are available for use by other components of the pipeline on both the encode and decode sides. In the example of Figure 3, both the image being compressed ^^^^and the ˆ^^−1, ^^are passed into a residual module part 210 of the pipeline. The residual module part 210 comprises an autoencoder system such as that of the autoencoder systems of Figures 1 and 2 but where the encoder neural network 211 has been trained to produce a latent representation of a residual map indicative of differences between the input mage ^^^^and the warped image ˆ^^−1, ^^. The latent representation of the residual map is then quantised and entropy encoded into a bitstream 212 and transmitted. The bitstream 212 is then entropy decoded and passed into a decoder neural network 213 which reconstructs a residual map ^^ from the decoded latent representation. Alternatively, a residual map may first be pre-calculated between ^^^^and the ˆ^^−1, ^^and the pre-calculated residual map may be passed into an autoencoder for compression only. This hand-crafted residual map approach is computationally simpler, but reduces the degrees of freedom with which the architecture may learn weights and parameters to achieve its goal during training of minimising the rate-distortion loss function. Finally, on the decode side, the residual map ^^ is applied (e.g. combined by addition, subtraction or a different operation) to the warped image to produce a reconstructed image ˆ^^which is a reconstruction of image ^^^^and accordingly corresponds to a P-frame at position t in a sequence of frames of a video stream. It will be appreciated that the reconstructed image ˆ^^can then be used to process the next frame. That is, it can be used to compress, transmit and decompress ^^^^+1, and so on until an entire video stream or chunk of a video stream has been processed. Thus, for a block of video frames comprising an I-frame and ^^ subsequent P-frames, the bitstream may contain (i) a quantised, entropy encoded latent representation of the I-frame image, and (ii) a quantised, entropy encoded latent representation of a flow map and residual map of each P-frame image. For completeness, whilst not illustrated in Figure 3, any of the autoencoder systems of Figure 3 may comprise hyper and hyper-hyper networks such as those described in connection with Figure 2. Accordingly, the bitstream may also contain hyper and hyper-hyper parameters, their latent quantised, entropy encoded latent representations and so on, of those networks as applicable. Finally, the above approach may generally also be extended to B-frames, for example as is described in Pourreza, R., and Cohen, T. (2021). Extending neural p-frame codecs for b-frame coding. In Proceedings of the IEEE / CVF International Conference on Computer Vision (pp. 6680-6689) which is hereby incorporated by reference. The above-described flow and residual based approach is highly effective at reducing the amount of data that needs to be transmitted because, as long as at least one reconstructed frame (e.g. I-frame ˆ^^−1) is available, the encode side only needs to compress and transmit a flow map and a residual map (and any hyper or hyper-hyper parameter information, as applicable) to reconstruct a subsequent frame. Figure 4 shows an example of an AI image or video compression process such as that described above in connection with Figures 1-3 implemented in a video streaming system 400. The system 400 comprises a first device 401 and a second device 402. The first and second devices 401, 402 may be user devices such as smartphones, tablets, AR / VR headsets or other portable devices. In contrast to known systems which primarily perform inference on GPUs such as Nvidia A100, Geforce 3090, Gefore 4090 GPU cards, the system 400 of Figure 4 performs inference on a CPU of the first and second devices respectively. That is, compute for performing both encoding and decoding are performed by the respective CPUs of the first and second devices 401, 402. This places very different power usage, memory and runtime constraints on the implementation of the above methods than when implementing AI-based compression methods on GPUs. In one example, the CPU of first and second devices 401, 402 may comprise a Qualcomm Snapdragon CPU. The first device 401 comprises a media capture device 403, such as a camera, arranged to capture a plurality of images, referred to hereafter as a video stream 404, of a scene 404. The video stream 404 is passed to a pre-processing module 406 which splits the video stream into blocks of frames, various frames of which will be designated as I-frames, P-frames, and / or B-frames. The blocks of frames are then compressed by an AI-compression module 407 comprising the encode side of the AI-based video compression pipeline of Figure 3. The output of the AI-compression module is accordingly a bitstream 408a which is transmitted from the first device 401, for example via a communications channel, for example over one or more of a WiFi, 3G, 4G or 5G channel, which may comprise internet or cloud-based 409 communications. The second device 402 receives the communicated bitstream 408b which is passed to an AI-decompression module 410 comprising the decode side of the AI-based video compression pipeline of Figure 3. The output of the AI-decompression module 402 is the reconstructed I-frames, P-frames and / or B-frames which are passed to a post-processing module 411 where they can prepared, for example passed into a buffer, in preparation for streaming 412 to and rendering on a display device 413 of the second device 402. It is envisaged that the system 400 of Figure 4 may be used for live video streaming at 30fps of a 1080p video stream, which means a cumulative latency of both the encode and decode side is below substantially 50ms, for example substantially 30ms or less. Achieving this level of runtime performance with only CPU compute on user devices presents challenges which are not addressed by known methods and systems or in the wider AI-compression literature. For example, execution of different parts of the compression pipeline during inference may be optimized by adjusting the order in which operations are performed using one or more known CPU scheduling methods. Efficient scheduling can allow for operations to be performed in parallel, thereby reducing the total execution time. It is also envisaged that efficient management of memory resources may be implemented, including optimising caching methods such as storing frequently-accessed data in faster memory locations, and memory reuse, which minimizes memory allocation and deallocation operations. A number of concepts related to the AI compression processes and / or their implementation in a hardware system discussed above will now be described. Although each concept is described separately, one or more of the concepts described below may be applied in an AI based compression process as described above. Flow composition network A method for more efficient encoding of the optical flow map in the flow-residual (aka motion compensation) video compression algorithm that utilises previously compressed frames and flow maps between them is presented below. We aim to encode a video group of frames (GOP) ^^0, ^^1, ... , ^^^^in a low latency regime using only I- and P- frames. For simplicity, we may assume that ^^0is the I-frame and all the subsequent frames ^^1, ... , ^^^^are P-frames. The full video may consist of multiple such GOPs. The I-frame encoding and decoding may be performed by an I-frame AI compression model, for example the pipeline described above, which encodes individual frames. The P-frame module consists of two parts: a flow composition network (FCN) part that calculates and compresses an optical flow map and a residual composition (or compression) (RCN) part that aims to encode the information not contained in the warped version of the previous frame. A previous frame may be a frame already compressed, transmitted and decompressed by the compression pipeline. Frames output by the compression pipeline that are approximations of frames input to the pipeline may be referred to as decoded frames corresponding to those frames. A schematic diagram of an example of a flow composition network is shown in figure 5. The FCN may comprise a neural network acing as an encoder and a neural network acting as a decoder. To encode the current P-frame ^^^^, we may have access to one or more previous frames ^^^^−1, ^^^^−2and the corresponding decoded versions ˆ^^−1, ˆ^^−2. We may also have access to a decoded optical flow map between these frames^ˆ^−1at the time of encoding and decoding. All this information may be available for all frames ^^^^where ^^ >= 2. If at least some of this information is unavailable, we may define the flow map^ˆ^−1:= 0. This approach assumes that there is no motion between the previous frames by defining and assuming the past two frames are the same: ^^^^−2:= ^^^^−1. Alternatively, we may define a separate flow compression network that does not require ^^^^−2and^ˆ^−1for the P-frames that do not have access to them, for example the first P-frame in the GOP. Additional inputs to the FCN encoder may include at least one of: ^^^^- the current P-frame that is being encoded, ^^^^- flow map between the previous frame ^^^^−1and the current frame ^^^^, usually obtained using a pre-trained optical flow estimation neural network, and ^^( ˆ^^−1, ^^^^) - a previously decoded frame ˆ^^−1warped with the flow map ^^^^. The neural network acting as an encoder produces the first latent representation ^^. The first latent representation may be quantised and transmitted in the bitstream in addition to the hyperlatents as set out in the AI based compression process described above. At decoding, in addition to the quantised latent ^ˆ^, the neural network acting as a decoder may take in as input at least one of the the past decoded frames ˆ^^−1, ˆ^^−2and the decoded optical flow map between these frames^ˆ^−1: ^^^^( ˆ^^−1, ˆ^^−2,^ˆ^−1). The transformation ^^^^: ^^ → ^^ is defined as a function that transforms an image to another image, ^^ = ^^^^ ^^× ^^ ^^× ^^ ^^and ^^ = ^^^^ ^^ × ^^ ^^ × ^^ ^^here denote spaces of images, of different resolutions. A purpose of the transformation ^^^^is to transform the input from the original image / flow space to the latent space to make it compatible with other inputs to the decoder. Examples of such transformations include downsampling, space-to-depth operation (PixelShuffle), Convolutional NN, etc. The transformation ^^^^: ^^ → ^^ may be applied to at least one of the past decoded frames ˆ^^−1, ˆ^^−2and the decoded optical flow map between these frames^ˆ^−1: ^^^^( ˆ^^−1, ˆ^^−2,^ˆ^−1) before these are used as an input to the decoder. ^^^^may be defined as a collection of transformations, one for each of the inputs, to account for different resolutions of each of the inputs. The output of the neural network acting as a decoder may be the decoded output flow map^ˆ^that compensates from ˆ^^−1to ˆ^^using the warping operation ^^ ( ˆ^^−1,^ˆ^). The warped version of the previously decoded frame ˆ^^^−^1 = ^^ ( ˆ^^−1,^ˆ^) may then used in the RCN part of the P-frame module. A neural network acting as a decoder may also be present on the encoding side to obtain the decoded output flow map^ˆ^on the encoding side, which may also be referred to as a predicted flow map. A schematic diagram of the complete video compression pipeline with the flow composition network is shown in figure 6. Further details of the RCN are provided below. The method described above may be expanded to any number of flowmaps (with corresponding frames) from any part of the video that has been decoded prior. The above method may also be applied to B-frames. Unlike P-frames, the B-frames can utilise information from both previous (P) and future (F) frames. The general scheme of the FCN generalisation to the B-frame module is shown in figure 7, where ^^^^ ^^is the optical flow map calculated between past reference frames ^^^^1and ^^^^2, ^^^^is the optical flow map calculated between the current frame ^^^^and the past reference frame ^^^^1, ^^^^ ^^is the optical flow map calculated between past reference frames ^^^^1and ^^^^2, and ^^^^is the optical flow map calculated between the current frame ^^^^and the past reference frame ^^^^1. Reference frames from a future point in the video may be referred to as future frames. When any of the encoders or decoders of the FCN or RCN receive a plurality of inputs, the separate inputs may be concatenated into a plurality of channels before being input into the network. RCN and Hyper-network flow We propose a method for more efficient encoding of the information that is not present in the warped version of the previous reconstructed frame by passing directly into the Encoder and Decoder of the RCN. We propose maximising the utilisation of the previously decoded frames by introducing a residual composition network. Unlike the standard flow-residual compression framework (Figure 8 shows a schematic example representation of the P-frame module in the standard flow-residual framework), the composition network does not require residuals to be calculated explicitly. The input to the RCN encoder may be a ground truth frame ^^^^, and ^^( ˆ^^−1,^ˆ^) - a previously decoded frame ˆ^^−1warped using the predicted flow map^ˆ^. To encode the current P-frame ^^^^, assume we have access to the previously decoded frame ˆ^^−1together with the decoded optical flow map^ˆ^−1between the ^^^^−1and ^^^^at the time of encoding and decoding. Inputs to the RCN encoder may be at least: ^^^^- the current P-frame that is encoded, ^^( ˆ^^−1) - previously decoded frame warped with the decoded flow map^ˆ^. The encoder produces the second latent representation ^^ that is quantised and passed in the bitstream as well as the hyperlatents following the general structure of the NN compression model. At the decoding, in addition to the quantised latent ^ˆ^, the FCN decoder may take ^^( ˆ^^−1,^ˆ^) as an input. The output of the decoder is the decoded frame ˆ^^. A transformation ^^ may be applied to ^^( ˆ^^−1,^ˆ^) prior to decoding by the FCN decoder: ^^ ( ^^( ˆ^^−1,^ˆ^)). The transformation ^^ : ^^ → ^^ may be defined as a function that transforms an image to another image, ^^ = ^^^^ ^^× ^^ ^^× ^^ ^^and ^^ = ^^^^ ^^ × ^^ ^^× ^^ ^^here denote spaces of images, of different resolutions. A purpose of the transformation is to transform the input from the original image / flow space to the latent space to make it compatible with other inputs to the decoder. Such transformations include downsampling, space-to-depth operation (PixelShuffle), a Haar transform, a Convolutional NN, etc. Figure 9 shows a schematic example representation of the P-frame module with the residual composition network, where the transformation ^^ may comprise at least two downsampling operations and a convolution operation, and Figure 10 shows a schematic example of the complete video compression pipeline with the residual composition network. Hyper modules as discussed above may additionally be included in these systems. As an extension of the above idea, we may directly use information already available at the time of the decoding in the entropy parameter estimation by passing ^^( ˆ^^−1,^ˆ^) directly to the hyper-encoder and hyper-decoder of the RCN, where the general concept of a hyper- network has already been discussed above. In order to transform ^^( ˆ^^−1,^ˆ^) to be used as a hyper-encoder input, we apply the appropriate transformations on the output of the transformation ^^ . The additional inputs to the hyper-encoder of the RCN is ^^ ( ^^ ( ^^( ˆ^^−1,^ˆ^))). Analogously, the additional input to the hyperdecoder is ^^( ^^ ( ^^ ( ^^( ˆ^^−1,^ˆ^)))). Figure 11 shows a schematic example of such an arrangement. The transformation ^^ may comprise at least two downsampling operations and a convolution operation. The transformation ^^ may comprise at least a downsampling operation and a convolution operation. The transformation ^^ may comprise at least a downsampling operation and a convolution operation. We can extend the method to the composition networks with a hyper-hyper module. The additional inputs to the hyper-hyper-encoder of the RCN isΦ( ^^ ( ^^ ( ^^( ˆ^^−1,^ˆ^)))). Analogously, the additional input to the hyperdecoder is ^^(Φ( ^^ ( ^^ ( ^^( ˆ^^−1,^ˆ^))))). Figure 12 shows a schematic example of such an arrangement. The transformation ^^ may comprise at least two downsampling operations and a convolution operation. The transformation ^^ may comprise at least a downsampling operation. The transformation ^^ may comprise at least a downsampling operation and a convolution operation. The transformation Φ may comprise at least one downsampling operation and a convolution operation. The transformation Ψ may comprise at least one downsampling operation and a convolution operation. This approach can be generalised to any number of hyper-modules and any number of previously decoded frames warped with corresponding flow maps. The methods above may be modified to reduce computing at decoding by only performing the transformations required for decoding time. A schematic first example is shown in figure 13, where the hyper-encoder input may be: ^^ ( ^^ ( ^^( ˆ^^−1,^ˆ^))), the hyper-decoder input may be: ^^( ^^ ( ^^( ˆ^^−1,^ˆ^))), the hyper-hyper-encoder input may be: Φ( ^^( ^^ ( ^^( ˆ^^−1,^ˆ^)))) and the hyper-hyper-decoder input may be: ^^( ^^( ^^ ( ^^( ˆ^^−1,^ˆ^)))). The transformation ^^ may comprise at least two downsampling operations and a convolution operation. The transformation ^^ may comprise at least a downsampling operation. The transformation ^^ may comprise at least a downsampling operation and a convolution operation. The transformation Φ may comprise at least one downsampling operation and a convolution operation. The transformation Ψ may comprise at least two downsampling operations and a convolution operation. A schematic second example of this approach is shown in Figure 14. The transformation ^^ may comprise at least two downsampling operations and a convolution operation. The transformation ^^ may comprise at least a downsampling operation. The transformation ^^ may comprise at least a downsampling operation and a convolution operation. The transformation Φ may comprise at least two downsampling operations and a convolution operation. The transformation Ψ may comprise at least two downsampling operations and a convolution operation. MSE Warped We propose training the compression pipelines discussed above with a modified loss that includes a flow interpolation loss - a distortion term between the previously decoded image warped using the decoded flow and the current frame. Minimising this loss may ensure that the warping of the previous image is as close to the ground truth as possible. This loss may allow the FCN in the P- and B-frame module to modify the flow in a way that benefits the overall performance of the compression model rather than simply compressing it. Flow interpolation loss may be added to the standard rate-distortion loss used for training compression models as discussed above. Figure 15 shows a schematic representation of an example of possible losses in the P-frame module of the video compression pipeline. ^^^^is image distortion loss - some metric measuring distortion between the ground-truth frame ^^^^and the decoded frame ˆ^^, ^^^^is flow-interpolation loss - some metric measuring distortion between the ground-truth frame ^^^^and the decoded version of the previous frame after a warping operation ^^ and ^^^^is flow distortion loss - some metric measuring distortion between the ground truth flow map ^^^^and the decoded flow map ^ˆ^. The training loss then may be: ^^ = ^^^^× ^^ + ^^^^× ^^^^( ^^^^, ˆ^^) + ^^^^× ^^^^( ^^( ˆ^^−1,^ˆ^), ^^^^) where ^^^^is some metric measuring distortion between the ground-truth frame ^^^^, and the decoded frame ˆ^^, ^^^^is some metric measuring distortion between the ground-truth frame ^^^^and the decoded previous frame ˆ^^after a warping operation ^^ with the decoded flow, ^^ is bits per pixel of the current frame, and ^^^^, ^^^^, ^^^^are corresponding coefficients for rate, distortion and interpolation losses for the current p-frame. Note that the flow interpolation loss can be applied to any combination of RCN and FCN including the standard flow-residual video compression, flow composition and hyper module composition. A weighting may be applied to each of the losses used in the training process. These weightings may be changed between training steps. A video compression training scheme may be be applied in which the early stages of training use flow distortion loss that is replaced with flow interpolation loss as the training progresses. At the training step ^^, the current value for ^^MSE FLOWand ^^MSE WARPEDmay be calculated using the following formula: ^^^^= ^^ + ( ^^ / ^^) × ( ^^ − ^^), for 0 <= ^^ <= ^^ , Where ^^ and ^^ are the starting and final values, and ^^ is the number of iterations in annealing. The training loss may then be: ^^ = ^^^^× ^^ + ^^^^^^× ^^^^( ^^^^, ˆ^^) + ^^^^^^ × ^^^^( ^^( ˆ^^−1,^ˆ^), ^^^^) + ^^^^^^× ^^^^( ^^^^,^ˆ^) This section details concepts regarding B-frame coding in AI-based video compression, which concern the encoding and decoding processes of frames referencing previously decoded past and future reference frames. In video compression, upon encoding a video sequence, a frame can be assigned to a certain frame type based on which frames are being referenced during the encoding and decoding of that frame. Specifically, the most common frame types are: • I-frame: A frame that is encoded without referencing any other frame. Also known as intra-frame. • P-frame: A frame that references past frames (chronologically) for encoding / decoding. • B-frame: A frame that references both past and future frames (chronologically) encoding / decoding. P-frames and B-frames may be jointly known as inter-frames. When B-frames are part of the video sequence, the encode order is different from the display order which is chronological. The encode order without B-frames enabled (i.e. low-delay mode) is visualised in the left image in Figure 16, and with them enabled in the right image of Figure 16. The encode order in each case is shown below each image. In practice, since the B-frame can only be decoded once a future P-frame has been decoded, a frame buffer (and buffers for other side information, if relevant) must be contained within the decode system to store previously decoded frames. Normally, once all preceding B-frames as well as the P-frame has been sent to display, the buffer can be cleared. The composition-based flow-residue frameworks discussed above may be extended to accom- modate bi-directional information. Each P-frame in the sequence references past frames during coding, most commonly the frame at the previous frame index. In contrast, to enable B-frame coding, a non-chronological encode order must be adopted in the coding. In the illustrated example in Figure 16, frame ^^1is being coded as an I-frame, ^^5is being coded as a P-frame which references ^^1, followed by ^^3being coded as a B-frame which references both ^^1and ^^5. ^^1, ^^3and ^^5may be considered a first, second and third frame in this example, where the second frame may be decoded after the first and third. The same logic applies for ^^2and ^^4, which can also be coded as B-frames since they can reference both past and future frames (ideally the "closest" frames available to them, so ^^1and ^^3for ^^2, and ^^3and ^^5for ^^4). An approach to incorporate B-frame coding is the following: given ^^1and ^^5and their reconstructed counterparts ˆ1and ˆ5, in order to encode the B-frame ^^3we estimate the past flow map ^^1,3= FlowModel( ^^1, ^^3) and future flow map ^^5,3= FlowModel( ^^5, ^^3) with an arbitrary motion estimation model. We then input these in the encoder of the flow composition model (the flow encoder / decoder), along with other relevant inputs that can be used for conditioning the flow compression (for example, warped and non-warped reference frames and the frame itself). The flow composition model produces a bitstream, which can be decoded into the reconstructed flow maps1ˆ,3and5ˆ,3. Then, we use these on the decode side to warp the reconstructed reference frames ˜1,3= warp( ˆ1,1ˆ,3) and ˜5,3= warp( ˆ5,5ˆ,3). These are the resulting past and future flow which we can use to compute the residual frame. Since we use a composition module as the compression pipeline for the residual frame, we can simply feed in ˜1,3and ˜5,3along with the original frame to be encoded ^^3. The information existing at the decoding side (which includes ˜1,3and ˜5,3) can be provided to the decoder in order to output the reconstructed frame ˆ3. Algorithm 1 describes the aforementioned encoding and decoding process in detail. Without composition models, it is also possible to incorporate B-frame modelling in the pipeline by weighing information from the past and future reference frames to obtain a motion-compensated reference frame ¯ used for residual coding. For instance, it could be a weighted sum of warped past and future reference frames ¯3= ^^ · warp( ˆ1,1ˆ,3) + ^^ · warp( ˆ5,5ˆ,3) where ^^ and ^^ are first and second The weighting parameters may be learned parameters and can either be real or set to 0 and 1. Motion-compensated frame is an alternative way to refer to the warped frame as discussed above. The flow from the past and future reference frames may also be combined before being used with any previous reference frame to obtain the motion-compensated frame. Algorithm 1 B-frame coding with residual composition model of frame index ^^ 1: Inputs: ^^^^−: past (original) reference frame ^^^^+: future (original) reference frame ˆ^^−: past reconstructed reference frame ˆ^^+: future reconstructed reference frame ^^^^: current B-frame to be encoded 2: Outputs: ^^^^ , ^^: flow bitstream for encoded B-frame ^^^^ , ^^: residual bitstream for encoded B-frame ˆ^^: reconstructed B-frame 3: 4: ^^^^−, ^^← FlowModel( ^^^^−, ^^^^) ⊲ Motion estimation from past frame 5: ^^^^+, ^^← FlowModel( ^^^^+, ^^^^) ⊲ Motion estimation from future frame 6: ^^^^ , ^^← FlowEncoder( ^^^^−, ^^, ^^^^+, ^^, ...) ⊲ Ellipses indicate other conditioning inputs 7:^ˆ^−, ^^,^ˆ^+, ^^← FlowDecoder( ^^^^ , ^^) 8: ˜^^−, ^^← warp( ˆ^^−,^ˆ^−, ^^) ⊲ Motion compensation of past frame 9: ˜^^+, ^^← warp( ˆ^^+,^ˆ^+, ^^) ⊲ Motion compensation of future frame 10: ^^^^ , ^^← ResidualEncoder( ^^^^, ˜^^−, ^^, ˜^^+, ^^, ...)11:ˆ^^← ResidualDecoder( ^^^^ , ^^, ˜^^−, ^^, ˜^^+, ^^, ...) ⊲ Composition decoder takes in decode-side information Though B-frame coding normally bootstraps from coded I-frames or P-frames in the buffer, subsequent B-frames can reference already coded B-frames as well. As such, a coding hierarchy emerges where we code in a "hierarchical" manner as opposed to a "sequential" (like in low-delay mode). One concept that arises as a natural consequence due to this hierarchical MEMC structure is flow composition. Flow composition for B-frames can be done by conditioning the flow compression on the so called top-level flow. This is defined as the reconstructed flow map that describes the motion between the two (past and future) reference frames. For example, if ^^1, ^^2and ^^3is an I-frame, a B-frame and a P-frame, respectively, the reconstructed flow map1ˆ,3would be the top-level flow when compressing the flow maps ^^1,2and ^^3,2for the B-frame. Since the top-level flow exists on both encoding and decoding side, we can condition both the encoder and decoder with it, which should yield substantial gains since in theory there should be a fair amount of redundant information between1ˆ,3, ^^1,2and ^^3,2. The relationships between the flows are illustrated in Figure 17, which shows a visualisation of the active top-level flows 1710 in a B-frame coding scheme with flow composition. Algorithm 2 describes the encoding and decoding process in detail. Algorithm 2 B-frame flow composition of frame index ^^ (residual coding step described in Algorithm 1) 1: Inputs: ^^^^−: past (original) reference frame ^^^^+: future (original) reference frame ^ˆ^−, ^^+: top-level flow between reference frames ^^ B-frame to be encoded 2: ^^^^ , ^^: flow bitstream for encoded B-frame ^ˆ^−, ^^,^ˆ^+, ^^: (potential) top-level flow for subsequently coded B-frames 3: 4: ^^^^−, ^^← FlowModel( ^^^^−, ^^^^) ⊲ Motion estimation from past frame 5: ^^^^+, ^^← FlowModel( ^^^^+, ^^^^) ⊲ Motion estimation from future frame 6: ^^^^ , ^^← FlowEncoder( ^^^^−, ^^, ^^^^+, ^^,^ˆ^−, ^^+, ...) ⊲ Composition encoder takes in top-level flow 7:^ˆ^−, ^^,^ˆ^+, ^^← FlowDecoder( ^^^^ , ^^,^ˆ^−, ^^+) ⊲ Composition decoder takes in top-level flow 8: ⊲ Residual coding same as in Algorithm 1 Global flow Video compression pipelines often employ flow computation as an integral part of exploiting frame-to-frame redundancies. In several cases, the flow between two frames contains motion that can be parameterized by global motion parameters (e.g. panning motion or zooming in), and is often called a global flow. Computing global flows can alleviate unnecessary computation and bits that would otherwise be spent for computing a flow on a point-to-point level. This section will describe methods to compute global flows and outline how they may be integrated into a video compression pipeline. We will use ( ^^, ^^′) to denote two feature-matched points in two distinct frames ^^, ^^′∈ R^^× ^^×3, where ^^, ^^ denote the height and width of the video sequence. In other words, if ^^ and ^^′ are two images with corresponding ( ^^, ^^′) pair then ^^ ( ^^) = ^^′( ^^′). In general ^^, ^^′∈ R2denote coordinates, which can either be indexes or real numbers discretizing a unit interval. Features here can refer to pixel intensity or some other computed feature (learned or otherwise). We will also use ^^ ^^ ^^ ^^ : R^^× ^^×3× R^^× ^^×3→ R^^× ^^×2to denote a function that computes the point-wise flow between two input frames for each point in R2. Consequently, we will use ^^ to denote the flow between ^^ and ^^′: ^^ = ^^ ^^ ^^ ^^ ( ^^, ^^′). We will use ^^ ^^ ^^ ^^( ^^, ^^) : R^^× ^^×3× R^^× ^^×2→ R^^× ^^×3to denote a function that aligns / warps ^^ using ^^. with a single matrix ^^ such that ^^′ = ^^ ^^ (1) where ^^ can represent any linear transformation. Such transforms include translation, rotation, affine, and projective. Below we define some of these transformations explicitly. Figure 18 shows a visualization of different linear global flows for a visualization of how different linear global act on their input. Examples of linear transformations include a translation 181, a Euclidean transformation 182, a similarity transformation 183, an affine transformation 184 and a projective transformation 185. Translation matrices take the following form: ^ ^^^^1 0 ^^^^^^^^^^ ^^where ^^^^, ^^^^∈ R represent the ^^ coordinates respectively. Rotation matrices take the following form: ^ cos ^^ sin ^^ ^^^^^^^^^^^^^^ ^^where ^^ is the angle of rotation coefficients as before. Affine matrices take the following form: ^ 1 + ^^00^^01^^^^^^^^^^^^^^ ^^ where ^^00, ^^01, ^^10, ^^11∈ R are the affine parameters that include shearing, and ^^^^, ^^^^denote the translation coefficients as before. A projective matrix, also dubbed a homography, generalizes the affine transform by including perspective shifts. This adds another row to the matrix as follows: ^^1 + ^^ ^^ ^^^^00 01 ^^^^^^^^^^^^ is compensated for in Equation 1 ^^ vector. Methods to compute linear global flows will now be discussed. We can use a neural network, ^^ ^^ , that takes a ( ^^, ^^′) pair and outputs a parameters for a transformation matrix ^^ that characterizes a global transformation between ^^ and ^^′. ^^ ^^ can be a pretrained neural network or learned during training. Moreover, this is done on the encode side, so that during encoding we would perform the following computation: ^^00, ^^01, ^^02, ^^10, ^^11, ^^12, ^^20, ^^21, ^^22= ^^ ^^ ( ^^, ^^′) which would then be used in a transformation matrix ^^: ^^^^^00^^01^^02^^^^^^^^^We can compute ^^ using a linear this case we assume we have a dense flow, ^^ ∈ R^^ ^^ ^^3(the last dimension is just padded with 0s), defined for every point in the frame, and fit a matrix ^^ to it using a least-squares regression. ^^ can be generated by a flow estimation network. In this framework, our set of points, denoted ^^ ∈ R^^ ^^ ^^ ^^, is the set of independent variables, and ^^ is the set of observed dependent variables. Here, ^^ ∈ {2, 3} depending on the transformation we seek. We therefore solve for ^^ ∈ R^^ ^^3via the following minimization problem: min ∥ ^^ ^^ − ^^∥2^^2The Fourier-Mellin method is a FFT based method that can be used to generate the parameters that characterize scaling, rotation, and translation. These would then be used to generate a corresponding matrix, as discussed in above. In other words, given a ( ^^, ^^′) pair, if we denote ^^ ^^ as the Fourier-Mellin function that outputs the aforementioned parameters we have: ^^, ^^, ^^^^, ^^^^= ^^ ^^ ( ^^, ^^′) where the output would be used in the following transformation matrix: ^ ^^^^ cos ^^ ^^ sin ^^ ^^^^^^^ ^^^^^ ^^ We can compute ^^ by directly error between two frames ^^ and ^^′: ∑ ^ min ∥ ^^′( ^^) − ^^ ( ^^ ^^ ^^ ^^( ^^, ^^))∥2^^2^^ Depending on how we matrices as described in the previous section. Once we find a global flow ^^, we have to integrate it into the video compression pipeline. There are several ways to do this, as we outline below. One of the main advantages of having a global flow ^^ as a 3 ^^3 matrix, is that it does not cost much in terms of BPP to send ^^ in the bitstream. Therefore, we can assume we have ^^ on both the encode and decode side. Moreover, we will also use ^^^^to denote the global flow ^^ defined over the entire grid with the same shape as ^^. In general, the global flow as discussed above may be considered a first flow. The local or point-wise flow may be considered a second flow. Each of these flows may be determined between a first frame and a second frame of a video and encoded using a neural network into a latent representation in a video compression pipeline. The latent may be decoded using a neural network to obtain an output flow that is then used to obtain an output frame. Specific methods to impliment this concept are set out below. The first method approaches the flow computation in two parts: we first warp the previous frame with ^^ and then compute the flow between the ^^-warped frame and the current frame. This can be summarized with the following equation: ^^′ = ^^ ^^ ^^ ^^ ( ^^ ^^ ^^ ^^( ^^, ^^), ^^′) The idea is that the new flow ^^′ will be easier to compress or more expressive than ^^, since it does not contain global motion. We will encode both ^^ and ^^′. On the decode side, we will first decode ^^′, then warp the existing reconstructed frame, and then warp it again with the global flow. This method is outlined in Algorithm 3. Algorithm 3 Algorithm detailing point-wise flow computations between globally-aligned frames Require: Flow Encoder function ^^^^^^Require: Flow Decoder function ^^^^^^Require: Flow Encoder function ^^^^^^ Require: Flow Decoder function ^^^^^^ Require: Global Flow function ^^^^Require: Pointwise Flow function ^^^^Require: Warping function ^^ Input: ^^^^−1, ^^^^ENCODE ^^^^← ^^^^( ^^^^−1, ^^^^) ⊲ Compute Global Flow between ^^^^−1and ^^^^^^^^← ^^( ^^^^−1, ^^^^) ⊲ Warp ^^^^−1with ^^^^to get a globally-aligned frame ^^^^← ^^^^( ^^^^, ^^^^) ⊲ Use flow estimation network to get local flow ^˜^← ^^( ^^^^, ^^^^) ⊲ Warp globally-aligned frame to get motion-compensated frame ^^^^^^←E^^^^( ^^^^, ^^^^) ⊲ Encode ^^^^using flow composition network ^^^^^^←E^^^^ ( ^^^^, ^^^^−1,ˆ^f^i−na1l, ^^^^,^˜^) ⊲ Encode residual using residual composition network DECODE ^ˆ^^^←D^^ˆ^^^^^^^^^) ⊲ Recover reconstructed flow ^ˆ^^^⊲ Recover reconstructed residual ˆ^^⊲ Warp previous reconstructed frame^ˆ^−1withˆ′⊲ Warp intermediate frameˆ^^with global flow^ˆ^⊲ Add back reconstructed residual ˆ^^to recover final reconstruction In the second method, instead of encoding ^^ with the flow compression network, we will encode ^^ − ^^^^, which is the original flow with the global flow removed. Therefore, on decode side, we will first decode ^^ − ^^^^and then add back ^^^^. This method is outlined in Algorithm 4. Algorithm 4 Algorithm detailing flow residual compression network Require: Flow Encoder function ^^^^^^Require: Flow Decoder function ^^^^^^Require: Flow Encoder function ^^^^^^ Require: Flow Decoder function ^^^^^^ Require: Global Flow function ^^^^Require: Pointwise Flow function ^^^^Require: Warping function ^^ Input: ^^^^−1, ^^^^ENCODE ^^^^← ^^^^( ^^^^−1, ^^^^) ⊲ Compute Global Flow between ^^^^−1and ^^^^^^^^← ^^^^^^ ^^^^⊲ ^ˆ^ ^^ ˆ^^^ˆ^ In the third method we input ^^^^and ^^^^into a flow composition network to generate a latent vector to be compressed. Here, we use ^^^^to denote the point-wise (local) flow between two frames that have been globally aligned. Since we can also send ^^ in the bitstream without much cost in BPP, we can also almost losslessly generate ^^^^on the decode side and also concatenate it within the network on the decode side. This method is outlined in Algorithm 5 Algorithm 5 Algorithm detailing flow compression using flow composition networks Require: Flow Encoder function ^^^^^^Require: Flow Decoder function ^^^^^^Require: Flow Encoder function ^^^^^^ Require: Flow Decoder function ^^^^^^ Require: Global Flow function ^^^^Require: Pointwise Flow function ^^^^Require: Warping function ^^ Input: ^^^^−1, ^^^^ENCODE ^^^^← ^^^^( ^^^^−1, ^^^^) ⊲ Compute Global Flow between ^^^^−1and ^^^^^^^^← ^^^^^^ ^^^^^ˆ^ ˆ^^^ˆ^ The methods discussed thus far are concerned with using linear global flows. Comparing with Equation 1, we can also consider feature-matched point transformations that take the following form: ^^′ = ^^ ( ^^, ^^′, ^^; ^^) (2) where ^^ is a function that can either be optimizer for, predicted, or learned. In this case, ^^ takes the dual role of both computing the flow and applying it in a nonlinear fashion. The global flows discussed here can also be computed on a block level. This involves partitioning the image into blocks (square or rectangular) and then computing the global flow in each block (assuming consistency of blocks between frames), refer to Figure 17, which shows a visualization of block-based global flows. The original input image 5 may be partitioned into a plurality of inout image blocks 195. This is useful because some videos only contain local (i.e. distinct pixel-to-pixel) flow in a small number of blocks, whereas the remaining blocks of the video may have the same uniform motion between frames. Metanetworks In this section, the concept of using a metanetwork to produce parameter modifications of the neural networks used in an AI based compression pipeline is discussed. As discussed above, in neural video compression, we train a pair of neural networks ( ^^ , ^^) (parameterised by ( ^^, ^^) respectively) such that 1. ^^ ( · ; ^^) maps a finite sequence of frames x = ( ^^0, ^^1, ... , ^^^^) to some latent represen- tation y = ( ^^0, ^^1, ... , ^^^^) 2. the latents ^^^^are then quantised and transmitted to the decoder as an entropy-coded bitstream 3. the quantised latents^^̂are then mapped by ^^( · ; ^^) to a reconstruction x̂ = (0̂,^^̂, ...^^̂) of the original frames. This is typically trained under a rate-distortion loss function of the form L^^ ^^( ^^, ^^) = Ex∼ ^^x[ ^^ ^^( ^^ (x; ^^)) + ^^ (x, ^^( ^^ (x; ^^); ^^))], (3) where ^^xis a trade-off parameter, ^^ is a rate loss function and ^^ is a distortion function. The parameters ¯¯obtained by successful implementation of this training procedure are (locally) optimal in expectation over the data distribution. However, given a particular video sequence x∗, they are not typically optimal. It is possible to finetune or “overfit” the parameters ^^, ^^ to a specific video sequence x∗ by taking a trained encoder / decoder with parameters¯ ¯and performing further optimisation steps with only samples from x∗rather than from the training distribution. This is then trained under the modified loss. L^^ ^^ ^^( ^^^^, ^^^^) = L^^ ^^(¯+ ^^^^,¯+^^̂) + ^^ ^^′(^^̂), (4) where ^^^^, ^^^^are updates may be referred to as parameter modifications. Note that ^^^^must be quantised and transmitted to the decoder and that the rate cost of these additional bits is incorporated into the ′ training objective by the model rate loss function ^^ . Figure 20 shows a schematic overview of a finetuning architecture. In the example shown in figure 20, the components of a hyper-network as described above are not shown but may be present. Components not discussed further may be considered equivalent to the components shown in figure 1. This method is effective for adapting a trained neural video compression model for a particular sequence, but may be computationally costly as it requires running many steps of a numerical optimisation algorithm for each sequence that we wish to adapt the model to. A metanetwork is an additional neural network that can be trained either alongside the neural compression model or on its own after a compression model has already been trained. The finetuning procedure described previously finds a mapping from a video sequence x∗ to a set of parameter modifications ^^^^, ^^^^by means of an iterative numerical optimisation algorithm such as stochastic gradient descent. The key idea behind the metanetworks is to train a neural network ℎ to learn this mapping x∗↦→ ( ^^^^, ^^^^). Once ℎ has been trained, it can produce the parameter modifications, which updates to adapt a compression model to a new sequence, in fractions of a to the hours or days of training required in finetuning. The parameter modifications may be used to update at least one of the encoder and decoder of the compression pipeline. At least some of the parameter modifications may correspond to the encoder and at least some of the parameter modifications may correspond to the decoder. The parameter modifications may replace, update or perturb one or more of the parameters of the encoder or the decoder, for example the weights of the encoder or the decoder. The metanetwork may transform a sequence of frames into a set of weight updates for a neural compression model in two steps: 1. First, the frames may be passed through a feature extraction network (FNet) which summarizes the information contained in the sequence into a feature map. This can either be a two-stage process in which features are extracted from each frame and then aggregated or the extraction and aggregation can be done jointly 2. Finally, for each weight that we wish to update, the features are processed by a weight update network (UpdateNet) to produce the weight update Figure 21 shows a schematic overview of metanetwork architecture. In the example shown in figure 21, the components of a hyper-network as described above are not shown but may be present. Components not discussed further may be considered equivalent to the components shown in figure 1. At its most generic, the FNet may be a simple feed-forward convolutional neural network (CNN) which may be followed by a global average pooling layer which aggregates the final convolutional layer output into a feature map of fixed size irrespective of the original frame dimensions. The amount of downsampling, the number of channels and the number of layers can be adjusted to fit a given computational budget. The convolutional layers can be two-dimensional (if features are extracted independently for each frame) or three-dimensional across a plurality of frames (if features are extracted jointly). If the frame features are extracted independently, there are several ways in which they can be aggregated, including: • simple temporal averaging across frames • recurrent neural networks (RNNs) and their variants, such as – simple RNNs – long short term memory networks (LSTMs) – gated recurrent networks (GRUs) • one-dimensional CNNs followed by further aggregation (such as averaging) • transformer networks or other attention-based networks followed by further aggregation (such as averaging) Aggregation can also be performed on the output of the FNet and on one or more sets of parameter modifications in the same manner. An example architecture of a FNet is as follows: Input: T x 3 x H x W video Conv192↓2-5x5 ReLU 192 channel output, downsample by 2, 5x5 kernels Conv192↓2-5x5 ReLU Conv192↓2-5x5 ReLU Conv256↓2-5x5 Intermediate features: T x 256 x H / 16 x W / 16 Global Average Pooling Spatially aggregated features: T x 256 Average in time dimension Output (V): 256 The purpose of the UpdateNet is to transform the features extracted from the video sequence into a weight update. The Updatenet may receive an input with a first number of elements and produce an output with a larger number of elements. This can be done with a variety of neural network architectures depending on training time and memory constraints, and the architectures may include: • dense linear transformations • sparse linear transformations • low-rank linear transformations such as tensor networks (TNs) • multi-layer perceptron (MLP) networks • networks combining features from the above Furthermore the UpdateNets may: • be entirely independent from one another • share a common “trunk” before diverging An example architecture of an UpdateNet is as follows: Input (V): 256 Linear128 ReLU LinearN - (here N is the number of elements in the parameter(s) that the network is predicting) Output: N parameter updates In order to quantize and losslessly compress the weight updates, we need to specify an entropy model. This model could be: • a global entropy model (shared across all weight updates) • a per-subunit entropy model (shared across some network sub-unit), where the sub-unit could be – a layer of the compression network – a parameter tensor in a layer – a channel in a parameter tensor Furthermore, the probability model can be • A fully-factorised model where each element is modelled independently from the others • A multivariate model where the joint probability distribution is specified over multiple weight updates • A model featuring some conditional dependencies between the weight updates The probability model can be arbitrary, but common choices of distribution include: • the Laplace distribution • “Spike and slab” type distributions The parameters of the chosen distribution and the quantization bin size can be: • specified as fixed values • learned during the training process of the metanetwork and then fixed • predicted in inference as a function of the input by a hypermetanetwork An example of the excution of a metanetwork is set out below. Algorithm 6Metanetwork execution Require: Batched video sequence ^^ Require: Quantization function ^^ Require: Generic compression model encoder weights Φ and decoder weights Θ Require: A trained feature extractor FNet Require: A collection of trained weight update networks {UpdateNet^^} Ensure: Instance-adapted compression model weights Φ and Θ ^^ ← FNet( ^^) ^^( ^^)in Φ do UpdateNet^^( ^^) ^^( ^^)+ ^^(^^^^) Θ do ^^( ^^ ) ^^← UpdateNet^^( ^^) (^^ )^^̂ ← ^^ ^^(^^^^ )^^ ) PID Controller In this section, a control loop mechanism used for controlling a given process variable (i.e. ^^ ^^) for reaching a desired setpoint (i.e. ^^ ^^) by adjusting the value of a control variable (i.e. ^^ ^^ or as denoted below with ^^( ^^)) is discussed together with its application to an AI based compression process. A PID controller is an example of such a mechanism. Such a control loop is equivalent to bringing the error which may be defined as the difference between the current value of the process variable and the setpoint ^^( ^^) = ^^ ^^ ( ^^) − ^^ ^^ close to 0. Note, with ^^ we denote the dependency of the process on time. The setpoint may be predetermined, for example set at a particular value for the entire training process. Alternatively, the setpoint may be modified during the training process, for example by increasing or decreasing a particular setpoint between at least one of the training steps. The PID controller changes the control variable, based on the values of ^^( ^^) with an integral, proportional and derivative terms: ∫^^^^( ^^) = ^^^^ ^^( ^^) ^^ ^^ + ^^ ^^ ^^( ^^) + ^^ ^^^^ ^^( ^^)(5) ^^ ^^ The controller may parameterised by the three term coefficients ^^^^, ^^^^, ^^^^. These parameters may scale the contribution of each of the terms to the control process. The parameters may be set to zero to remove one or more of the terms. An analogous definition of the PID controller can be obtained by differentiating both sides of eq.5 w.r.t. ^^ and setting initial value for ^^(0): ^^ ^^( ^^)2=^^ ^^( ^^) + ^^^^ ^^( ^^)+^^^^( ^^) ^^ ^ ^^(6) ^^ ^^^^^ ^^^^^^ ^^2Often the PID controller is applied in a discrete setting (i.e. the time variable ^^ is discretised). In algorithm 7 we present pseudocode for the discrete algorithm: Algorithm 7 Discrete algorithm for PID control from eq.6. Here the functions error_derivative_estimate(errors, dt) and error_second_derivative_estimate(errors, dt) produce estimates of the first and second order derivatives of error, based on evaluations of the error function at discrete time steps. This can be achieved with finite difference numerical differentiation. Inputs: Discrete time interval: dt Initial value for control variable: control_variable = ^^(0) Integral coefficient: ^^^^Proportional coefficient: ^^^^Derivative coefficient: ^^^^Procedure: control_variable← ^^(0) errors ← [] loop: error← ^^ ^^ − ^^ ^^ errors ← errors + [error] integral← error proportional← error_derivative_estimate(errors, dt) derivative← error_second_derivative_estimate(errors, dt) control_variable_delta← ^^ ^^× integral + ^^ ^^× proportional + ^^ ^^× derivative control_variable← control_variable + control_variable_delta wait(dt) goto loop In the usual setting the PID controller is used for controlling processes, determined by a deterministic function. However, whenever the process is stochastic in nature, it is helpful to apply smoothening (for example, some form of averaging) to the deltas that change the control variable. This means that the value of at least one previous update to the control varaible is used during the current update. control_variable_delta could be interpreted as a control_variable_step (similar to a gradient step in gradient-based optimisation). A way of achieving this smoothening is by passing control_variable_step through a gradient descent / ascent optimiser (e.g. SGD, Adagrad, RMSProp, Adam). Many of those stochastic optimisers feature a momentum term, which applies an exponential moving average of updates. Such a process can be added to the traning process of an AI based compression system as discussed above. With the optimiser in mind the updated algorithm is presented in algorithm 8

[0002] Algorithm 8 Modified version of the discrete algorithm for PID control from algo.7 with the addition of gradient descent / ascent optimiser. Inputs: Discrete time interval: dt Initial value for control variable: control_variable = ^^(0) Integral coefficient: ^^^^Proportional coefficient: ^^^^Derivative coefficient: ^^^^Gradient descent / ascent optimiser: optim (e.g. Adam) Optimiser’s initialisation parameters: optim_params (e.g. learning_rate, momentum_coefficient) Procedure: control_variable← ^^(0) errors ← [] optim = optim(optim_params) loop: error← ^^ ^^ − ^^ ^^ errors ← errors + [error] integral← error proportional← error_derivative_estimate(errors, dt) derivative← error_second_derivative_estimate(errors, dt) control_variable_step← ^^ ^^× integral + ^^ ^^× proportional + ^^ ^^× derivative optim.add_new_step(control_variable_step) control_variable_modified_step ← optim.get_modified_step() control_variable← control_variable + control_variable_modified_step wait(dt) goto loop The optimiser-based discrete PID controller can be employed in the context of learning-based compression for controlling at least one variable that is updated during a learning step of an AI based compression process. For example, a PID controller may be used for controlling different metrics (i.e. process variables) in the loss function of an AI based compression process. Corresponding control variables, for example, may be the learning rates of one or more of the neural networks used in the AI based compression process. An example of such process variable is the fake_out value produced by the discriminator in a compression pipeline, which indicates what is the discriminator’s estimated likelihood of a compressed image or video of being real as discussed above. A stronger discriminator would results in low fake_out values, while a weaker one - in high fake_out values. There is an optimal fake_out value that leads to the best adversarial training between the encoder-decoder and the discriminator. In order to reach a desired value for fake_out, one option is to change the learning rate of the discriminator (i.e. lr_disc or our control variable). The relationship between the lr_disc and fake_out is an unknown function and this is why a controller is desired. Other options of a control variable for controlling the fake_out are lambda_adv (i.e. the multiplier for the adversarial loss of the generator) and lambda_jp (i.e. the multiplier for the Jacobian penalty loss of the discriminator). However, as discussed above, this is an example of the application of a controller to the discriminator, and the PID controller may alternatively be applied where the process variable is the output of one or more neural networks of the AI based compression process and the control variable is the learning rate or another variable that affects the output of the one ore more neural networks. Another example of how the optimiser-based PID controller can be used in the context of learning-based compression is for rate control. In that scenario the multiplier for the rate or the distortion loss could be used as a control variable. Parallel RANS Variational latent-variable models are a popular choice for modern learnt image and video compression. In short, successful variational latent-variable models can represent relevant information in the data they are trained for in a compact fashion. This can be achieved by using a variational auto-encoder, which can be trained in such a way that given some uncompressed image or video frame, ^^, produces two outputs, a latent representation, ^^, and a set of entropy parameters ^^^^, ^^^^, based on the distribution of the latent representation. The entropy parameters may be referred to as probability parameters. The universal representation of any compressed data today is the binary channel, which may be restricted compared to the richness of latent representations obtained from variational latent-variable models. Thus, for practical compression, the latent representations need to be converted to a binary bistream, as discussed above. An efficient method for this is done through entropy coding, a form of source coding that leverages the relevance of the entropy as a measure of a random variable’s information content. Given the latent representation, ^^, and its entropy parameters, ^^, the latent, ^^, can compressed using entropy coding as discussed in more detail in the context of an AI based compression process above. The entropy parameters are an example of probability parameters that describe the probability distribution of the latent representation ^^. For such we need a lossless entropy coding algorithm. Asymmetrical Numeral Systems (ANS), or more specifically its range variant, known as RANS is such an algorithm and provides close to optimal performance in terms of compression rate. ANS and RANS algorithms are discussed in more detail in Duda, Jarek. "Asymmetric numeral systems: entropy coding combining speed of Huffman coding with compression rate of arithmetic coding." arXiv preprint arXiv:1311.2540 (2013), which is hereby incorporated by reference. In this section, we outline examples of methods of parallelisation of the RANS algorithm in AI based compression pipelines, which may not adversely affecting the compression rate. RANS is a serial algorithm, meaning one operation executes after the other in a serial fashion. The flow of a standard RANS implementation, for both an entropy encoding process 2200 and entropy decoding process 2201, can be observed in Figure 22, which shows a schematic example of a serial RANS pipeline. To encode, an input set of symbols ^^ and entropy parameters ^^ are used in a RANS encoding process 2210 to obtain a bitstream 130. To decode, the bitstream 130 and ^^ undergo and RANS decoding process 2220 to retrieve ^^. Below is described an example of a parallel version of the RANS algorithm for learnt image and video compression, which modifies on the original implementation in at least 2 main ways. Such an implementation may be incorporated into the AI based compression pipeline discussed above at the entropy encoding process 150 to produce a bitstream 130, followed by the entropy decoding process 160 to obtain the transmitted latent representation, where the entropy encoding process 150 and the entropy decoding process 160 are modified as follows (The flow of an example parallel implementation, for both the encoder and decoder, can be seen in Figure 23, which shows a schematic example of a parallel RANS encoder 2300 and decoder 2301 pipeline): 1. When encoding, the original set of symbols, ^^, and entropy parameters, ^^, are split into ^^ chunks, ^ˆ^^^and ^ˆ^^^, respectively, which may be referred to as sub-latent representations and sub-probability parameters. The chunks or sub-representations may have different numbers of elements. For example, When splitting the original set of symbols, ^^, a load balancing process 2310 be applied to ensure that the chunks, ^ˆ^^^, have evenly distributed or equivalent computational complexity, which may improve decoding speeds and throughput. A set of indices or tags, ^^^^ ^^ ^^may also produced at this stage, these provide starting positions for each chunk, ^ˆ^^^, into the original set of symbols, ^^. This may be sent with the output, ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^, to allow the decoder to reconstruct the original set of symbols, ^^, from the chunks ^^^^in a symbol concatination process 2360. 2. RANS encoding and decoding are applied across symbol chunks, ^ˆ^^^, concurrently, meaning that a set number of threads will encode / decode chunks asynchronously. This means that the sub-latent representations may be encoded and decoded in parallel in entropy encoding processes 2320 and entropy decoding processes 2350 respectively. A thread pool may be used to manage the execution of encoding and decoding chunks, where chunks are encoded / decoded on threads as they become available. When encoding, a ’sub-bitstream’, ^^^^, is generated for each chunk, ^^. These ’sub-bitstreams’ undergo a concatenation process 2330 to give, ^ˆ^. A set of indices, ^^^^ ^^ ^^, are also produced, which indicate the starting positions of each ’sub-bitstream’, ^^^^, in, ^ˆ^. This is sent with the output, ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^, and allows for ^ˆ^ to be decomposed by a bistream parsing process 2340 into its original ’sub-bitstreams’, ^^^^, for decoding . State overhead exists in all implementations of the original RANS algorithm. When encoding with RANS, we keep a running state and a stack. The stack is the bitstream, ^^, and the state can be defined as ^^. The state is used as a temporary buffer for encoding symbols, from which bits are moved onto the stack once the state becomes full. At the end of encoding, the state is flushed and all bits in the state are added into the stack. For example, say we have a 32-bit state and a stack made up of 8-bit data types. Each time we encode a symbol we push its new RANS encoded bits onto the 32 bits state, adding bits. Once the state reaches its maximum size we move 8 bits from the state into the stack, freeing 8 bits in the state for further RANS encoded bits. This pattern is repeated until all symbols have been encoded and we are left with a final state, ^^ and a stack ^^. To complete the RANS encoding we need to flush the state, which means moving all of the 32 bits onto the stack. In the case of the example parallel implementation described above, we may expand on the above notion, where each of the ^^ chunks, ^ˆ^^^, has its own stack, ^^^^, and corresponding states, ^^^^, during encoding. If we do not flush the state, ^^^^, to the sub-bitstream, ^^^^, at the end of the encoding, we can define our sub-bitstreams as, (( ^^0, ^^0), ( ^^1, ^^1), ..., ( ^^^^, ^^^^)). This means the concatenated bitstream can be defined as follows; ^ˆ^ = concat(( ^^0, ^^1, ..., ^^^^), ( ^^0, ^^1, ..., ^^^^)) (7) If each state, ^^^^, is 32 bits in size, we can have anywhere in [0, 31] bits that are not required for the theoretical optimal compression rate. These redundant bits may affect our compression rate and stop us from achieving optimal performance. As the number of states grows in line the number of chunks, ^^, it means that there is an inversely proportional relationship between ^^ and compression rate. Once ^^ becomes sufficiently large the state overhead may begin to affect compression performance. In order to achieve optimal throughput and decode speeds from our parallel RANS implemen- tation, we may wish to not limit the number of chunks, ^^. At the same time, we still want to achieve the near-optimal compression performance obtained from using the original serial rANS implementation. Thus we can make the parallel implementation free of these state overhead restrictions, and to do so can minimise the state overhead per chunk. A method for doing such is to take the representation of ^ˆ^ seen in Equation 1.1, and split this into its two components ( ^^0, ^^1, ..., ^^^^) and ( ^^0, ^^1, ..., ^^^^) We concatenate the stacks, or sub-^^as normal, lets call this; ^ˆ^′= ^^ ^^ ^^ ^^ ^^ ^^ (( ^^0, ^^1, ..., ^^^^)) We then take the states, ( ^^0, ..., , RANS once again. The states are therefore entropy encoded independently of their associated stacks. To achieve optimal compression of the states, we must use a distribution that models them appropriately, for this we can use the fact that they follow Benford’s law, and use the Benford Distribution. Using this we can thus encode the states ( ^^0, ^^1, ..., ^^^^) to some new bitstream, ˆ and finally return the new output as ( ˆ ^ˆ^′). The entropy encoded states are thus combined with the sub-bitstreams corresponding to the plurality of latent representations to obtain the bitstream for transmission. In more detail, entropy encoding of the states may be done by splitting the states as follows; ^^0, ( ^^1, ..., ^^^^) We then want to push the elements ^^1, ..., ^^^^onto ( ^^0, ^ˆ^′), using RANS to get near optimal compression. Essentially we want to RANS encode each chunk’s states onto the concatenated bitstream. In order to entropy code (RANS encode / decode) the symbols ^^1, ..., ^^^^we need a distribution to model them. It has been shown that in practice the elements ^^^^follow Benford’s law . This means that the Benford distribution can be used to encode the symbols, ^^1, ..., ^^^^, to ( ^^0, ^ˆ^′). This may enable the previously described parallel implementation of RANS to be used with no restrictions on the number of chunks, ^^, while still achieving close to optimal compression. The modifications to the entropy encoding and decoding steps described above may also be implemented in a AI-based compression pipeline comprising a hyper-network including a hyper-encoder and a hyper-decoder as discussed above. In this case, the modification to the entropy encoding and entropy decoding steps discussed above may also be applied to the entropy encoding step 155 and entropy decoding step 165 of the hyper-network. The state compression solution described above assumes that the distribution of states follows Benford’s law, i.e the probability of state ^^ is proportional to1^^. This assumption may lead to decreased coding efficiency When compared to rANS encoding with per-symbol distributions. We can improve the effectiveness of this state compression method by forcing smaller values of ^^. Given states ^^^^and stacks ^^^^resulting from rANS encoding an input in parallel ^^ chunks as described above, a post-processing step may be applied. In this step, each of the plurality of states is entropy decoded to retrieve a set of symbols, that may be referred to as a state latent. That is, for each ( ^^^^, ^^^^) where ^^ ∈ [1, ^^], we decode until the state has reached some lower bound, which may be flushed to the corresponding bitstream. The resulting symbols may then be re-encoded with ( ^^0, ^^0). We are then left with a set of states that can be encoded very efficiently according to the Benford distribution. An example of this process is shown in figure 24. Four separate chunks are entropy encoded to obtain four sub-bitstreams 2411,2412,2413,2414 and corresponding states 2421,2422,2423,2424. An entropy decoding process 2430 is then performed on all of the states 2422,2223,2424 as discussed above to obtain a state latent, with the reminder of the state flushed to the corresponding sub-bistream 2412,2413,2414. An entropy encoding process 2440 is then performed on the state latent, the result of which may combined with the first sub-bistream 2411 as an additional sub-bistream section 2451 An alternative lossy or lossless compression process may be performed on the state latent prior to the re-encoding. Where a hyper-network is present in the AI based compression process, this post-process step may also be performed in the hyper-network. In addition to this, the post processing step may be performed on the states from the entropy encoding of the latent and the hyper-latent at the same time. This means that the state latent and the hyper state latent obtained in the post-processing step may be combined to obtain a combined latent that is then entropy encoded to obtain a bitstream. The post processing step may also be applied in the case where the AI based compression process is being used to compress, transmit and decompress a video comprising a plurality of frames or input images. More than one input image or frame may be encoded in parallel prior to transmission. In this case, the entropy encoding of a further input image will result in a plurality of further states. The plurality of further states may be entropy decoded to obtain a further state latent, which may be combined with the state latent prior to entropy encoding the resulting combined latent. Further compression rate issues may arise due to parallelisation of an entropy coder such as rANS. Particularly, the production of several independent subsections (the result of encoding a chunk of the input) in the encoded bitstream for a particular input increases the size of required metadata. The metadata required to parse a concatenated bitstream for decoding may include; • the number of chunks • the size of the bitstream corresponding to each chunk. In practice, there are a number of other metadata parameters that also scale with the level of parallelism. Each of these parameters is associated with a rANS state, therefore the size of the metadata scales with the number of independent rANS states. To address the problem of metadata overhead, we may implement a method of persisting rANS states across inputs. Suppose we wish to encode in parallel a set of inputs ^^, ^^, ... for later decoding. First we can create the number of desired rANS states ^^^^, each with their corresponding stacks ^^^^. However, instead of concatenating the states and stacks and appending them to the bitstream we may maintain them in memory. We may then continue to encode further inputs until we desire to serialise them into the bitstream. As such, the metadata overhead is reduced by a factor proportional to the number of inputs we choose to encode in this manner compared to encoding each input independently. Many use cases may require inputs to be decoded in a particular order. For many entropy coders this would not present a challenge for the described method but rANS codes as a stack, whereby symbols are decoded from the bitstream in reverse order of encode. This constraint extends to inputs when we allow rANS states to persist. Therefore, for a set of inputs ^^, ^^, ^^, ^^, we encode in order ^^ → ^^ → ^^ → ^^ so that we may decode in order ^^ → ^^ → ^^ → ^^. A major tradeoff with rANS coding is that, given missing data in the encoded bitstream, it is impossible to decode beyond that missing data. This is because decoding an element of the bitstream updates the rANS state and only the updated rANS state can accurately decode the next element of the bitstream. Since, in order to code with multiple threads, we may split the input symbols into slices each with its own rANS state. In principle, this reduces the penalty of missing data to only not being able to decode the remainder of the slice with missing data. For this, we can; • identify that data is missing • identify which slice(s) the data is missing from • determine how much of the slice(s) is decodable • determine the extent of the output buffer that is unrecoverable (to be zeroed out) • identify the next decodable slice This may be achieved by prepending each slice with some sentinel bytes, eg. [255, 255, 255, 255], and an index. Using a combination of the searchable sentinel bytes, the indices, and the expected sizes of the slices, we can determine when packet loss has occurred and to what extent. Our use case is one where the loss of adjacent data is undesired. We would prefer to distribute the loss evenly across inputs. Instead of encoding symbols in order of their index in the input, we may generate a randomised indexing. The index randomisation algorithm may use a fixed random number generation seed so that it is reproducible during both encoding and decoding. Symbols are encoded in order of the randomised indices and decoded in the reverse order. As a result, the loss of part of the bitstream will result in missing symbols spread across the output rather than a block of adjacent missing symbols. Masked rANS Below is described an example of a masked version of the rANS algorithm for learnt image and video compression, which modifies the original implementation. Such an implementation may be incorporated into the AI based compression pipeline discussed above at the entropy encoding process 150 to produce a bitstream 130, followed by the entropy decoding process 160 to obtain the transmitted latent representation, where the entropy encoding process 150 and the entropy decoding process 160 are modified as follows. It may also be used together with the parallel rANS approach described above. Masked rANS uses the following approach. When encoding, a mask (e.g. a tensor of 1s and 0s) is generated for one or more partitions of the latent representation ^^. The partition may be a channel (e.g. a 2D slice of the latent representation tensor), or some other partition of the latent representation tensor selected based on some other criteria, for example based on one or more entropy parameters ^^ associated with the latent representation and / or one or more of its elements. Specifically, the values of the elements of the latent representation tensor associated with a given partition are combined, for example by calculating a mean or some other statistical parameter of those values, and compared against a threshold. The threshold may be determined empirically or estimated based on, for example, the one or more entropy parameters ^^. The threshold may accordingly be dynamic and vary for different partitions. If the mean or other statistical parameter for the partition is below the threshold, the mask value associated with the individual element values of that partition may be set to 0 to indicate that partition can be skipped and left out of subsequent rANS encoding. Conversely, if the mean or other statistical parameter is above the threshold, the mask value for the individual element values of that partition may be set to 1, indicating the partition should not be skipped and should be rANS encoded in the usual way. Before applying the rANS encoder, the mask is applied to the latent representation ^^ (for example on a partition-wise basis) to produce a masked latent representation ^^^^ ^^ ^^ ^^ ^^ ^^. Finally, rANS encoding is performed on the masked latent representation ^^^^ ^^ ^^ ^^ ^^ ^^to produce the bitstream. The mask itself, if not derivable entirely from the entropy parameters ^^, may also be sent with the bitstream as metainformation. When decoding, the mask is either reconstructed from the metainformation or recon- structed using the entropy parameters ^^ which the decode side has reconstructed, for example, from the hyper network and / or hyper hyper network such as that shown in Figure 2. Reconstructing the mask from the entropy parameters ^^ effectively means no additional bits are needed to apply the presently described masked rANS method. Conversely, reconstructing the mask from meta information has a small additional bit overhead. With the reconstructed mask, the bitstream can now be rANS decoded by the rANS decoder. During rANS decoding, the mask specifies where rANS decoding can be skipped, allowing sections of the buffer into which the decoded bitstream is being stored to be left as 0s until the next non-masked partition of the latent representa- tion tensor being reconstructed is reached. Given that the latent representation ^^ in AI-based image and video compression can be sparse, the effect of masking the latent representation ^^ and skipping rANS encoding and decoding for any masked partitions results in substantial increases in computational efficiency, particularly in terms of a drop of overal operations performed by the CPU, GPU and / or NPU on which the rANS encoding and or decoding is being performed. This in turn leads to an overall drop in power consumption when implementing rANS encoding and decoding in AI-based image and video compression pipelines. Figure 25 illustratively shows a masked rANS encoder and decoder implementation according to the present disclosure using a sequential rANS implementation such as that shown in Figure 22. On the encode side 2500 the latent representation ^^ (e.g. a tensor) is partitioned 2501 into a plurality of partitions. This may be for example based on channels (i.e. 2D slices of the tensor) or based on some other partitioning approach. At a general level, the sparsity of the latent representation ^^ means there are partitions that are empty or mostly empty i.e. they comprise 0s or mostly 0s, it is these partitions that can be masked in the masked latent representation ^^^^ ^^ ^^ ^^ ^^ ^^. In some implementations, it is envisaged that the entropy parameter ^^ or parameters of a given latent representation may be used to partition the latent representation ^^ as the entropy parameters provide an indication as to whether or not a given set of elements of the ^^ tensor are likely to be empty or not. In other implementations the partitioning may perforemd naively, e.g. on a channel basis. The mean or other statistical parameter of the values of each partition is then estimated 2502 and a threshold is applied 2503 to produce a mask value associated with all the elements of that given partition. The process is repeated for all partitions to produce the full mask that associated with a given latent representation ^^. By way of specific example, when performing masking based on channels, the encode side 2500 effectively performs: ^^. ^^ ^^ ^^(). ^^ ^^ ^^ ^^( ^^ ^^ ^^ ^^ = (−1,−2)) to produce tensor of mean values, which is thresholded to produce the mask covering all the values of the latent representation ^^, with mask values associated with each channel being either all 0s or all 1s depending on whether or not a given channel is to be masked or not. In other examples with different partitions, the same reasoning applies and the mask values assocaited with each partition will be all 0s or all 1s as applicable. Together, the mask values make up the mask tensor which is applied 2504 to the latent representation ^^ to produce the masked latent representation ^^^^ ^^ ^^ ^^ ^^ ^^which is fed into the rANS encoder 2210 together with the one or more entropy parameters ^^ in the same way as in Figure 22 to produce the bitstream 130. The rANS encoder skips performing rANS encoding on any elements that are masked, instead filling (or leaving unchanged) any buffer sections associated with any skipped sections with 0s. On the decode side 2507, the mask is recovered 2508 from the metainformation in the bitstream 130 (or from the entropy parameters ^^ directly - not shown) and fed into the rANS decoder 2220 together with the bitstream 130 and the entropy parameters ^^. While decoding, the rANS decoder 2220 initialises the buffer it is using to 0s and sequentially proceeds to decode the bitstream 130 using the entropy parameters ^^. Where the mask indicates that a masked partition section is starting, the rANS decoder skips ahead until the next non-zero, non-masked section starts, leaving the buffer’s 0s unchanged until that point. In this way, the rANS decoder is able to efficiently skip large sections of the latent representation when reconstructing it by simply leaving the buffer values as 0. Finally, the final reconstructed latent representation ^^ is output on the decode side. Figure 26 illustratively shows a masked rANS approach where the mask is estimated from the entropy parameters ^^ alone. This provides the advantage that no additional bits are used to reconstruct the mask on the decode side. The same features as in Figure 25 are present except now instead of a partitioning step on the encode side 2600, a mask estimation step 2611 is performed to produce the mask. Similarly on the decode side 2607, a mask estimation step 2612 is performed on the entropy parameters ^^ to estimate the mask that the rANS decoder uses to to skip partitions of the bitstream 130 during decoding. In order to estimate the mask from the entropy parameters alone, entropy parameters ^^ associated with each value of each element of the latent representation ^^ can be analysed and, for example, where there is little to know variance in entropy parameters for a given sequence of elements, it can be inferred that that sequence is substantially empty and can be masked. It is envisaged that other statistical techniques may be used to analyse the entropy parameters ^^ in to find sequences of values of the latent representation ^^ that are substantially empty. Finally, it will be appreciated that the above techniques may be applied to the above-described parallel rANS approach. In this case, it is envisaged that the mask may generated for each sub-latent of Figure 23, thereby producing a plurality of masks, each which can be used on decode side to skip partitions of each sub-latent when rANS decoding in the same way as described above for the sequential or serial rANS case. In the case of combining masked rANS with parallel rANS, for encode, a mask is generated as described above, as are the set of rANS states that different slices of the latent are encoded into according to the parallel rANS implementation. With channel masking disabled, the latent is evenly divided with each slice encoded into one of the rANS states. When channel masking is enabled, a set of indices is calculated that correspond to the non-masked channels and the channels are used themselves as slices. With this strategy, subsets of the channel slices are allocated for encoding into one of the rANS states. In some implementation, at the processor level, a thread is assigned a rANS state and a set of indices (and sizes) into the latent for which it should begin encoding from. In that way, the masked regions of the latent are not considered for encoding thereby reducing the rANS compute and power usage overheads. For decode, as described above, the buffer is initialised to zeros as masked channels are entirely ignored. The slices are calculated in the same way as for encode so that slice assignment to states is exactly the same as during encode. Of course, the slices for a given state must be decoded in reverse order to which they were encoded and they symbols within a slice must also be decoded in reverse order. When batching several latents into the same set of states, the latents themselves must be decode in reverse order of encode. Further, optionally, in either parallel or serial rANS it is also possible to simply skip symbols on an element-wise basis based an the symbol’s corresponding ^^ value. We can determine from the ^^ value ’how likely the models think’ the symbol is the be zero and apply a threshold accordingly. The subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. The subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory program carrier for execution by, or to control the operation of, data processing apparatus. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. The computer storage medium is not, however, a propagated signal. The term “data processing apparatus” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A computer program (which may also be referred to or described as a program, software, a software application, a module, a software module, a script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network. The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Computers suitable for the execution of a computer program include, by way of example, can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a VR headset, a game console, a Global Positioning System (GPS) receiver, a server, a mobile phones, a tablet computer, a notebook computer, a music player, an e-book reader, a laptop or desktop computer, a PDAs, a smart phone, or other stationary or portable devices, that includes one or more processors and computer readable media, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. The subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), e.g., the Internet. The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. While this specification contains many specific implementation details, these should be construed as descriptions of features that may be specific to particular examples of particular inventions. Certain features that are described in this specification in the context of separate examples can also be implemented in combination in a single example. Conversely, various features that are described in the context of a single example can also be implemented in multiple examples separately or in any suitable subcombination. Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the examples described above should not be understood as requiring such separation in all examples, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

Claims

CLAIMS 1. A method for lossy video encoding, transmission and decoding, the method comprising the steps of: receiving an input frame and a previous frame at a first computer system; encoding the input frame and an input based on the previous frame using a first trained neural network to produce a first latent representation; encoding the input frame using a second trained neural network to produce a second latent representation; transmitting the first and the second latent representation to a second computer system; decoding the first latent representation using a third trained neural network to obtain an output flow map between the input frame and the previous frame; and decoding the second latent representation and an input based on the output flow map using a fourth trained neural network to produce an output frame, wherein the output frame is an approximation of the input frame.

2. The method of claim 1, wherein the output flow map is used to obtain an output warped frame from a previously decoded frame corresponding to the previous frame and the input based on the output flow map is based on the output warped frame.

3. The method of claim 2, wherein the output warped frame is transformed to obtain the input based on the output warped frame, wherein the transformation is at least one of the following: down-sampling, a convolution, a space-to-depth operation, a decomposition, a haar transform and a neural network.

4. The method of claim 1 to 3, further comprising the step of, at the first computer system, decoding the first latent representation using a fifth trained neural network to obtain a predicted flow map between the input frame and the previous frame; wherein the second trained neural network additionally encodes an input based on the predicted flow map to produce the second latent representation.

5. The method of claim 4, wherein the predicted flow map is used to obtain an input warped frame from the previously decoded frame corresponding to the previous frame and the input based on the predicted flow map is based on the input warped frame.

6. The method of any one of claims 1 to 5, wherein the input based on the previous frame comprises a flow map between the input frame and the previous frame.

7. The method of any one of claims 1 to 6, wherein the flow map between the input frame and the previous frame is used to obtain a first warped frame from the previously decoded framecorresponding to the previous frame; and the input based on the previous frame comprises the first warped frame.

8. The method of any one of claims 1 to 7, wherein the previously decoded frame corresponding to the previous frame is additionally encoded by the first trained neural network to produce the first latent representation.

9. The method of any one of claims 1 to 8, wherein a further previous frame is received at the first computer system; and a further previously decoded frame corresponding to the further previous frame is additionally encoded by the first trained neural network to produce the first latent representation.

10. The method of claim 9, wherein a predicted flow map between the further previous frame and the previous frame is additionally encoded by the first trained neural network to produce the first latent representation.

11. The method of claims 9 or 10, wherein at least two of the previously decoded frame corresponding to the previous frame, further previously decoded frame corresponding to the further previous frame and the predicted flow map between the further previous frame and the previous frame are concatenated before being encoded by the first trained neural network.

12. The method of any one of claims 8 to 11, wherein the previously decoded frame corresponding to the previous frame is additionally decoded by the third trained neural network to produce the output flow map.

13. The method of any one of claims 9 to 12, wherein the further previously decoded frame corresponding to the further previous frame is additionally decoded by the third trained neural network to produce the output flow map.

14. The method of any one of claims 9 to 13, wherein the predicted flow map between the further previous frame and the previous frame is additionally decoded by the third trained neural network to produce the output flow map.

15. The method of claims 13 or 14, wherein at least two of the previously decoded frame corresponding to the previous frame, further previously decoded frame corresponding to the further previous frame and the predicted flow map between the further previous frame and the previous frame are concatenated before being decoded by the third trained neural network.

16. The method of any one of claims 12 to 15, wherein at least one of the previously decoded frame corresponding to the previous frame, further previously decoded frame corresponding to the further previous frame and the predicted flow map between the further previous frame and the previous frame are transformed before being decoded by the third neural network,wherein the transformation is at least one of the following: down-sampling, a convolution, a space-to-depth operation, a decomposition, a haar transform and a neural network.

17. The method of claim 16, wherein a separate transformation is used for each of the at least one of the previously decoded frame corresponding to the previous frame, further previously decoded frame corresponding to the further previous frame and the predicted flowmap between the further previous frame and the previous frame.

18. The method of any one of claims 1 to 17, wherein a future frame is received at the first computer system; and an input based on the future frame is additionally encoded by the first trained neural network to produce the first latent representation.

19. The method of claim 18, wherein the input based on the future frame comprises a flow map between the input frame and the future frame.

20. The method of claim 18 or claim 19, wherein the flow map between the input frame and the future frame is used to obtain a second warped frame from a previously decoded future frame corresponding to the future frame; and the input based on the future frame comprises the second warped frame.

21. The method of any one of claims 18 to 20, wherein the previously decoded future frame corresponding to the future frame is additionally encoded by the first trained neural network to produce the first latent representation.

22. The method of any one of claims 18 to 21, wherein the previously decoded future frame corresponding to the future frame is additionally decoded by the third trained neural network to produce the output flow map.

23. The method of any one of claims 18 to 22, wherein a further future frame is received at the first computer system; and a further previously decoded future frame corresponding to the further future frame is additionally encoded by the first trained neural network to produce the first latent representation.

24. The method of claim 23, wherein a predicted flow map between the further future frame and the future frame is additionally encoded by the first trained neural network to produce the first latent representation.

25. The method of claims 23 or 24, wherein at least two of the previously decoded future frame corresponding to the future frame, further previously decoded future frame corresponding to the further future frame and the predicted flow map between the further future frame and the future frame are concatenated before being encoded by the first trained neural network.

26. The method of any one of claims 22 to 25, wherein the previously decoded future frame corresponding to the future frame is additionally decoded by the third trained neural network to produce the output flow map.

27. The method of any one of claims 23 to 26, wherein the further previously decoded future frame corresponding to the further future frame is additionally decoded by the third trained neural network to produce the output flow map.

28. The method of any one of claims 23 to 27, wherein the predicted flow map between the further future frame and the future frame is additionally decoded by the third trained neural network to produce the output flow map.

29. The method of claims 27 or 28, wherein at least two of the previously decoded future frame corresponding to the future frame, further previously decoded future frame corresponding to the further future frame and the predicted flow map between the further future frame and the future frame are concatenated before being decoded by the third trained neural network.

30. The method of any one of claims 27 to 29, wherein at least one of the previously decoded future frame corresponding to the future frame, further previously decoded future frame corresponding to the further future frame and the predicted flow map between the further future frame and the future frame are transformed before being decoded by the third neural network, wherein the transformation is at least one of the following: down-sampling, a convolution, a space-to-depth operation, a decomposition, a haar transform and a neural network.

31. The method of claim 30, wherein a separate transformation is used for each of the at least one of the previously decoded future frame corresponding to the future frame, further previously decoded future frame corresponding to the further future frame and the predicted flow map between the further future frame and the future frame.

32. The method of any one of claims 1 to 31, further comprising the steps of: encoding the second latent representation using a sixth trained neural network to produce a hyper-latent representation; additionally transmitting the hyper-latent representation to the second computer system; and decoding the hyper-latent representation using a seventh trained neural network to obtain an output latent representation wherein the output latent representation is used to obtain the output frame; and an input based on the output flow map is additionally encoded by the sixth trained neural network to produce the hyper-latent representation.

33. The method of claim 32, wherein the output flow map is used to obtain an output warped frame from a previously decoded frame corresponding to the previous frame and the input based on the output flow map additionally encoded by the sixth trained neural network is based on the output warped frame.

34. The method of claim 33, wherein the output warped frame is transformed to obtain the input based on the output flow map additionally encoded by the sixth trained neural network, wherein the transformation is at least one of the following: down-sampling, a convolution, a space-to-depth operation, a decomposition, a haar transform and a neural network.

35. The method of claim 34, wherein the transformation is applied a plurality of times.

36. The method of any one of claims 32 to 35, wherein the input based on the output flow map is additionally decoded by the seventh trained neural network to produce the output latent representation.

37. The method of claim 36, wherein the output flow map is used to obtain an output warped frame from a previously decoded frame corresponding to the previous frame and the input based on the output flow map additionally decoded by the seventh trained neural network is based on the output warped frame.

38. The method of claim 37, wherein the output warped frame is transformed to obtain the input based on the output flow map additionally decoded by the seventh trained neural network, wherein the transformation is at least one of the following: down-sampling, a convolution, a space-to-depth operation, a decomposition, a haar transform and a neural network.

39. The method of claim 38, wherein the transformation is applied a plurality of times.

40. The method of any one of claims 32 to 39, further comprising the steps of: encoding the hyper-latent representation using an eighth trained neural network to produce a hyper-hyper-latent representation; additionally transmitting the hyper-hyper-latent representation to the second computer system; and decoding the hyper-hyper-latent representation using a ninth trained neural network to obtain an output hyper-latent representation wherein the output hyper-latent representation is used to obtain the output frame; and an input based on the output flow map is additionally encoded is additionally encoded by the eighth trained neural network to produce the hyper-latent representation.

41. The method of claim 40, wherein the output flow map is used to obtain an output warped frame from a previously decoded frame corresponding to the previous frame and the inputbased on the output flow map additionally encoded by the eighth trained neural network is based on the output warped frame.

42. The method of claim 41, wherein the output warped frame is transformed to obtain the input based on the output flow map additionally encoded by the eighth trained neural network, wherein the transformation is at least one of the following: down-sampling, a convolution, a space-to-depth operation, a decomposition, a haar transform and a neural network.

43. The method of claim 42, wherein the transformation is applied a plurality of times.

44. The method of any one of claims 40 to 43, wherein the input based on the output flow map is additionally decoded by the ninth trained neural network to produce the output latent representation.

45. The method of claim 44, wherein the output flow map is used to obtain an output warped frame from a previously decoded frame corresponding to the previous frame and the input based on the output flow map additionally decoded by the ninth trained neural network is based on the output warped frame.

46. The method of claim 45, wherein the output warped frame is transformed to obtain the input based on the output flow map additionally decoded by the ninth trained neural network, wherein the transformation is at least one of the following: down-sampling, a convolution, a space-to-depth operation, a decomposition, a haar transform and a neural network.

47. The method of claim 46, wherein the transformation is applied a plurality of times.

48. A method for lossy video encoding, transmission and decoding, the method comprising the steps of: receiving an input frame and a previous frame at a first computer system; encoding the input frame and an input based on the previous frame using a first trained neural network to produce a first latent representation; transmitting the first latent representation to a second computer system; decoding the first latent representation using a second trained neural network to obtain an output flow map between the input frame and the previous frame; and using the output flow map to obtain an output frame, wherein the output frame is an approximation of the input frame.

49. A method of training one or more neural networks, the one or more neural networks being for use in lossy video encoding, transmission and decoding, the method comprising the steps of: receiving an input frame and a previous frame at a first computer system; encoding the input frame and an input based on the previous frame using a first neuralnetwork to produce a first latent representation; encoding the input frame using a second neural network to produce a second latent representation; decoding the first latent representation using a third neural network to obtain an output flow map between the input frame and the previous frame; and decoding the second latent representation and an input based on the output flow map using a fourth neural network to produce an output frame, wherein the output frame is an approximation of the input frame; evaluating a function based on a difference between the output frame and the input frame; updating the parameters of at least one of the first, second, third and fourth neural network based on the evaluated difference; and repeating the above steps using a first set of input images to produce a first, second, third and fourth trained neural network.

50. The method of claim 49, wherein the function is additionally based on a difference between the output flow map and a flow map between the input frame and the previous frame.

51. The method of claim 49 or 50, wherein the output flow map is used to obtain an output warped frame from a previously decoded frame corresponding to the previous frame; and the function is additionally based on a difference between the output warped frame and the input frame.

52. The method of claim 51, wherein a first weighting is applied to the contribution of the difference between the output flow map and the latent flow map to the function and a second weighting is applied to the contribution of the difference between the output warped frame and the input frame.

53. The method of claim 52, wherein at least one of the first weighting and the second weighting is changed between at least one of the repeated steps.

54. The method of claim 53, wherein the second weighting is increased between at least one of the repeated steps.

55. A method for lossy video encoding and transmission, the method comprising the steps of: receiving an input frame and a previous frame at a first computer system; encoding the input frame and an input based on the previous frame using a first trained neural network to produce a first latent representation; encoding the input frame using a second trained neural network to produce a second latent representation; transmitting the first and the second latent representation.

56. A method for lossy video receipt and decoding, the method comprising the steps of: receiving the first and the second latent representation transmitted according to the method of claim 55 at a second computer system; decoding the first latent representation using a third trained neural network to obtain an output flow map between the input frame and the previous frame; and decoding the second latent representation and an input based on the output flow map using a fourth trained neural network to produce an output frame, wherein the output frame is an approximation of the input frame.

57. A data processing system configured to perform the method of any one of claims 1 to 54.

58. A data processing apparatus configured to perform the method of claims 55 or 56.

59. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of claims 55 or 56.

60. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of claims 55 or 56.

61. A method for lossy video encoding, transmission and decoding, the method comprising the steps of: receiving a first frame, a second frame and a third frame at a first computer system; determining a first flow between the first frame and the second frame; determining a second flow between the second frame and the third frame; encoding the first flow and the second flow using a first trained neural network to produce a latent representation; transmitting the latent representation to a second computer system; decoding the latent representation using a second trained neural network to produce a first output flow and a second output flow; and using the first output flow and the second output flow to obtain an output frame, wherein the output frame is an approximation of the second frame.

62. The method of claim 61, further comprising the steps of: encoding the second frame using a third trained neural network to produce a latent frame representation; additionally transmitting the latent frame representation to the second computer system; and decoding the latent frame representation using a fourth trained neural network to produce the output frame.

63. The method of claim 62, wherein the first flow is used to warp a previously decoded first frame corresponding to the first frame to obtain a first warped frame; the second flow is used to warp a previously decoded third frame corresponding to the third frame to obtain a third warped frame; and the first warped frame and the third warped frame are additionally encoded by the third trained neural network to produce the latent frame representation.

64. The method of claim 63, wherein the first warped frame and the third warped frame are additionally decoded by the fourth trained neural network to produce the output frame.

65. The method of any one of claims 61 to 64, further comprising the step of obtaining a combined flow; wherein the combined flow is based on the first output flow and the second output flow; and the combined flow is used to obtain the output frame.

66. The method of claim 64 or claim 65, wherein a first weighting is applied to the first output flow and a second weighting is applied to the second output flow to obtain the combined flow.

67. The method of any one of claims 63 to claim 66, further comprising the step of obtaining a combined warped frame; wherein the combined warped frame is based on the first warped frame and the third warped frame; and the combined warped frame is used to obtain the output frame.

68. The method of claim 67, wherein a first weighting is applied to the first warped frame and a second weighting is applied to the third warped frame to obtain the combined warped frame.

69. The method of any one of claims 61 to 68, further comprising the step of: determining a third flow between the first frame and the third frame; wherein an input based on the third flow is additionally encoded by the first trained neural network to produce the latent representation.

70. The method of any one of claims 62 to 69, wherein the input based on the third flow is additionally encoded by the third trained neural network to obtain the latent frame representation.

71. The method of any one of claims 62 to 70, wherein the input based on the third flow is additionally decoded by the fourth trained neural network to produce the output frame.

72. The method of any one of claims 59 to 71, wherein the input based on third flow is a third output flow corresponding to the third flow, the third output flow being previously decoded by the second trained neural network.

73. A method of training one or more neural networks, the one or more neural networks being for use in lossy video encoding, transmission and decoding, the method comprising the steps of: receiving a first frame, a second frame and a third frame at a first computer system; determining a first flow between the first frame and the second frame; determining a second flow between the second frame and the third frame; encoding the first flow and the second flow using a first trained neural network to produce a latent representation; transmitting the latent representation to a second computer system; decoding the latent representation using a second trained neural network to produce a first output flow and a second output flow; and using the first output flow and the second output flow to obtain an output frame, wherein the output frame is an approximation of the second frame; evaluating a function based on a difference between the output frame and the second frame; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps using a first set of input frames to produce a first trained neural network and a second trained neural network.

74. The method of claim 73, further comprising the step of obtaining a combined flow; wherein the combined flow is based on the first output flow and the second output flow; a first weighting is applied to the first output flow and a second weighting is applied to the second output flow to obtain the combined flow; and the combined flow is used to obtain the output frame; wherein the first weighting and the second weighting are additionally updated based on the evaluated difference.

75. A method for lossy video encoding and transmission, the method comprising the steps of: receiving a first frame, a second frame and a third frame at a first computer system; determining a first flow between the first frame and the second frame; determining a second flow between the second frame and the third frame; encoding the first flow and the second flow using a first trained neural network to produce a latent representation; and transmitting the latent representation.

76. A method for lossy video receipt and decoding, the method comprising the steps of: receiving the latent representation transmitted according to the method of claim 75 at a second computer system; decoding the latent representation using a second trained neural network to produce a first output flow and a second output flow; and using the first output flow and the second output flow to obtain an output frame, wherein the output frame is an approximation of the second frame.

77. A data processing system configured to perform the method of any one of claims 61 to 74.

78. A data processing apparatus configured to perform the method of claims 75 or 76.

79. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of claims 75 or 76.

80. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of claims 75 or 76.

81. A method for lossy video encoding, transmission and decoding, the method comprising the steps of: receiving a first frame and a second frame at a first computer system; determining a first flow between the first frame and the second frame; determining a second flow based on the first frame and the second frame; encoding an input based on the first flow and the second flow using a first trained neural network to produce a latent representation; transmitting the latent representation to a second computer system; decoding the latent representation using a second trained neural network to produce an output flow; and using the output flow to obtain an output frame, wherein the output frame is an approximation of the second frame.

82. The method of claim 81, further comprising the step of: transforming the first frame using the first flow to obtain a warped frame; wherein the second flow is determined between the warped frame and the second frame.

83. The method of claim 82, wherein the input based on the first flow and the second flow is the second flow.

84. The method of claim 82 or 83, wherein the step of obtaining the output frame comprises applying the output flow to a frame corresponding to the first frame to obtain an intermediate frame.

85. The method of claim 84 wherein the step of obtaining the output frame comprises applying the first flow to the intermediate frame to obtain the output frame.

86. The method of claim 81, wherein the second flow is a point-wise flow; and the input based on the first flow and the second flow is the first flow subtracted from the second flow.

87. The method of claim 86, wherein the step of obtaining the output frame comprises adding the first flow to the output flow to obtain a reconstructed flow.

88. The method of claim 87, wherein the step of obtaining the output frame comprises applying the reconstructed flow to a frame corresponding to the first frame to obtain the output frame.

89. The method of claim 82 or 83, wherein the input based on the first flow and the second flow is the first flow and the second flow.

90. The method of claim 89, wherein the step of obtaining the output frame comprises applying the output flow to a frame corresponding to the first frame to obtain the output frame.

91. The method of any one of claims 81 to 90, further comprising the step of dividing each of the first frame and the second frame into a plurality of corresponding sub-frames; and a first flow is determined for each of corresponding sub-frames.

92. The method of any one of claims 81 to 91, wherein the first flow is a linear function.

93. The method of claim 92, wherein the first flow is determined by one of the following methods: a neural network that receives the first frame and the second frame as input and outputs one or more parameters of a matrix defining the first flow, a linear-least squares fit, a Fourier-Mellin transformation and an optimisation that minimizes the photometric error between the first frame and the second frame.

94. The method of any one of claims 81 to 91, wherein the first flow is a non-linear function.

95. The method of claim 94, wherein the first flow comprises a frame transformation.

96. The method of claim 94 or 95, wherein the non-linear function is a trained neural network.

97. The method of any one of claims 81 to 96, wherein the first flow is a global flow.

98. The method of any one of claims 81 to 97, wherein the second flow is a local flow.

99. A method of training one or more neural networks, the one or more neural networks being for use in lossy video encoding, transmission and decoding, the method comprising the steps of:receiving a first frame and a second frame at a first computer system; determining a first flow between the first frame and the second frame; determining a second flow based on the first frame and the second frame; encoding an input based on the first flow and the second flow using a first neural network to produce a latent representation; transmitting the latent representation to a second computer system; decoding the latent representation using a second neural network to produce an output flow; using the output flow to obtain an output frame, wherein the output frame is an approximation of the second frame; evaluating a function based on a difference between the output frame and the second frame; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps using a first set of input frames to produce a first trained neural network and a second trained neural network.

100. A method for lossy video encoding and transmission, the method comprising the steps of: receiving a first frame and a second frame at a first computer system; determining a first flow between the first frame and the second frame; determining a second flow based on the first frame and the second frame; encoding an input based on the first flow and the second flow using a first trained neural network to produce a latent representation; transmitting the latent representation.

101. A method for lossy video receipt and decoding, the method comprising the steps of: receiving the latent representation transmitted according to the method of claim 100 at a second computer system; decoding the latent representation using a second trained neural network to produce an output flow; and using the output flow to obtain an output frame, wherein the output frame is an approximation of the second frame.

102. A data processing system configured to perform the method of any one of claims 81 to 99.

103. A data processing apparatus configured to perform the method of claims 100 or 101.

104. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of claims 100 or 101.

105. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of claims 100 or 101.

106. A method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image or video at a first computer system; converting an input based on the input image or video using a first trained neural network to produce a plurality of parameter modifications; encoding the input image or video using a second trained neural network to produce a latent representation; transmitting the parameter modifications and the latent representation to a second computer system; modifying at least one of the parameters of a third trained neural network using the plurality of parameter modifications and decoding the latent representation using the third trained neural network to produce an output image or video, wherein the output image or video is an approximation of the input image or video.

107. The method of claim 106, further comprising the step of modifying at least one of the parameters of the second trained neural network using the plurality of parameter modifications prior to encoding the input image or video.

108. The method of claim 107, wherein the plurality of parameter modifications comprises pa- rameter modifications corresponding to the second neural network and parameter modifications corresponding to the third neural network.

109. The method of any one of claims 106 to 108, wherein the input based on the input image or video converted by the first trained neural network is the output of a fourth trained neural network that receives the input image or video as an input.

110. The method of claim 109, wherein the fourth trained neural network comprises a global average pooling layer.

111. The method of any one of claims 109 to 110, wherein the output of the fourth trained neural network is downsampled with respect to the input.

112. The method of any one of claims 106 to 111, wherein the first trained neural network comprises at least one of the following: a dense linear transformation, a sparse linear transformation, a low-rank linear transformation and a multi-layer perceptron network.

113. The method of any one of claims 106 to 112, wherein the output of the first trained neural network has a larger number of elements than the input.

114. The method of any one of claims 106 to 113, wherein the modification of at least one of the parameters of the third trained neural network using the plurality of parameter modifications comprises a perturbation of the at least one of the parameters using one of the plurality of parameter modifications.

115. The method of any one of claims 106 to 114, wherein the modification of at least one of the parameters of the third trained neural network using the plurality of parameter modifications comprises replacing at least one of the parameters with one of the plurality of parameter modifications.

116. The method of any one of claims 106 to 114, further comprising converting the input based on the input image or video using a further trained neural network to produce the plurality of parameter modifications; wherein the first trained neural network outputs a first subset of the plurality of parameter modifications and the further trained neural network outputs a second subset of the plurality of parameter modifications.

117. The method of any one of claims 106 to 116, wherein, when the input image or video is an input video, the input based on the input video is based on at least two frames of the input video.

118. The method of claim 117, wherein the fourth trained neural network comprises a convolution across the at least two frames of the input video.

119. The method of claim 117, wherein the input based on the input video is obtained by combining the at least two frames of the input video by at least one of the following: an averaging, a recurrent neural network, a convolutional neural network and a transformer network.

120. The method of any one of claims 109 to 116 when dependent on claim 109, wherein, when the input image or video is an input video, separate outputs of the fourth trained neural network are obtained for at least two frames of the input video, each output corresponding to a frame of the at least two frames of the input video; and the input based on the input video is based on the separate outputs of the fourth trained neural network.

121. The method of claim 119, wherein the input based on the input video is obtained by combining the separate outputs of the fourth trained neural network by at least one of the following: an averaging, a recurrent neural network, a convolutional neural network and a transformer network.

122. The method of any one of claims 106 to 115, wherein, when the input image or video is an input video, separate pluralities of parameter modifications are obtained for at least two frames of the input video, each separate plurality of parameter modifications corresponding to a frame of the at least two frames of the input video; and the separate pluralities of parameter modifications are combined before being transmitted to the second computer system; and the combined parameter modifications are the plurality of parameter modification used to modify the third neural network prior to decoding the at least two frames of the input video.

123. The method of claim 122, wherein the separate pluralities of parameter modifications are combined by at least one of the following: an averaging, a recurrent neural network, a convolutional neural network and a transformer network.

124. The method of any one of claims 106 to 123, further comprising the steps of: at the first computer system, entropy encoding the parameter modifications and the latent representation to obtain a bitstream; and at the second computer system, entropy decoding the bitstream to retrieve the parameter modifications and the latent representation; wherein the parameter modifications are entropy encoded and decoded using a first probability distribution and the latent representation is entropy encoded and decoded using a second probability distribution; and the first probability distribution is different to the second distribution.

125. The method of claims 124, wherein the first probability distribution is a Laplace distribution or a spike and slab distribution.

126. The method of claim 124 or 125, further comprising the steps of: encoding the plurality of parameter modifications using a fifth trained neural network to produce a hyper-latent representation; transmitting the hyper-latent representation to the second computer system; and decoding the hyper-latent representation using a sixth trained neural network and using the output of the sixth trained neural network during the entropy decoding to retrieve the parameter modifications.

127. The method of claim 124, wherein the first probability distribution comprises a plurality of probability distributions; and each of the plurality of probability distributions corresponds to a group of parameter modifications corresponding to a group of parameters of the third trained neural network.

128. A method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprisingthe steps of: receiving an input image or video at a first computer system; converting an input based on the input image or video using a first neural network to produce a plurality of parameter modifications; encoding the input image or video using a second neural network to produce a latent representation; transmitting the parameter modifications and the latent representation to a second computer system; modifying at least one of the parameters of a third neural network using the plurality of parameter modifications and decoding the latent representation using the third neural network to produce an output image or video, wherein the output image or video is an approximation of the input image or video; evaluating a function based on a difference between the output image or video and the input image or video; updating the parameters of the first neural network based on the evaluated function; and repeating the above steps using a first set of input images or videos to produce a first trained neural network.

129. The method of claim 128, wherein the function is additionally based on the rate of the plurality of parameter modifications.

130. The method of claims 128 or 129, further comprising the steps of: at the first computer system, entropy encoding the parameter modifications and the latent representation to obtain a bitstream; and at the second computer system, entropy decoding the bitstream to retrieve the parameter modifications and the latent representation; wherein the parameter modifications are entropy encoded and decoded using a first probability distribution; and the parameters of the first probability distribution are additionally updated based on the evaluated function to produce learned parameters.

131. The method of any one of claims 128 to 130, wherein the second neural network and the third neural network are trained neural networks.

132. The method of any one of claims 128 to 130, wherein the parameters of the second and third neural network are additionally updated based on the evaluated functions to produce a second trained neural network and a third trained neural network.

133. A method for lossy image or video encoding and transmission, the method comprising the steps of: receiving an input image or video at a first computer system;converting an input based on the input image or video using a first trained neural network to produce a plurality of parameter modifications; encoding the input image or video using a second trained neural network to produce a latent representation; transmitting the parameter modifications and the latent representation.

134. A method for lossy image or video receipt and decoding, the method comprising the steps of: receiving the parameter modifications and the latent representation transmitted according to the method of claim 133 at a second computer system; and modifying at least one of the parameters of a third trained neural network using the plurality of parameter modifications and decoding the latent representation using the third trained neural network to produce an output image or video, wherein the output image or video is an approximation of the input image or video.

135. A data processing system configured to perform the method of any one of claims 106 to 132.

136. A data processing apparatus configured to perform the method of claims 133 or 134.

137. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of claims 133 or 134.

138. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of claims 133 or 134.

139. A method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first neural network to produce a latent representation; decoding the latent representation using a second neural network to produce an output image, wherein the output image is an approximation of the input image; evaluating a function based on a difference between the output image and the input image; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps using a first set of input images to produce a first trained neural network and a second trained neural network; wherein the difference between the output image and the input image is determined based on the output of a neural network acting as a discriminator;the parameters of the neural network acting as a discriminator are additionally updated based on the evaluated function; and the parameters of the neural network acting as a discriminator are updated at a first learning rate; wherein, after at least one of the updates of the parameters of the neural network acting as a discriminator, the first learning rate is updated; and the update to the first learning rate is based on an error of the output of the neural network acting as a discriminator.

140. The method of claim 139, wherein the error is based on a difference between the the output of the neural network acting as a discriminator and a target value.

141. The method of claim 140, wherein the target value is predetermined.

142. The method of claim 140, wherein the target value is modified in at least one of the repeats of the training steps.

143. The method of any one of claims 139 to 142, wherein the update to the first learning rate is based on a calculation comprising the error.

144. The method of any one of claims 139 to 143, wherein the update to the first learning rate is based on a calculation comprising the integral of the error.

145. The method of any one of claims 139 to 144, wherein the update to the first learning rate is based on a calculation comprising the derivative of the error.

146. The method of any one of claims 141 to 145, wherein the contribution of at least one of the error, the integral of the error and the derivative of the error is scaled by a predetermined value.

147. The method of any one of claims 139 to 146, wherein the update to the first learning rate is additionally based on an update to the first learning rate in at least one previous repeat of the training steps.

148. The method of claim 147, wherein the update to the first learning rate comprises the step of: determining a current modification of the first learning rate based on an error of the output of the neural network acting as a discriminator; and the update to the first learning rate is based on an average of the current modification of the first learning rate and the update to the first learning rate in at least one previous repeat of the training steps.

149. The method of claim 148, wherein the average is an exponential moving average.

150. The method of any one of claims 139 to 147, wherein the update to the first learning rate is based on the output of a gradient descent algorithm which receives the error of the output of the neural network acting as a discriminator as an input.

151. The method of claim 150, wherein the gradient descent algorithm is one of the following algorithms: SGD, Adagrand, RMSProp or Adam.

152. A method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first neural network to produce a latent representation; decoding the latent representation using a second neural network to produce an output image, wherein the output image is an approximation of the input image; evaluating a function based on a difference between the output image and the input image and the rate of the latent representation; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps using a first set of input images to produce a first trained neural network and a second trained neural network; wherein at the contribution of at least one of the difference between the output image and the input image and the rate of the latent representation to the function is scaled by a first parameter; after at least one of the updates of the parameters of the the first neural network and the second neural network, the first parameter is updated; and the update to the first parameter is based on an error of the at least one of the difference between the output image and the input image and the rate of the latent representation.

153. A method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first trained neural network to produce a latent representation; transmitting the latent representation to a second computer system; decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image; wherein at least one of the first trained neural network and the second trained neural network has been trained according to the method of any one of claims 139 to 152.

154. A method for lossy image or video encoding and transmission, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first trained neural network to produce a latent representation; and transmitting the latent representation; wherein the first trained neural network has been trained according to the method of any one of claims 139 to 152.

155. A method for lossy image or video receipt and decoding, the method comprising the steps of: receiving the latent representation transmitted according to the method of claim 152 at a second computer system; and decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image; wherein the second trained neural network has been trained according to the method of any one of claims 139 to 152.

156. A data processing system configured to perform the method of any one of claims 139 to 153.

157. A data processing apparatus configured to perform the method of claims 154 or 155.

158. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of claims 154 or 155.

159. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of claims 154 or 155.

160. A method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first trained neural network to produce a latent representation, wherein the latent representation has a probability distribution described by at least one probability parameter; dividing the latent representation into a plurality of sub-latent representations, wherein each sub-latent representation has a sub-probability distribution described by at least one sub-probability parameter; entropy encoding the plurality of sub-latent representations using the plurality of at least one sub-probability parameters to produce a bitstream; transmitting the bitstream to a second computer system;entropy decoding the bitstream using the plurality of at least one sub-probability parameters to retrieve the plurality of sub-latent representations and combining the plurality of sub-latent representations to retrieve the latent representation; and decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image.

161. The method of claim 160, wherein at least one of the plurality of sub-latent representations has a different number of elements to another of the plurality of sub-latent representations.

162. The method of claim 160 or 161, wherein the number of elements in each of the plurality of sub-latent representations is selected such that the entropy encoding of each of the plurality of sub-latent representations has equivalent computational complexity.

163. The method of any one of claims 160 to 162, further comprising the step of: after dividing the latent representation into a plurality of sub-latent representations, adding a tag to each sub-latent representation, the tag indicating the position of the sub-latent representation within the latent representation; and using the tag when combining the plurality of sub-latent representations to retrieve the latent representation.

164. The method of any one of claims 160 to 163, wherein the bitstream comprises a plurality of sub-bitstreams, each sub-bitstream corresponding to one of the plurality of sub-latent representations, and the method further comprises the step of: adding a tag to each sub-bitstream, the tag indicating the start of each sub-bitstream; wherein, after receipt of the bitstream at the second computer system, the bitstream is divided to retrieve each sub-bitstream using the tag indicating the start of each sub-bitstream.

165. The method of claim any one of claims 160 to 164, wherein at least two of the plurality of sub-latent representations are encoded in parallel.

166. The method of claim any one of claims 160 to 165, wherein at least two of the plurality of sub-latent representations are retrieved by the decoding of the bitstream in parallel.

167. The method of any one of claims 160 to 166, wherein the step of entropy encoding the plurality of sub-latent representations using the plurality of at least one sub-probability parameters to produce a bitstream comprises the steps of: entropy encoding each of the plurality of sub-latent representations to obtain a plurality of states and a plurality of sub-bitstreams; entropy encoding the plurality of states; and combining the plurality of sub-bitstreams and the entropy encoded plurality of states to obtain the bitstream.

168. The method of claim 160 to 167, further comprising the steps of: encoding the latent representation using a third trained neural network to produce a hyper-latent representation; entropy encoding the hyper-latent representation and including the entropy encoded hyper-latent representation in the bitstream; at the second computer system, entropy decoding the entropy encoded hyper-latent representation to retrieve the hyper-latent representation; and decoding the hyper-latent representation using a fourth trained neural network, wherein the output of the fourth trained neural network is used to obtain the output image.

169. The method of claim 168, wherein the step of entropy encoding the hyper-latent representation comprises dividing the hyper-latent representation into a plurality of sub-hyper- latent representations, wherein each sub-hyper-latent representation has a sub-probability distribution described by at least one sub-probability parameter; and entropy encoding the plurality of sub-hyper-latent representations using the plurality of at least one sub-probability parameters; and the step of entropy decoding the entropy encoded hyper-latent representation comprises entropy decoding the the entropy encoded hyper-latent representation using the plurality of at least one sub-probability parameters to retrieve the plurality of sub-hyper-latent representations and combining the plurality of sub-hyper-latent representations to retrieve the hyper-latent representation.

170. The method of any one of claims 160 to 169, wherein the step of entropy encoding the plurality of sub-latent representations using the plurality of at least one sub-probability parameters to produce a bitstream comprises the steps of: entropy encoding each of the plurality of sub-latent representations to obtain a plurality of states and a plurality of sub-bitstreams; entropy decoding the plurality of states to obtain a state latent; entropy encoding the state latent to obtain a state bitstream; and combining the plurality of sub-bitstreams and the state bitstream to obtain the bitstream.

171. The method of claim 170, wherein a lossy or lossless compression process is performed on the state latent prior to the entropy encoding of the state latent to obtain the state bitstream.

172. The method of any one of claims 169 to 171, wherein the step of entropy encoding the plurality of sub-hyper-latent representations using the plurality of at least one sub-probability parameters comprises the steps of: entropy encoding each of the plurality of sub-hyper-latent representations to obtain a plurality of hyper-states and a plurality of sub-hyper-bitstreams; entropy decoding the plurality of hyper-states to obtain a hyper-state latent;entropy encoding the hyper-state latent to obtain a hyper-state bitstream; and combining the plurality of sub-bitstreams and the hyper-state bitstream to obtain the bitstream.

173. The method of claim 172, wherein a lossy or lossless compression process is performed on the hyper-state latent prior to the entropy encoding of the hyper-state latent to obtain the bitstream.

174. The method of claim 169, wherein the step of entropy encoding the plurality of sub-latent representations using the plurality of at least one sub-probability parameters to produce a bitstream comprises the steps of: entropy encoding each of the plurality of sub-latent representations to obtain a plurality of states and a plurality of sub-bitstreams; and entropy decoding the plurality of states to obtain a state latent; wherein the step of entropy encoding the plurality of sub-hyper-latent representations using the plurality of at least one sub-probability parameters comprises the steps of: entropy encoding each of the plurality of sub-hyper-latent representations to obtain a plurality of hyper-states and a plurality of sub-hyper-bitstreams; and entropy decoding the plurality of hyper-states to obtain a hyper-state latent; wherein the method further comprises the steps of: combining the state latent and the hyper-state latent to obtain a combined state latent; entropy encoding the combined state latent to obtain a combined state bitstream; and combining the plurality of sub-bitstreams and the combined state bitstream to obtain the bitstream.

175. The method of claim 174, wherein a lossy or lossless compression process is performed on the combined state latent prior to the entropy encoding of the combined state latent to obtain the bitstream.

176. The method of any one of claims 1 to 175, further comprising the steps of: receiving a further input image at the first computer system, wherein the input image and the further input image are frames of an input video; encoding the further input image using the first trained neural network to produce a further latent representation, wherein the further latent representation has a probability distribution described by at least one probability parameter; dividing the further latent representation into a plurality of further sub-latent repre- sentations, wherein each further sub-latent representation has a sub-probability distribution described by at least one sub-probability parameter; entropy encoding the plurality of further sub-latent representations using the plurality of at least one sub-probability parameters to produce a further bitstream;transmitting the further bitstream to a second computer system; entropy decoding the further bitstream using the plurality of at least one sub-probability parameters to retrieve the plurality of further sub-latent representations and combining the plurality of further sub-latent representations to retrieve the further latent representation; and decoding the further latent representation using the second trained neural network to produce a further output image, wherein the further output image is an approximation of the further input image.

177. The method of claim 176 when dependent on claim 170, wherein the step of entropy encoding the further plurality of sub-latent representations using the plurality of at least one sub-probability parameters to produce a further bitstream comprises the steps of: entropy encoding each of the plurality of further sub-latent representations to obtain a plurality of further states and a plurality of further sub-bitstreams; entropy decoding the plurality of further states to obtain a further state latent; combining the state latent and the further state latent to obtain a combined state latent; entropy encoding the combined state latent to obtain a combined state bitstream; and combining the plurality of sub-bitstreams, the plurality of further sub-bitstreams and the combined state bitstream to obtain the bitstream.

178. A method for bitstream encoding, transmission and decoding, the method comprising the steps of: receiving an input at a first computer system, wherein the input has a probability distribution described by at least one probability parameter; dividing the input into a plurality of sub-inputs, wherein each sub-input has a sub-probability distribution described by at least one sub-probability parameter; entropy encoding the plurality of sub-inputs using the plurality of at least one sub-probability parameters to produce a bitstream; transmitting the bitstream to a second computer system; and entropy decoding the bitstream using the plurality of at least one sub-probability parameters to retrieve the plurality of sub-inputs and combining the plurality of sub-latent inputs to retrieve the input.

179. A method for lossy image or video encoding and transmission, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first trained neural network to produce a latent representation, wherein the latent representation has a probability distribution described by at least one probability parameter; dividing the latent representation into a plurality of sub-latent representations, whereineach sub-latent representation has a sub-probability distribution described by at least one sub-probability parameter; entropy encoding the plurality of sub-latent representations using the plurality of at least one sub-probability parameters to produce a bitstream; transmitting the bitstream.

180. A method for lossy image or video receipt and decoding, the method comprising the steps of: receiving the bitstream transmitted according to the method of claim 179 at a second computer system; entropy decoding the bitstream using the plurality of at least one sub-probability parameters to retrieve the plurality of sub-latent representations and combining the plurality of sub-latent representations to retrieve the latent representation; and decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image.

181. A data processing system configured to perform the method of any one of claims 160 to 178.

182. A data processing apparatus configured to perform the method of claims 160 to 178.

183. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of claims 160 to 178.

184. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of claims 160 to 178.

185. A method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first trained neural network to produce a latent representation, wherein the latent representation has a probability distribution described by at least one probability parameter; generating a mask based on at least one of the latent representation or the at least one probability parameter; applying the mask to the latent representation to produce a masked latent representation; entropy encoding the masked latent representations using the at least one probability parameter to produce a bitstream; transmitting the bitstream to a second computer system; entropy decoding the bitstream using the at least one probability parameter and the maskto retrieve the latent representation; and decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image.

186. The method of claim 185, comprising: dividing the latent representation into a plurality of partitions, estimating at least one statistical value associated with the plurality of partitions, and using the at least one estimated statistical value to produce the mask.

187. The method of claim 186, wherein the statistical value comprises a mean.

188. The method of claim 186 or 187, wherein producing the mask comprises applying a threshold to the at least one statistical value to generate a mask value for at least one partition of the plurality of partitions.

189. The method of any of claims 185 to 188, wherein applying the mask comprises setting one or more elements of the latent representation to substantially zero.

190. The method of any of claims 185 to 189, comprising using the at least one probability parameter to estimate a value of at least one element of the latent representation, and using the estimated value to generate a mask value for the at least one element.

191. The method of any of claims 185 to 190, wherein entropy encoding the masked latent representation comprises skipping masked elements of the masked latent representation.

192. Themethod of any of claims 185 to 191, wherein entropy decoding the bitstream comprises skipping elements corresponding to masked elements of the masked latent representation.

193. The method of claim 192, wherein skipping elements comprises initialising a buffer with zeros and skipping a predetermined number of zeros based on a size of a masked section of the masked latent representation.

194. The method of claim 193, wherein skipping elements comprises inserting a predetermined number of zeros in a buffer based on a size of a masked section of the masked latent representation.

195. The method of any of claims 185 to 194, comprising retrieving the mask from the bitstream.

196. The method of any of claims 185 to 195, wherein entropy decoding comprises using the at least one probability parameter to estimate a value of at least one element of the latent representation, and using the estimated value to generate a mask value for the at least one element.

197. A method for lossy image or video encoding and transmission, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first trained neural network to produce a latent representation, wherein the latent representation has a probability distribution described by at least one probability parameter; generating a mask based on at least one of the latent representation or the at least one probability parameter; applying the mask to the latent representation to produce a masked latent representation; entropy encoding the masked latent representations using the at least one probability parameter to produce a bitstream; and transmitting the bitstream to a second computer system.

198. A method for lossy image or video receiving and decoding, the method comprising the steps of: receiving a bitstream a a second computer system, the bitstream produced by entropy encoding a masked latent representation associated with an input image using at least one entropy parameter; entropy decoding the bitstream using the at least one probability parameter and a mask to retrieve the latent representation; and decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image.

199. A data processing system configured to perform the method of any one of claims 185 to 198.

200. A data processing apparatus configured to perform the method of claims 185 to 198.

201. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of claims 185 to 198.

202. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of claims 185 to 198.

Images