Method and data processing system for lossy image or video compression

Abstract

A method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving first and second input images; estimating optical flow information indicative of a difference between representations of the first and second input images; receiving first and second output images; estimating optical flow information indicative of a difference between representations of the first output image and the second output image; updating the parameters of the one or more neural networks based on the evaluated function; and repeating the above steps using a first set of input images to produce one or more trained neural network.

Description

[0001]Method and data processing system for lossy image or video encoding, transmission and decoding BACKGROUND This invention relates to a method and system for lossy image or video encoding, transmission and decoding, a method, apparatus, computer program and computer readable storage medium for lossy image or video encoding and transmission, and a method, apparatus, computer program and computer readable storage medium for lossy image or video receipt and decoding. There is increasing demand from users of communications networks for images and video content. Demand is increasing not just for the number of images viewed, and for the playing time of video; demand is also increasing for higher resolution content. This places increasing demand on communications networks and increases their energy use because of the larger amount of data being transmitted. To reduce the impact of these issues, image and video content is compressed for transmission across the network. The compression of image and video content can be lossless or lossy compression. In lossless compression, the image or video is compressed such that all of the original information in the content can be recovered on decompression. However, when using lossless compression there is a limit to the reduction in data quantity that can be achieved. In lossy compression, some information is lost from the image or video during the compression process. Known compression techniques attempt to minimise the apparent loss of information by the removal of information that results in changes to the decompressed image or video that is not particularly noticeable to the human visual system. JPEG, JPEG2000, AVC, HEVC and AVI are examples of compression processes for image and / or video files. In general terms, known lossy image compression techniques use the spatial correlations between pixels in images to remove redundant information during compression. For example, in an image of a blue sky, if a given pixel is blue, there is a high likelihood that the neighbouring pixels, and their neighbouring pixels, and so on, are also blue. There is accordingly no need to retain all the raw pixel data. Instead, we can retain only a subset of the pixels which take up fewer bits and infer the pixel values of the other pixels using information derived from spatial correlations. A similar approach is applied in known lossy video compression techniques. That is, spatial correlations between pixels allow the removal of redundant information during compression. However, in video compression, there is further information redundancy in the form of temporal correlations. For example, in a video of an aircraft flying across a blue-sky background, most of the pixels of the blue sky do not change at all between frames of the video. The most of the blue sky pixel data for the frame at position t = 0 in the video is identical to that at position t = 10. Storing this identical, temporally correlated, information is inefficient. Instead, only the blue sky pixel data for a subset of the frames is stored and the rest are inferred from information derived from temporal correlations. In the realm of lossy video compression in particular, the removal of redundant temporally correlated information in a video sequence is known inter-frame redundancy. One technique using inter-frame redundancy that is widely used in standard video compression algorithms involves the categorization of video frames into three types: I-frames, P-frames, and B-frames. Each frame type carries distinct properties concerning their encoding and decoding process, playing different roles in achieving high compression ratios while maintaining acceptable visual quality. I-frames, or intra-coded frames, serve as the foundation of the video sequence. These frames are self-contained, each one encoding a complete image without reference to any other frame. In terms of compression, I-frames are least compressed among all frame types, thus carrying the most data. However, their independence provides several benefits, including being the starting point for decompression and enabling random access, crucial for functionalities like fast-forwarding or rewinding the video. P-frames, or predictive frames, utilize temporal redundancy in video sequences to achieve greater compression. Instead of encoding an entire image like an I-frame, a P-frame represents the difference between itself and the closest preceding I- or P-frame. The process, known as motion compensation, identifies and encodes only the changes that have occurred, thereby significantly reducing the amount of data transmitted. Nonetheless, P-frames are dependent on previous frames for decoding. Consequently, any error during the encoding or transmission process may propagate to subsequent frames, impacting the overall video quality. B-frames, or bidirectionally predictive frames, represent the highest level of compression. Unlike P-frames, B-frames use both the preceding and following frames as references in their encoding process. By predicting motion both forwards and backwards in time, B-frames encode only the differences that cannot be accurately anticipated from the previous and next frames, leading to substantial data reduction. Although this bidirectional prediction makes B-frames more complex to generate and decode, it does not propagate decoding errors since they are not used as references for other frames. Artificial intelligence (AI) based compression techniques achieve compression and decom- pression of images and videos through the use of trained neural networks in the compression and decompression process. Typically, during training of the neutral networks, the difference between the original image and video and the compressed and decompressed image and video is analyzed and the parameters of the neural networks are modified to reduce this difference while minimizing the data required to transmit the content. However, AI based compression methods may achieve poor compression results in terms of the appearance of the compressed image or video or the amount of information required to be transmitted. An example of an AI based image compression process comprising a hyper-network is described in Ballé, Johannes, et al. “Variational image compression with a scale hyperprior.” arXiv preprint arXiv:1802.01436 (2018), which is hereby incorporated by reference. An example of an AI based video compression approach is shown in Agustsson, E., Minnen, D., Johnston, N., Balle, J., Hwang, S. J., and Toderici, G. (2020), Scale-space flow for end-to-end optimized video compression. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 8503-8512), which is hereby incorporated by reference. A further example of an AI based video compression approach is shown in Mentzer, F., Agustsson, E., Ballé, J., Minnen, D., Johnston, N., and Toderici, G. (2022, November). Neural video compression using gans for detail synthesis and propagation. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVI (pp. 562-578), which is hereby incorporated by reference. Figure 3 of which shows an architecture that calculates optical flow with a flow model, UFlow, and encodes the calculated optical flow with a flow encoder, Eflow. SUMMARY According to an aspect of the invention, there is provided a method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first neural network to produce a latent representation; decoding the latent representation using a second neural network to produce an output image, wherein the output image is an approximation of the input image; evaluating a function based on a difference between the output image and the input image; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps using a first set of input images to produce a first trained neural network and a second trained neural network, wherein encoding the input image using the first neural network comprises at least one of clamping an output of at least one layer of the first neural network at or above a first value or clamping an output of at least one layer of the first neural network below a second value. Optionally, encoding the input image using the first neural network comprises clamping the output of the at least one layer of the first neural network between the first value and the second value. Optionally, decoding the latent representation using a second neural network comprises clamping at least one output of at least one layer of the second neural network below a first value and / or above the second value. Optionally, one or more activation layers of the first or second neural network are defined by a hardtanh activation function. Optionally, the hardtanh activation function has a minimum value corresponding to the first value, and wherein the hardtanh activation function has a maximum value corresponding to the second value. Optionally, one or more activation layers of the first or second neural network are defined by a ReLu activation function and wherein said clamping comprises modifying the at least one output of the ReLu activation function to be at or below the second value. Optionally, said encoding the input image using the first neural network comprises clamping each output of each layer of the first neural network below a first value and / or above a second value. Optionally, said decoding the latent representation using the second neural network comprises clamping each output of each layer of the second neural network below a first value and / or above a second value. Optionally, encoding the input image using the first neural network or decoding the latent representation with the second neural network comprises adding first noise to the output of at least one layer of the first or second neural network. Optionally, adding second noise, different to the first noise, to the output of at least one other layer of the first or second neural network. Optionally, encoding the input image using the first neural network or decoding the latent representation with the second neural network comprises adding noise to the output of each layer of the first and second neural network. Optionally, said encoding comprises applying a predetermined number of convolution layers and activation layers of the first neural network to the input image, and wherein said clamping is performed to the output of one or more of the activation layers. According to a further aspect, there is provided a method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first neural network to produce a latent representation; decoding the latent representation using a second neural network to produce an output image, wherein the output image is an approximation of the input image; evaluating a function based on a difference between the output image and the input image; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps using a first set of input images to produce a first trained neural network and a second trained neural network, wherein decoding the latent representation using a second neural network comprises at least one of clamping an output of at least one layer of the first neural network at or above a first value or clamping an output of at least one layer of the first neural network below a second value. Optionally, one or more activation layers of the second neural network are defined by a hardtanh activation function. Optionally, the hardtanh activation function has a minimum value corresponding to the first value, and wherein the hardtanh activation function has a maximum value corresponding to the second value. Optionally, said decoding comprises applying a predetermined number of convolution layers and activation layers of the second neural network to the latent representation, and wherein said clamping is performed to the output of one or more of the activation layers. Optionally, encoding the input image using the first neural network or decoding the latent representation with the second neural network comprises adding first noise to the output of at least one layer of the first or second neural network. Optionally, the method comprises adding second noise, different to the first noise, to the output of at least one other layer of the first or second neural network. According to a a further aspect, there is provided a method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first trained neural network to produce a latent representation; transmitting the latent representation to a second computer system; and decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image, wherein encoding and / or decoding the input image using the first neural network comprises clamping at least one output of at least one layer of the first and / or neural network below a first value and / or above a second value. Accoridng to a further aspect, there is provided a method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first neural network to produce a latent representation; decoding the latent representation using a second neural network to produce an output image, wherein the output image is an approximation of the input image; evaluating a function based on a difference between the output image and the input image; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps using a first set of input images to produce a first trained neural network and a second trained neural network, wherein encoding the input image using the first neural network or decoding the latent representation with the second neural network comprises adding first noise to the output of at least one layer of the first or second neural network. Optionally, the method comprises adding second noise, different to the first noise, to the output of at least one other layer of the first or second neural network. Optionally, said encoding comprises applying a predetermined number of convolution layers and activation layers of the first neural network to the input image, wherein the noise is added to the output of one or more of the activation layers. Optionally, said decoding comprises applying a predetermined number of convolution layers and activation layers of the second neural network to the latent representation, wherein the noise is added to the output of one or more of the activation layers. Optionally, encoding the input image using the first neural network or decoding the latent representation with the second neural network comprises adding noise to the output of each layer of the first or second neural network. Optionally, the first noise comprises Gaussian or Laplacian noise. Optionally, the first noise comprises uniform noise. Optionally, during said repeating of the steps, the first noise added in a first step is different to the first noise added in a second step. Optionally, the first noise and / or the second noise are based on a property of the output of the at least one layer from a previous step during said repeating of the steps. According to a further aspect, there is provided a data processing system configured to perform the above methods. According to a further aspect, there is provided a method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first neural network to produce a latent representation; transmitting the latent representation to a second computer system; decoding the latent representation using a second neural network to produce an output image, wherein the output image is an approximation of the input image, wherein said encoding and / or decoding comprises performing a quantisation operation on an output of one layer of the first or second neural network using one or more quantisation parameters to produce a quantized output of the one layer; and wherein the method comprises using the one or more quantisation parameters to perform a quantisation operation on an output of a different layer of the first or second neural network to produce a quantized output of the different layer. Optionally, the one or more quantisation parameters comprise a scale parameter and / or a location parameter. Optionally, the one or more quantisation parameters comprise a quantisation bin size value and / or a zero point value. Optionally, the quantisation operation comprises a rounding operation. Optionally, said encoding and / or decoding comprises processing the quantized output of the one layer by performing one or more of an addition, subtraction, multiplication, and / or concatenation operations using the quantized output of the one layer. Optionally, the one or more quantisation parameters used to perform the quantisation operation on the output of the different layer are based on the processing type performed to the quantized output of the one layer. Optionally, the method comprises storing the one or more quantisation parameters used to perform the quantisation operation on the output of the one layer in association with the processing type performed on said output. Optionally, the method comprises identifying a processing type to be performed on the output of the different layer, retrieving the stored one or more quantisation parameters based on the identified processing type, and performing said quantisation operation on the output of the different layer using the retrieved one or more quantisation parameters. Optionally, the method further comprises the steps of: performing a quantisation operation on the latent representation using one or more latent quantisation parameters; encoding the quantized latent representation using a third neural network to produce a hyper-latent representation; transmitting the hyper-latent representation to the second computer system; and decoding the quantized hyper-latent representation using a fourth trained neural network to obtain the latent representation, processing the quantized latent representation to obtain the one or more latent quantisation parameters; comparing the one or more latent quantisation parameters to the stored one or more quantisation parameters, retrieving the stored one or more quantisation parameters based on a result of said comparing; and performing said quantisation operation on the output of the different layer using the retrieved one or more quantisation parameters. Optionally, said storing is performed at the first computer system and at the second computer system before receiving, encoding, transmitting and decoding. Optionally, the quantized output of the one layer and / or the different layer comprises integer type data. Optionally, a result of performing said processing to said quantized output of the one layer and / or the different layer comprises integer type data. Optionally, the one or more quantisation parameters used to perform a quantisation operation on the output of the different are based on a position of the one layer in the first or second neural network. According to a further aspect, there is provided a method for lossy image or video encoding and transmission, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first neural network to produce a latent representation; and transmitting the latent representation, wherein said encoding comprises performing a quantisation operation on an output of one layer of the first neural network using one or more quantisation parameters to produce a quantized output of the one layer; and wherein the method comprises using the one or more quantisation parameters to perform a quantisation operation on an output of a different layer of the first neural network to produce a quantized output of the different layer. Optionally, said encoding comprises processing the quantized output of the one layer by performing one or more of an addition, subtraction, multiplication, and / or concatenation operations using the quantized output of the one layer. Optionally, the method comprises storing the one or more quantisation parameters used to perform the quantisation operation on the output of the one layer in association with the processing type performed on said output. Optionally, the method comprises identifying a processing type to be performed on the output of the different layer, retrieving the stored one or more quantisation parameters based on the identified processing type, and performing said quantisation operation on the output of the different layer using the retrieved one or more quantisation parameters. According to a further aspect there is provided a method for lossy image or video receipt and decoding, the method comprising the steps of: receiving a latent representation transmitted by a first computer system at a second computer system, the latent representation corresponding to an input image; and decoding the latent representation using a second neural network to produce an output image, wherein the output image is an approximation of the input image wherein said decoding comprises performing a quantisation operation on an output of one layer of the second neural network using one or more quantisation parameters to produce a quantized output of the one layer; and wherein the method comprises using the one or more quantisation parameters to perform a quantisation operation on an output of a different layer of the second neural network to produce a quantized output of the different layer. Optionally, said decoding comprises processing the quantized output of the one layer by performing one or more of an addition, subtraction, multiplication, and / or concatenation operations using the quantized output of the one layer. Optionally, the method comprises storing the one or more quantisation parameters used to perform the quantisation operation on the output of the one layer in association with the processing type performed on said output. Optionally, the method comprises identifying a processing type to be performed on the output of the different layer, retrieving the stored one or more quantisation parameters based on the identified processing type, and performing said quantisation operation on the output of the different layer using the retrieved one or more quantisation parameters. According to a further aspect there is provided a method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image and a previously decoded image at a first computer system; estimating a first block and a second block of optical flow information using a first neural network, the first block of optical flow information being indicative of a difference between a block of the previously decoded image and a block of the input image, and the second block of optical flow information being a sub-block of the first block; transmitting the optical flow information to a second computer system; decoding the optical flow information using a second neural network; and using the previously decoded image and the decoded optical flow information to produce an output image, wherein the output image is an approximation of the input image. Optionally, the method comprises performing a quantisation operation on the first and on second block using one or more quantisation parameters to produce quantized first and second blocks of flow information. Optionally, the one or more quantisation parameters used to perform the quantisation operation on the first block are different to the one or more quantisation parameters used to perform the quantisation operation on the second block. Optionally, the one or more quantisation parameters comprise a scale parameter and / or a location parameter. Optionally, the quantized first and second blocks of flow information comprise fixed point precision type data. Optionally, the integer type data comprises 4-bit, 8-bit or 16-bit integer type data. Optionally, using the previously decoded image and the decoded optical flow information comprises combining the quantized first block and second block of integer type optical flow information as floating point type data. Optionally, the method comprises warping the previously decoded image using the floating point type data. Optionally, using the previously decoded image and the decoded optical flow information comprises combining the quantized first block and second block of integer type optical flow information as 16-bit integer type data. Optionally, the method comprises warping the previously decoded image using the 16-bit integer type data. Optionally, said decoding the optical flow information is performed with a central processing unit (CPU), neural processing unit (NPU) or tensor processing unit (TPU), and wherein said warping is performed with a graphical processing unit (GPU). Optionally, the second block of optical flow information provides optical flow information at a quarter pixel resolution of the optical flow information of the first block. According to a further aspect, there is provided a data processing system configured to perform the above methods. Optionally, the data processing system of comprises at least one of a CPU, NPU or TPU, and a GPU. According to a further aspect there is provided a method for lossy image or video encoding and transmission, the method comprising the steps of: receiving an input image and a previously decoded image at a first computer system; estimating a first block and a second block of optical flow information using a first neural network, the first block of optical flow information being indicative of a difference between a block of the previously decoded image and a block of the input image, and the second block of optical flow information being a sub-block of the first block; transmitting the optical flow information to a second computer system. According to a further aspect there is provided a method for lossy image or video receipt and decoding, the method comprising the steps of: receiving a first block and a second block of optical flow information transmitted by a first computer system at a second computer system, the first block of optical flow information being indicative of a difference between a block of a previously decoded image and a block of an input image, and the second block of optical flow information being a sub-block of the first block; and decoding the first and second blocks of optical flow information using a second neural network; and using a previously decoded image and the decoded optical flow information to produce an output image, wherein the output image is an approximation of the input image. According to a further aspect there is provided a data processing apparatus configured to perform the above methods. According to a further aspect there is provided a method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image and a previously decoded image at a first computer system; estimating optical flow information using a first neural network, the optical flow information being indicative of a difference between the previously decoded image and the input image; producing a latent representation of the optical flow information; transmitting the latent representation of the optical flow information to a second computer system; decoding the latent representation of the optical flow information using a second neural network to produce first decoded optical flow information; decoding the latent representation of the optical flow information using a third neural network and the first decoded optical flow information to produce second decoded optical flow information; using the previously decoded image and the first and second decoded optical flow information to produce an output image, wherein the output image is an approximation of the input image. Optionally, the first decoded optical flow information comprises optical flow information associated with a first block of the latent representation, and wherein the second decoded optical flow information comprises optical flow information associated with a sub-block of the first block. Optionally, the method comprises splitting the latent representation into a plurality of blocks, producing the first decoded optical flow information with the second neural network using a first block of the plurality of blocks, and producing the second decoded optical flow information with the third neural network using: (i) the first decoded optical flow information and (ii) a sub-block of the first block. Optionally, the method comprises combining the first decoded optical flow information and the sub-block by concatenation before inputting the concatenated first decoded optical flow information and the sub-block of the first block into the third neural network. Optionally, the method comprises decoding the latent representation of the optical flow information using: a fourth neural network, the first decoded optical flow information, and the second decoded optical flow information to produce third decoded optical flow information. Optionally, the method comprises splitting the latent representation into a plurality of blocks, producing the first decoded optical flow information with the second neural network using a first block of the plurality of blocks, producing the second decoded optical flow information with the third neural network using the first decoded optical flow information and a sub-block of the first block, and producing the third optical flow information with the fourth neural network using: (i) the first decoded optical flow information, (ii) the second optical flow information, and (iii) a sub-block of the sub-block. Optionally, the method comprises combining the second decoded optical flow information and the sub-sub-block by concatenation before inputting the concatenated second decoded optical flow information and the sub-sub-block into the fourth neural network. Optionally, the first and second decoded optical flow information are defined in fixed point precision. Optionally, the method comprises combining the first and the second decoded optical flow information to produce combined optical flow information; and using the combined optical flow information to produce the output image. Optionally, the combined optical flow information is defined in floating point precision. According to a further aspect, there is provided a method for lossy image or video receipt and decoding, the method comprising the steps of: receiving a latent representation transmitted by a first computer system at a second computer system, the latent representation corresponding to optical flow information being indicative of a difference between a previously decoded image and a input image; and decoding the latent representation using a second neural network to produce first decoded optical flow information; decoding the latent representation using a third neural network and using the first decoded optical flow information to produce second decoded optical flow information, using the previously decoded image and the first and second decoded optical flow information to produce an output image, wherein the output image is an approximation of the input image. Optionally, the first decoded optical flow information comprises optical flow information associated with a first block of the latent representation, and wherein the second decoded optical flow information comprises optical flow information associated with a sub-block of the first block. Optionally, the method comprises splitting the latent representation into a plurality of blocks, producing the first decoded optical flow information with the second neural network using a first block of the plurality of blocks, and producing the second decoded optical flow information with the third neural network using: (i) the first decoded optical flow information and (ii) a sub-block of the first block. According to a further aspect, there is provided a method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image and a previously decoded image at a first computer system; estimating optical flow information using a first neural network, the optical flow information being indicative of a difference between the previously decoded image and the input image; producing a latent representation of the optical flow information; transmitting the latent representation of the optical flow information to a second computer system; decoding the latent representation of the optical flow information using a second neural network to produce first decoded optical flow information; decoding the latent representation of the optical optical flow information using a third neural network and the first decoded optical flow information to produce second decoded optical flow information; using the previously decoded image and the first and second decoded optical flow information to produce an output image, wherein the output image is an approximation of the input image; evaluating a function based on a difference between the input image and the output image; updating the parameters of the first, second, and third neural networks based on the evaluated function; and repeating the above steps using a first set of input images to produce first, second and third trained neural networks. Optionally, the first decoded optical flow information comprises optical flow information associated with a first block of the latent representation, and wherein the second decoded optical flow information comprises optical flow information associated with a sub-block of the first block. Optionally, the method comprises splitting the latent representation into a plurality of blocks, producing the first decoded optical flow information with the second neural network using a first block of the plurality of blocks, and producing the second decoded optical flow information with the third neural network using: (i) the first decoded optical flow information and (ii) a sub-block of the first block. Optionally, the latent representation and the output of the second neural network are defined in floating point precision, and wherein the method comprising performing a quantisation operation on the output of the second neural network to produce the first decoded optical flow information. Optionally, the output of the third neural network is defined in floating point precision, and wherein the method comprises performing a quantisation operation on the output of the third neural network to produce the second optical flow information. According to a further aspect, there is provbided a method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first neural network to produce a latent representation; decoding the latent representation using a second neural network to produce an output image, wherein the output image is an approximation of the input image; evaluating a function based on a difference between an output of a layer of the first or second neural network in a first precision and said output quantised to a second precision; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps using a first set of input images to produce a first trained neural network and a second trained neural network. Optionally, the first precision comprises floating point precision and wherein the second precision comprises integer precision. Optionally, the layer of the first or second neural network comprises an activation layer, and wherein the output comprises an output of the activation layer. Optionally, the method comprises quantising the output from the first precision to the second precision. Optionally, said quantising comprises applying a hook to the activation. Optionally, the layer of the first or second neural network comprises a convolution layer. Optionally, the method comprises quantising the tensor output by the convolution layer from the first precision to the second precision. Optionally, said quantising comprises applying a hook to the tensor. Optionally, the parameters of the first neural network and the second neural network are stored in the first precision for a first predetermined number of said steps, and wherein the method comprises: quantising the parameters of the first neural network and the second neural network into the second precision after said predetermined number of said steps, and performing said steps for a second predetermined number of said steps. Optionally, when performing said second predetermined number of steps, the function is based on a difference between the input image and the output image. Optionally, said difference comprises a mean squared error between the output in the first precision and the output quantised to the second precision. Optionally, the output of the layer of the first or second neural network comprises an output of a channel of the output. Optionally, the method comprises: entropy encoding the latent representation into a bitstream, and entropy decoding the bitstream to produce the latent representation, wherein the function is further based on the size of the bitstream. Optionally, the function is further based on a difference between the input image and the output image. According to a further aspect, there is provided a method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first neural network to produce a latent representation; decoding the latent representation using a second neural network to produce an output image, wherein the output image is an approximation of the input image; evaluating a function; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps using a first set of input images to produce a first trained neural network and a second trained neural network wherein the function is based on a difference between one or more of the parameters of the first or second neural networks in a first precision and said one or more parameters quantised to a second precision. Optionally, the first precision comprises floating point precision and wherein the second precision comprises integer precision. Optionally, the method comprises, between said steps, quantising the one or more parameters from the first precision into the second precision. Optionally, said quantising comprises, between said steps, applying a hook to one or more parameters. According to a further aspect, there is provided a data processing system configured to perform the above methods. BRIEF DESCRIPTION OF THE DRAWINGS Aspects of the invention will now be described by way of examples, with reference to the following figures in which: Figure 1 illustrates an example of an image or video compression, transmission and decom- pression pipeline. Figure 2 illustrates a further example of an image or video compression, transmission and decompression pipeline including a hyper-network. Figure 3 illustrates an example of a video compression, transmission and decompression pipeline. Figure 4 illustrates an example of a video compression, transmission and decompression system. Figure 5 illustrates a schematic of an example encoding phase of an AI-based compression algorithm, for video and image compression. Figure 6 illustrates an example decoding phase of an AI-based compression algorithm, for video and image compression. Figure 7 illustrates an example of a distribution of values for quantisation bin sizes that can be learned while training the AI-based compression pipeline. Figure 8 shows a heatmap indicating how the sizes of learned quantisation bins vary for a given image, across latent channels. Figure 9 shows a schematic of an example encoding phase of an AI-based compression algorithm utilizing a hyperprior and learned quantisation bin sizes, for video and image compression. Figure 10 shows a schematic of an example decoding phase of an AI-based compression algorithm utilizing a hyperprior and learned quantisation bin sizes, for video and image compression. Figure 11 illustrates parametric probability distributions over positive numbers. Figure 12 illustrates example activation functions. Figure 13 illustrates an example activation function. Figure 14 illustrates an example activation function. Figure 15 illustrates a sequence of layers from a neural network of a compression pipeline. Figure 16a illustrates a flow residual compression pipeline. Figure 16b illustrates a quantisation parameter lookup table. Figure 17 illustrates an example of a flow module of a flow residual compression pipeline. Figure 18a schematically illustrates a method of determining macro and micro flows in a flow residual compression pipeline. Figure 18b schematically illustrates a method of decoding a latent representation of flows in a flow residual compression pipeline. Figure 18c schematically illustrates a method of decoding a latent representation of flows in a flow residual compression pipeline. Figure 18d schematically illustrates a method of decoding a latent representation of flows in a flow residual compression pipeline during training. Figure 19 illustrates a sequence of layers from a neural network of a compression pipeline. DETAILED DESCRIPTION OF THE DRAWINGS Compression processes may be applied to any form of information to reduce the amount of data, or file size, required to store that information. Image and video information is an example of information that may be compressed. The file size required to store the information, particularly during a compression process when referring to the compressed file, may be referred to as the rate. In general, compression can be lossless or lossy. In both forms of compression, the file size is reduced. However, in lossless compression, no information is lost when the information is compressed and subsequently decompressed. This means that the original file storing the information is fully reconstructed during the decompression process. In contrast to this, in lossy compression information may be lost in the compression and decompression process and the reconstructed file may differ from the original file. Image and video files containing image and video data are common targets for compression. In a compression process involving an image, the input image may be represented as ^^. The data representing the image may be stored in a tensor of dimensions ^^ × ^^ × ^^, where ^^ represents the height of the image, ^^ represents the width of the image and ^^ represents the number of channels of the image. Each ^^ × ^^ data point of the image represents a pixel value of the image at the corresponding location. Each channel ^^ of the image represents a different component of the image for each pixel which are combined when the image file is displayed by a device. For example, an image file may have 3 channels with the channels representing the red, green and blue component of the image respectively. In this case, the image information is stored in the RGB colour space, which may also be referred to as a model or a format. Other examples of colour spaces or formats include the CMKY and the YCbCr colour models. However, the channels of an image file are not limited to storing colour information and other information may be represented in the channels. As a video may be considered a series of images in sequence, any compression process that may be applied to an image may also be applied to a video. Each image making up a video may be referred to as a frame of the video. The output image may differ from the input image and may be represented by ˆ. The difference between the input image and the output image may be referred to as distortion or a difference in image quality. The distortion can be measured using any distortion function which receives the input image and the output image and provides an output which represents the difference between input image and the output image in a numerical way. An example of such a method is using the mean square error (MSE) between the pixels of the input image and the output image, but there are many other ways of measuring distortion, as will be known to the person skilled in the art. The distortion function may comprise a trained neural network. Typically, the rate and distortion of a lossy compression process are related. An increase in the rate may result in a decrease in the distortion, and a decrease in the rate may result in an increase in the distortion. Changes to the distortion may affect the rate in a corresponding manner. A relation between these quantities for a given compression technique may be defined by a rate-distortion equation. AI based compression processes may involve the use of neural networks. A neural network is an operation that can be performed on an input to produce an output. A neural network may be made up of a plurality of layers. The first layer of the network receives the input. One or more operations may be performed on the input by the layer to produce an output of the first layer. The output of the first layer is then passed to the next layer of the network which may perform one or more operations in a similar way. The output of the final layer is the output of the neural network. Each layer of the neural network may be divided into nodes. Each node may receive at least part of the input from the previous layer and provide an output to one or more nodes in a subsequent layer. Each node of a layer may perform the one or more operations of the layer on at least part of the input to the layer. For example, a node may receive an input from one or more nodes of the previous layer. The one or more operations may include a convolution, a weight, a bias and an activation function. Convolution operations are used in convolutional neural networks. When a convolution operation is present, the convolution may be performed across the entire input to a layer. Alternatively, the convolution may be performed on at least part of the input to the layer. Each of the one or more operations is defined by one or more parameters that are associated with each operation. For example, the weight operation may be defined by a weight matrix defining the weight to be applied to each input from each node in the previous layer to each node in the present layer. In this example, each of the values in the weight matrix is a parameter of the neural network. The convolution may be defined by a convolution matrix, also known as a kernel. In this example, one or more of the values in the convolution matrix may be a parameter of the neural network. The activation function may also be defined by values which may be parameters of the neural network. The parameters of the network may be varied during training of the network. Other features of the neural network may be predetermined and therefore not varied during training of the network. For example, the number of layers of the network, the number of nodes of the network, the one or more operations performed in each layer and the connections between the layers may be predetermined and therefore fixed before the training process takes place. These features that are predetermined may be referred to as the hyperparameters of the network. These features are sometimes referred to as the architecture of the network. To train the neural network, a training set of inputs may be used for which the expected output, sometimes referred to as the ground truth, is known. The initial parameters of the neural network are randomized and the first training input is provided to the network. The output of the network is compared to the expected output, and based on a difference between the output and the expected output the parameters of the network are varied such that the difference between the output of the network and the expected output is reduced. This process is then repeated for a plurality of training inputs to train the network. The difference between the output of the network and the expected output may be defined by a loss function. The result of the loss function may be calculated using the difference between the output of the network and the expected output to determine the gradient of the loss function. Back-propagation of the gradient descent of the loss function may be used to update the parameters of the neural network using the gradients ^^ ^^ / ^^ ^^ of the loss function. A plurality of neural networks in a system may be trained simultaneously through back-propagation of the gradient of the loss function to each network. In the context of image or video compression, this type of system, where simultaneous training with back-propagation through each element or the whole network architecture may be referred to as end-to-end, learned image or video compression. Unlike in traditional compression algorithms that use primarily handcrafted, manually constructed steps, an end-to-end learned system learns itself during training what combination of parameters best achieves the goal of minimising the loss function. This approach is advantageous compared to systems that are not end-to-end learned because an end-to-end system has a greater flexibility to learn weights and parameters that might be counter-intuitive to someone handcrafting features. It will be appreciated that the term "training" or "learning" as used herein means the process of optimizing an artificial intelligence or machine learning model, based on a given set of data. This involves iteratively adjusting the parameters of the model to minimize the discrepancy between the model’s predictions and the actual data, represented by the above-described rate-distortion loss function. The training process may comprise multiple epochs. An epoch refers to one complete pass of the entire training dataset through the machine learning algorithm. During an epoch, the model’s parameters are updated in an effort to minimize the loss function. It is envisaged that multiple epochs may be used to train a model, with the exact number depending on various factors including the complexity of the model and the diversity of the training data. Within each epoch, the training data may be divided into smaller subsets known as batches. The size of a batch, referred to as the batch size, may influence the training process. A smaller batch size can lead to more frequent updates to the model’s parameters, potentially leading to faster convergence to the optimal solution, but at the cost of increased computational resources. Conversely, a larger batch size involves fewer updates, which can be more computationally efficient but might converge slower or even fail to converge to the optimal solution. The learnable parameters are updated by a specified amount each time, determined by the learning rate. The learning rate is a hyperparameter that decides how much the parameters are adjusted during the training process. A smaller learning rate implies smaller steps in the parameter space and a potentially more accurate solution, but it may require more epochs to reach that solution. On the other hand, a larger learning rate can expedite the training process but may risk overshooting the optimal solution or causing the training process to diverge. The training described herein may involve use of a validation set, which is a portion of the data not used in the initial training, which is used to evaluate the model’s performance and to prevent overfitting. Overfitting occurs when a model learns the training data too well, to the point that it fails to generalize to unseen data. Regularization techniques, such as dropout or L1 / L2 regularization, can also be used to mitigate overfitting. It will be appreciated that training a machine learning model is an iterative process that may comprise selection and tuning of various parameters and hyperparameters. As will be appreciated, the specific details, such as hyper parameters and so on, of the training process may vary and it is envisaged that producing a trained model in this way may achieved in a number of different ways with different epochs, batch sizes, learning rates, regularisations, and so on, the details of which are not essential to enabling the advantages and effects of the present disclosure, except where stated otherwise. The point at which an “untrained” neural network is considered be “trained” is envisaged to be case specific and depend on, for example, on a number of epochs, a plateauing of any further learning, or some other metric and is not considered to be essential in achieving the advantages described herein. More details of an end-to-end, learned compression process will now be described. It will be appreciated that in some cases, end-to-end, learned compression processes may be combined with one or more components that are handcrafted or trained separately. In the case of AI based image or video compression, the loss function may be defined by the rate distortion equation. The rate distortion equation may be represented by ^^ ^^ ^^ ^^ = ^^ + ^^ ∗ ^^, where ^^ is the distortion function, ^^ is a weighting factor, and ^^ is the rate loss. ^^ may be referred to as a lagrange multiplier. The langrange multiplier provides as weight for a particular term of the loss function in relation to each other term and can be used to control which terms of the loss function are favoured when training the network. In the case of AI based image or video compression, a training set of input images may be used. An example training set of input images is the KODAK image set (for example at www.cs.albany.edu / xypan / research / snr / Kodak.html). An example training set of input images is the IMAX image set. An example training set of input images is the Imagenet dataset (for example at www.image-net.org / download). An example training set of input images is the CLIC Training Dataset P (“professional”) and M (“mobile”) (for example at http: / / challenge.compression.cc / tasks / ). An example of an AI based compression, transmission and decompression process 100 is shown in Figure 1. As a first step in the AI based compression process, an input image 5 is provided. The input image 5 is provided to a trained neural network 110 characterized by a function ^^^^acting as an encoder. The encoder neural network 110 produces an output based on the input image. This output is referred to as a latent representation of the input image 5. In a second step, the latent representation is quantised in a quantisation process 140 characterised by the operation ^^, resulting in a quantized latent. The quantisation process transforms the continuous latent representation into a discrete quantized latent. An example of a quantization process is a rounding function. In a third step, the quantized latent is entropy encoded in an entropy encoding process 150 to produce a bitstream 130. The entropy encoding process may be for example, range or arithmetic encoding. In a fourth step, the bitstream 130 may be transmitted across a communication network. In a fifth step, the bitstream is entropy decoded in an entropy decoding process 160. The quantized latent is provided to another trained neural network 120 characterized by a function ^^^^acting as a decoder, which decodes the quantized latent. The trained neural network 120 produces an output based on the quantized latent. The output may be the output image of the AI based compression process 100. The encoder-decoder system may be referred to as an autoencoder. Entropy encoding processes such as range or arithmetic encoding are typically able to losslessly compress given input data up to close to the fundamental entropy limit of that data, as determined by the total entropy of the distribution of that data. Accordingly, one way in which end-to-end, learned compression can minimise the rate loss term of the rate-distortion loss function and thereby increase compression effectiveness is to learn autoencoder parameter values that produce low entropy latent representation distributions. Producing latent representations distributed with as low an entropy as possible allows entropy encoding to compress the latent distributions as close to or to the fundamental entropy limit for that distribution. The lower the entropy of the distribution, the more entropy encoding can losslessly compress it and the lower the amount of data in the corresponding bitstream. In some cases where the latent representation is distributed according to a gaussian or Laplacian distribution, this learning may comprise learning optimal location and scale parameters of the gaussian or Laplacian distributions, in other cases, it allows the learning of more flexible latent representation distributions which can further help to achieve the minimising of the rate-distortion loss function in ways that are not intuitive or possible to do with handcrafted features. Examples of these and other advantages are described in WO2021 / 220008A1, which is incorporated in its entirety by reference. Something which is closely linked to the entropy encoding of the latent distribution and which accordingly also has an effect on the effectiveness of compression of end-to-end learned approaches is the quantisation step. During inference, a rounding function may be used to quantise a latent representation distribution into bins of given sizes, a rounding function is not differentiable everywhere. Rather, a rounding function is effectively one or more step functions whose gradient is either zero (at the top of the steps) or infinity (at the boundary between steps). Back propagating a gradient of a loss function through a rounding function is challenging. Instead, during training, quantisation by rounding function is replaced by one or more other approaches. For example, the functions of a noise quantisation model are differentiable everywhere and accordingly do allow backpropagation of the gradient of the loss function through the quantisation parts of the end-to-end, learned system. Alternatively, a straight-through estimator (STE) quantisation model or one other quantisation models may be used. It is also envisaged that different quantisation models may be used for during evaluation of different term of the loss function. For example, noise quantisation may used to evaluate the rate or entropy loss term of the rate-distortion loss function while STE quantisation may be used to evaluate the distortion term. In a similar manner to how learning parameters to produce certain distributions of the latent representation facilitates achieving better rate loss term minimisation, end-to-end learning of the quantisation process achieves a similar effect. That is, learnable quantisation parameters provide the architecture with a further degree of freedom to achieve the goal of minimising the loss function. For example, parameters corresponding to quantisation bin sizes may be learned which is likely to result in an improved rate-distortion loss outcome compared to approaches using hand-crafted quantisation bin sizes. Further, as the rate-distortion loss function constantly has to balance a rate loss term against a distortion loss term, it has been found that the more degrees of freedom the system has during training, the better the architecture is at achieving optimal rate and distortion trade off. Exemplary details of how to implement quantisation of a latent representation in a learned compression pipeline are described in WO2023 / 012231A1, which is incorporated in its entirety by reference, and described below. In more detail, one approach is to achieve quantisation by rounding data to the nearest integer. This can be suboptimal, because some regions of images and video can tolerate higher information loss, while other regions require fine-grained detail. Below it is discussed how the size of quantisation bins can be learned, instead of fixed to nearest-integer rounding. We detail several architectures to achieve this, such as predicting bin sizes from hypernetworks, context modules, and additional neural networks. We also document changes to the loss function and quantisation procedure required for training AI-based compression pipelines with learned quantisation bin sizes, and show how to introduce Bayesian priors to control the distribution of bin sizes that are learned during training. We show how learned quantisation bins can be used both with or without split quantisation. This innovation also allows distortion gradients to flow through the decoder and to the hypernetwork. Finally, we give a detailed account of Generalised Quantisation Functions, which give performance and runtime improvements. The methods for learning quantisation bins described herein are compatible with all ways of transmitting metainformation, such as hyperpriors, autoregressive models, and implicit models. The following discussion will outline the functionality, scope and future outlook of learned quantisation bins and generalised quantisation functions for usage in, but not limited to, AI-based image and video compression. A compression algorithm may be broken into two phasess: the encoding phase and the decoding phase. In the encoding phase, input data is transformed into a latent variable with a smaller representation (in bits) than the original input variable. In the decode phase, a reverse transform is applied to the latent variable, in which the original data (or an approximation of the original data) is recovered. An AI-based compression system must be also be trained. This is the procedure of choosing parameters of the AI-based compression system that achieve good compression results (small file size and minimal distortion). During training, parts of the encoding and decoding algorithm are run, in order to decide how to adjust the parameters of the AI-based compression system. To be precise, in AI-based compression, encoding typically takes the following form: ^^ = ^^enc( ^^, ^^) (1a) ^ˆ^ = ^^( ^^) (1b) bitstream = ^^ ^^ ( ^ˆ^, ^^( ^ˆ^)) (1c) Here ^^ is the data to be compressed (image or video), ^^encis the encoder, which is usually a neural network with parameters ^^ that are trained. The encoder transforms the input data ^^ into a latent representation ^^, which is lower-dimensional and in an improved form for further compression. To compress ^^ further and transmit as a stream of bits, an established lossless encoding algorithm such as arithmetic encoding may be used. These lossless encoding algorithms may require ^^ to be discrete, not continuous, and also may require knowledge of the probability distribution of the latent representation. To achieve this, a quantisation function ^^ (usually nearest integer rounding) is used to convert the continuous data into discrete values ^ˆ^. The probability distribution ^^( ^ˆ^) is found by fitting a probability distribution onto the latent space. The probability distribution can be directly learned, or as often is the case, is a parameteric distribution with parameters determined by a hyper-network consisting of a hyper-encoder and hyper-decoder. If using a hyper-network, an additional bitstream ˆ (also known as “side information”) may be encoded, transmitted, and decoded: ^^ = ^^enc( ^^, ^^) (2a) ˆ = ^^( ^^) (2b) bitstream = ^^ ^^ ( ˆ ( ˆ)) (2c) ˆ = ^^ ^^ (bitstream, ^^( ˆ)) (2d) ^^^^, ^^^^= ^^dec( ˆ ) (2e) where ^^^^, ^^^^are the mean and scale parameters that determine the quantised latent distribution ^^( ^ˆ^). The encoding process (with a hyper-network) is depicted in Figure 5, which shows a schematic of an example encoding phase of an AI-based compression algorithm, for video and image compression. Decoding proceeds as follows: ^ˆ^ = ^^ ^^ (bitstream, ^^( ^ˆ^)) (3a) ˆ = ^^dec( ^ˆ^, ^^) (3b) Summarising: the distribution of latents ^^( ^ˆ^) is used in the arithmetic decoder (or other lossless decoding algorithm) to turn the bitstream into the quantised latents ^ˆ^. Then a function ^^dectransforms the quantised latents into a lossy reconstruction of the input data, denoted ˆ. In AI-based compression, ^^decis usually a neural network, depending on learned parameters ^^. If using a hyper-network, the side information bitstream is first decoded and then used to obtain the parameters needed to construct ^^( ^ˆ^), which is needed for decoding the main bitstream. An example of the decoding process (with a hyper-network) is depicted in Figure 6, which shows an example decoding phase of an AI-based compression algorithm, for video and image compression. AI-based compression depends on learning parameters for the encoding and decoding neural networks, using typical optimisation techniques with a “loss function.” The loss function is chosen to balance the goals of compressing the image or video to small file sizes, while maximising reconstruction quality. Thus the loss function consists of two terms: ^^ = ^^( ^^( ^ˆ^)) + ^^ ^^ ( ^^, ˆ) (4) Here ^^ determines the cost of encoding the quantised latents according to the distribution ^^( ^ˆ^), ^^ measures the reconstructed image quality, and ^^ is a parameter that determines the tradeoff between low file size and reconstruction quality. A typical choice of ^^ is the cross entropy ^^ = −E^^∼ ^^( ^^)[log ^^^ˆ^( ^ˆ^)] (5a) ^^^ˆ^( ^ˆ^) = ^^^^( ^ˆ^ + 1 / 2) − ^^^^( ^ˆ^ − 1 / 2) (5b) The choice of ^^^ˆ^( ^ˆ^) is due to quantisation: the latents are rounded to the nearest integer, so the probability distribution of ^^( ^ˆ^) is given by the integral of the (unquantised) latent distribution ^^( ^^) from ^ˆ^ − 1 / 2 to ^ˆ^ + 1 / 2, which is given in terms of the cumulative distribution function ^^( ^ˆ^). The function ^^ may be chosen to be the mean squared error, but can also be a combination of other metrics of perceptual quality, such as MS-SSIM, LPIPS, and / or adversarial loss (if using an adversarial neural network to enforce image quality). If using a hyper-network, an additional term may be added to ^^ to represent the cost of transmitting the additional side information: ^^ = ^^( ^^( ^ˆ^)) + ^^( ^^( ˆ)) (6) Altogether, note that the loss function depends explicitly on the choice of quantisation scheme through the ^^ term, and implicitly, because ^ˆ^ depends on the choice of quantisation scheme. It will now be discussed how learned quantisation bins may be used in AI-based image and video compression. The steps discussed are: • Architectures needed to learn and predict quantisation bin sizes • Modifications to the standard quantisation function and encoding and decoding process to incorporate learned quantisation bins • Methods for training neural networks that use learned quantisation bins A significant step in the typicalAI-based image and video compression pipeline is “quantisation,” where the pixels of the latent representation are usually rounded to the nearest integer. This is required for the algorithms that losslessly encode the bitstream. However, the quantisation step introduces its own information loss, which impacts reconstruction quality. It is possible to improve the quantisation function by training a neural network to predict the size of the quantisation bin that should be used for each latent pixel. Normally, the latents ^^ are rounded to the nearest integer, which corresponds to a “bin size” of 1. That is, every possible value of ^^ in an interval of length 1 gets mapped to the same ^ˆ^: ^^ ∈[round( ^^) − 1 / 2, round( ^^) + 1 / 2)↦−−^→^^ˆ^ (7)However, this may not be the optimal choice of information loss: for some latent pixels, more information can be disregarded (equivalently: using bins larger than 1) without impacting reconstruction quality much. And for other latent pixels, the optimal bin size is smaller than 1. This issue can be resolved by predicting the quantisation bin size, per image, per pixel. We do this with a tensor Δ ∈ R^^×R^^×R^^, which then modifies the quantisation function as follows:ˆ^^= ^^Δ( ^^) = round(^^Δ ) (8) We refer toˆ^^as the “quantised latent residuals.” Thus Equation 7 becomes: ^^ ∈[round( ^^) − Δ / 2, round( ^^) + Δ / 2)↦−−^→^ˆ^^ (9)indicating that values in an interval of length Δ get mapped to the same quantised latent value. Figure 7 shows an example of a distribution of values for quantisation bin sizes that can be learned while training the AI-based compression pipeline. Figure 8 shows a heatmap indicating how the sizes of learned quantisation bins vary for a given image, across latent channels. Different pixels are predicted to benefit from larger or smaller quantisation bins, corresponding to larger or smaller information loss. Note that because the learned quantisation bin sizes are incorporated into a modification of the quantisation function ^^, any data that we wish to encode and transmit can make use of learned quantisation bin sizes. For example, if instead of encoding the latents ^^, we wish to encode the mean-subtracted latents ^^ − ^^^^, this can be achieved: ^^ − ^^ ˆ^^^^= ^^Δ( ^^ − ^^^^) = round( ) (10) Δ Similarly, hyperlatents, hyperhyperlatents, and other objects we might wish to quantise can all use the modified quantisation function ^^Δ, for an appropriately learned Δ. Several architectures for predicting quantisation bin sizes will now be discussed. A possible architecture is predicting the quantisation bin sizes Δ using a hypernetwork. The bitstream is encoded as follows: ˆ^^= ^^Δ( ^^) = round(^^Δ ) (11a) where the division is element-wise.ˆ^^is now the object that is losslessly encoded and sent as a bitstream (we refer toˆ^^as the quantised latent residuals). An example of the modified encoding process using a hypernetwork is depicted in Figure 9, which shows a schematic of an example encoding phase of an AI-based compression algorithm utilizing a hyperprior and learned quantisation bin sizes, for video and image compression. When decoding, the bitstream is losslessly decoded as usual. We then use Δ to rescale ^^^^by multiplying the two element-wise. The result of this transformation is what is now denoted ^ˆ^ and passed to the decoder network as usual:ˆ^^= ^^ ^^ (bitstream) (12a) ^ˆ^ = Δ ⊙ˆ^^(12b) An example if the modified decoding process using a hypernetwork is depicted in Figure 10, which shows a schematic of an example decoding phase of an AI-based compression algorithm utilizing a hyperprior and learned quantisation bin sizes, for video and image compression. Applying the above techniques may lead to a 1.5 percent improvement in the rate of an AI based compression pipeline and and a 1.9 percent improvement in the distortion when measured by MSE. The performance of the AI based compression process is therefore improved. We detail several variants of the above architectures which are of use: • Quantisation bins sizes for the hyperlatents can be learned parameters, or we can include a hyperhyper-network that predicts these variables. • Prediction of quantisation bin sizes can be augmented with a so-called “context module” - any function that uses neighboring pixels to improve predictions for a given pixel. • After obtaining the quantisation bin sizes Δ^^from the hyper decoder, this tensor can be processed further by an additional nonlinear function. This nonlinear function would in general be a neural network (but is not limited to this choice). We also emphasise that our methods for learning quantisation bins are compatible with all ways of transmitting metainformation, such as hyperpriors, hyperhyperpriors, autoregressive models, and implicit models. To train neural networks with learned quantisation bins for AI-based compression, we may modify the loss function. In particular, the cost of encoding data described Equation 5a may be modified as follows: ^^ = −E^^∼ ^^( ^^)[log ^^^ˆ^( ^ˆ^)] −−→ −E^^∼ ^^( ^^)[log ^^ˆ ^^(ˆ^^)] (13a) ^^^ˆ^( ^ˆ^) = The idea is that we need to integrate the probability distribution of latents fromˆ^^− Δ^^ / 2 toˆ^^+ Δ^^ / 2 instead of integrating over an interval of length 1. Similarly, if using a hypernetwork, the term ^^^^= −E^^∼ ^^( ^^)[log ^^ˆ( ˆ)] corresponding to the cost of encoding the hyperlatents is modified in exactly the same way as the cost of encoding the latents is modified to incorporate learned quantisation bin sizes. Neural networks are usually trained by variants of gradient descent, utilising backpropagation to update the learned parameters of the network. This requires computing gradients of all layers in the network, which in turn requires the layers of the network to be composed of differentiable functions. However, the quantisation function ^^ and its learned bins modification ^^Δare not differentiable, because of the presence of the rounding function. In AI-based compression, one of two differentiable approximations to quantisation are used during training of the network to replace ^^( ^^) (no approximation is used once the network is trained and used for inference): Noise quantisation: ^˜^noise:= ^^ + ^^, ^^ ∼ ^^ (−1 / 2, 1 / 2), (14a) ^^^^ ^^ Straight-Through Estimator (STE) quantisation: ^˜^STE:= round( ^^),^˜^= = 1 ^^ ^^ ^^ ^^ When using learned quantisation bins, the approximations to quantisation during training are Noise quantisation: (˜^^)noise:= ^^ + Δ ⊙ ^^, ^^ ∼ ^^ (−1 / 2, 1 / 2), (15a) ^^ (˜)( ) q^^ STE^^ ^^ STE Δ ), = (15b) ^^ ^^ ^^ ^^ Δ Instead of choosing one of these differentiable approximations during training, AI-based compression pipelines can also be trained with “split quantisation,” where we use (˜^^)noisewhen calculating the rate loss ^^, but we send (˜^^)STEto the decoder in training. AI-based compression networks can be trained with both split quantisation and learned quantisation bins. First, note that split quantisation comes in two flavours: • Hard Split Quantisation: Decoder receives (˜^^)STE • Soft Split Quantisation: Decoder receives (˜^^)STEin the forward pass, but (˜^^)noiseis ˜ used in the backward pass of backpropagation when calculating^^^^^^ ^^^^ Notice that with integer rounding quantisation, hard split and soft split quantisation are equivalent, because in the backward pass ^^ ^˜^noise^^ ( ^^ + ^^) ^^ ^^ ^^ ^˜^ = = 1 = =STE(16) ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ However, hard split and soft split quantisation are not equivalentwhen using learned quantisation bins, because in the backward pass ^^ (˜^^)noise^^ ( ^^ + Δ ⊙ ^^) ˜ == 1 +^^Δ⊙ ^^ ≠1(1 −^^Δ) =^^ (^^)STE(17a) ^^ ^^ ^^ ^^ ^^ ^^ Δ ^^ ^^ ^^ ^^ ^^ (˜^^)noise^^ ( ^^ + Δ ⊙ ^^)^^ (== ^^ ≠ −1^^ ^^ = −^^ ˜^^)STE2 = (17b) ^^Δ ^^Δ Δ Δ ^^Δ Now we examine loss function gradients in each quantisation scheme. Rate gradients w.r.t. Δ are negative in every quantisation scheme: ^^ ^^ ^^ ^^^^˜^^^^ ^^ = + (18) ^^Δ^^˜^^^^Δ ^^Δ In every quantisation scheme, the rate term receives^^˜^^n˜oise, so^^Δ= ^^ . Then we have ^^ ^^˜^^ + Δ / 2˜^^ − Δ / 2= −1( ^^^( ) ( ^^ + 1 / 2) + ^^ ( ) (1 / 2 − ^^)) (19a) ^^ ^˜˜^^^^^^^^ because 1 / 2 − ^^ ≥ 0 always. So the rate gradients always Δ to increase Δ. Distortion gradients w.r.t. Δ differ by quantisation scheme. The gradients are ^^ ^^ ^^ ^^^^ ^^=dec(˜^^) ^^˜^^(20) ^^˜^^˜ In soft split quantisation,^^˜^^^^Δ= ^^ and ^^ [˜^^^^] = ^^ [˜^^] ^^ [ ^^], because the random noise ^^ in the backward pass is of the STE rounding that makes˜^^in the forward pass. This means that ^^^^ ^^ (] = ^^ [ dec˜^^)^^ [^^ ^^ ^^^^] (21a) ^^˜^^˜ Therefore, in soft split quantisation, rate gradients drive Δ larger, while distortion gradients are on average 0, so overall Δ → ∞ and training the network is not possible. Conversely, in hard split quantisation ^^ ^^ ( ) ^^ ^^ ^^ ( )˜ ^^ [^^ ^^dec˜^^˜^^˜˜] ≠ ^^ [^^ ^^dec˜^^˜] ^^ [^^^^] (22) ^^ ^^ ^^ ^^˜ because˜^^ (˜^^)STE^^is not independent of^^Δ. Altogether, if using split quantisation with learned quantisation bins, we use hard split quantisation and not soft split quantisation. The nontrivial distortion gradients that can be achieved with or without split quantisation mean that distortion gradients flow through the decoder and to the hypernetwork. This is normally not possible in models with a hypernetwork, but is a feature introduced by our methods for learning quantisation bin sizes. In some compression pipelines it is important to control the distribution of values that are learned for the quantisation bin sizes (this is not always necessary). When needed, we achieve this by introducing an additional term into the loss function: ^^ = ^^ + ^^ ^^ + ^^Δ(23a) ^^Δ= −E^^∼ ^^( ^^)[log ^^Δ(Δ)] (23b) ^^Δis characterised by the choice of distribution ^^Δ(Δ), which we refer to as a “prior” on Δ, following terminology used in Bayesian statistics. Several choices can be made for the prior: • Any parametric probability distribution over positive numbers. In particular, the inverse gamma distribution, several examples of which are shown in Figure 11. The distribution is shown for several values of its parameters. The parameters of the distribution can be fit from other models, chosen a priori, or learned during training. • A neural network that learns the prior distribution during training. In the previous section, we give detailed descriptions of a simplistic quantisation function ^^ that makes use of a tensor of bin sizes Δ: ^^^^= round( ^^simple( ^^)) = round(^^Δ ) (24) We can extend all of these methods to more generalised quantisation functions. In the general case ^^ is some invertible function of ^^ and Δ. Then encoding is given by: ^^^^= round( ^^general( ^^,Δ)) (25) and decoding is achieved by ^ˆ^ = ^^−ge1neral( ^^^^,Δ) (26a) This quantisation function is more flexible than Equation 24, resulting in improved performance. The generalised quantisation function can also be made context-aware e.g. by incorporating quantisation parameters that use an auto-regressive context model. All methods of the previous section are still compatible with the generalised quantisation function framework: • We predict the parameters from hypernetworks, context modules, and implicit equations if needed. • We modify the loss function appropriately. • We employ hard split quantisation if using split quantisation. • We introduce Bayesian priors on Δ in the loss function to control its behaviour as necessary. The more flexible generalised quantisation functions improve performance. In addition to this, the generalised quantisation function can depend on parameters that are determined auto-regressively, meaning that quantisation depends on pixels that have already been encod- ed / decoded:^ˆ^= ^^(ˆ^^,Δ), ^^ < ^^ (27) , In general, using auto-regressive context models improves AI-based compression performance. Auto-regressive Generalised Quantisation Functions are additionally beneficial from a runtime perspective. Other standard auto-regressive models such as PixelCNN require executing the arithmetic decoder (or other lossless decoder) each time a pixel is being decoded using the context model. This is a severe performance hit in real-world applications of image and video compression. However, the Generalised Quantisation Function framework allows is to incorporate an auto-regressive context model into AI-based compression, without the runtime problems of e.g. PixelCNN. This is because ^^−1acts auto-regressively onˆ, which is fully decoded from the bitstream. Thus the arithmetic decoder does not need to be run auto-regressively and the runtime problem is solved. The generalised quantisation function can be any invertible function. For example: • Invertible Rational Functions: ^^(·,Δ) = ^^(·,Δ) / ^^(·,Δ), where ^^, ^^ are polynomials. • Functions of Logarithms, Exponentials, trigonometric functions, with suitably restricted domains so that these functions are invertible. • Invertible functions of matrices ^^(·,Δ) • Invertible functions that include context parameters ^^ predicted by the hyper-decoder: ^^(·,Δ, ^^) Furthermore, ^^ in general need not have a closed form, or be invertible. For example, we can define ^^enc(·,Δ) and ^^dec(·,Δ) where these functions are not necessarily inverses of each other and train the whole pipeline end-to-end. In this case, ^^encand ^^deccould be neural networks, or modelled as specific processes such as Gaussian Processes, Probabilistic Graphical Models (simple example: Hidden Markov Models). To train AI-based Compression pipelines that use Generalised Quantisation Functions, we use many of the same tools as described above: • Modify the loss function appropriately (in particular the rate term) • Use hard split quantisation if using split quantisation • Introduce Bayesian priors and / or regularisation terms ( ^^1, ^^2, second moment penalties) into the loss function as necessary to control the distribution parameters Δ, and if needed, the distribution of context parameters ^^. Depending on the choice of Generalised Quantisation Function, other tools may be used for training the AI-based compression pipeline: • Techniques from Reinforcement Learning (Q-Learning, Monte Carlo Estimation, DQN, PPO, SAC, DDPG, TD3) • General Proximal Gradient Methods • Continuous Relaxations. Here the discrete Quantisation residuals are approximated by a continuous function. This function can have hyperparameters which control how smooth the function is, which are modified at various points in training to improve network training and final performance. There are several possibilities for context modelling that are compatible with the Generalised Quantisation Function framework: • ^^ can be an auto-regressive neural network. The most common example of this that can be adapted to ^^ is a PixelCNN-style network, but other neural network building blocks such as Resnets, Transformers, Recurrent Neural Networks, and Fully-Connected networks can be used in an auto-regressive ^^ function. • Δ can be predicted as some linear combination of previously-decoded pixels: ∑^^−^1Δ^^= ^^^^ ^^^ˆ^^^(29) Here ^^ is a matrix that can be For example ^^ could be banded, upper / lower triangular, sparse, only non-zero for ^^ elements preceding the current position being decoded (in raster-scan order). If we obtain important metainformation e.g. from an attention mechanism / focus mask we can incorporate this into Δ predictions. In this case, the size of quantisation bins adapts even more precisely to sensitive areas of images and videos, where knowledge of the sensitive regions is stored in this metainformation. In this way, less information is lost from perceptually important regions, while performs gains result from disregarding information from unimportant regions, in an enhanced way compared to AI-based compression pipelines that do not have adaptable bin sizes. We further outline a connection between learned quantisation bins and variable rate models: one form of variable ratemodels trains anAI-based compression pipelinewith a free hyperparameter ^^ controlling bin sizes. At inference time, ^^ is transmitted as metainformation to control the rate (cost in bits) of transmission - lower delta means small bins and larger transmission cost, but better reconstruction quality. In the variable rate framework, ^^ is a global parameter in the sense that it controlls all bin sizes simultaneously. In our innovation, we obtain the tensor of bin sizes Δ locally, that is predicted per pixel, which is an improvement. In addition, variable rate models that use ^^ to control the rate of transmission are compatible with our framework, because we can scale the local prediction element-wise by the global prediction as needed to control the rate during inference: Δlocal + global= ^^Δ (30) Returning to the compression pipeline more generally, the systems described above may be distributed across multiple locations and / or devices. For example, the encoder 110 may be located on a device such as a laptop computer, desktop computer, smart phone or server. The decoder 120 may be located on a separate device which may be referred to as a recipient device. The system used to encode, transmit and decode the input image 5 to obtain the output image 6 may be referred to as a compression pipeline. As described above in the context of quantisation, the AI based compression process may further comprise a hyper-network 105 for the transmission of meta-information that improves the compression process. The hyper-network 105 comprises a trained neural network 115 acting as a hyper-encoder ^^ℎand a trained neural network ℎ ^^125 acting as a hyper-decoder ^^^^. An example of such a is shown in Figure 2. Components of the system not further discussed may be assumed to be the same as discussed above. The neural network 115 acting as a hyper-decoder receives the latent that is the output of the encoder 110. The hyper-encoder 115 produces an output based on the latent representation that may be referred to as a hyper-latent representation. The hyper-latent is then quantized in a quantization process 145 characterised by ^^ℎto produce a quantized hyper-latent. The quantization process 145 characterised by ^^ℎmay be the same as the quantisation process 140 characterised by ^^ discussed above. In a similar manner as discussed above for the quantized latent, the quantized hyper-latent is then entropy encoded in an entropy encoding process 155 to produce a bitstream 135. The bitstream 135 may be entropy decoded in an entropy decoding process 165 to retrieve the quantized hyper-latent. The quantized hyper-latent is then used as an input to trained neural network 125 acting as a hyper-decoder. However, in contrast to the compression pipeline 100, the output of the hyper-decoder may not be an approximation of the input to the hyper-decoder 115. Instead, the output of the hyper-decoder is used to provide parameters for use in the entropy encoding process 150 and entropy decoding process 160 in the main compression process 100. For example, the output of the hyper-decoder 125 can include one or more of the mean, standard deviation, variance or any other parameter used to describe a probability model for the entropy encoding process 150 and entropy decoding process 160 of the latent representation. In the example shown in Figure 2, only a single entropy decoding process 165 and hyper-decoder 125 is shown for simplicity. However, in practice, as the decompression process usually takes place on a separate device, duplicates of these processes will be present on the device used for encoding to provide the parameters to be used in the entropy encoding process 150. Further transformations may be applied to at least one of the latent and the hyper-latent at any stage in the AI based compression process 100. For example, at least one of the latent and the hyper latent may be converted to a residual value before the entropy encoding process 150,155 is performed. The residual value may be determined by subtracting the mean value of the distribution of latents or hyper-latents from each latent or hyper latent. The residual values may also be normalised. To perform training of the AI based compression process described above, a training set of input images may be used as described above. During the training process, the parameters of both the encoder 110 and the decoder 120 may be simultaneously updated in each training step. If a hyper-network 105 is also present, the parameters of both the hyper-encoder 115 and the hyper-decoder 125 may additionally be simultaneously updated in each training step. The training process may further include a generative adversarial network (GAN). When applied to an AI based compression process, in addition to the compression pipeline described above, an additional neutral network acting as a discriminator is included in the system. The discriminator receives an input and outputs a score based on the input providing an indication of whether the discriminator considers the input to be ground truth or fake. For example, the indicator may be a score, with a high score associated with a ground truth input and a low score associated with a fake input. For training of a discriminator, a loss function is used that maximizes the difference in the output indication between an input ground truth and input fake. When a GAN is incorporated into the training of the compression process, the output image 6 may be provided to the discriminator. The output of the discriminator may then be used in the loss function of the compression process as a measure of the distortion of the compression process. Alternatively, the discriminator may receive both the input image 5 and the output image 6 and the difference in output indication may then be used in the loss function of the compression process as a measure of the distortion of the compression process. Training of the neural network acting as a discriminator and the other neutral networks in the compression process may be performed simultaneously. During use of the trained compression pipeline for the compression and transmission of images or video, the discriminator neural network is removed from the system and the output of the compression pipeline is the output image 6. Incorporation of a GAN into the training process may cause the decoder 120 to perform hallucination. Hallucination is the process of adding information in the output image 6 that was not present in the input image 5. In an example, hallucination may add fine detail to the output image 6 that was not present in the input image 5 or received by the decoder 120. The hallucination performed may be based on information in the quantized latent received by decoder 120. Details of a video compression process will now be described. As discussed above, a video is made up of a series of images arranged in sequential order. AI based compression process 100 described above may be applied multiple times to perform compression, transmission and decompression of a video. For example, each frame of the video may be compressed, transmitted and decompressed individually. The received frames may then be grouped to obtain the original video. The frames in a video may be labelled based on the information from other frames that is used to decode the frame in a video compression, transmission and decompression process. As described above, frames which are decoded using no information from other frames may be referred to as I-frames. Frames which are decoded using information from past frames may be referred to as P-frames. Frames which are decoded using information from past frames and future frames may be referred to as B-frames. Frames may not be encoded and / or decoded in the order that they appear in the video. For example, a frame at a later time step in the video may be decoded before a frame at an earlier time. The images represented by each frame of a video may be related. For example, a number of frames in a video may show the same scene. In this case, a number of different parts of the scene may be shown in more than one of the frames. For example, objects or people in a scene may be shown in more than one of the frames. The background of the scene may also be shown in more than one of the frames. If an object or the perspective is in motion in the video, the position of the object or background in one frame may change relative to the position of the object or background in another frame. The transformation of a part of the image from a first position in a first frame to a second position in a second frame may be referred to as flow, warping or motion compensation. The flow may be represented by a vector. One or more flows that represent the transformation of at least part of one frame to another frame may be referred to as a flow map. An example AI based video compression, transmission, and decompression process 200 is shown in Figure 3. The process 200 shown in Figure 3 is divided into an I-frame part 201 for decompressing I-frames, and a P-frame part 202 for decompressing P-frames. It will be understood that these divisions into different parts are arbitrary and the process 200 may be also be considered as a single, end-to-end pipeline. As described above, I-frames do not rely on information from other frames so the I-frame part 201 corresponds to the compression, transmission, and decompression process illustrated in Figures 1 or 2. The specific details will not be repeated here but, in summary, an input image ^^0is passed into an encoder neural network 203 producing a latent representation which is quantised and entropy encoded into a bitstream 204. The subscript 0 in ^^0indicates the input image corresponds to a frame of a video stream at position t = 0. This may be the first frame of an entire video stream or the first frame of a chunk of a video stream made up of, for example, an I-frame and a plurality of subsequent P-frames and / or B-frames. The bitstream 204 is then entropy decoded and passed into a decoder neural network 205 to reproduce a reconstructed image ˆ0which in this case is an I-frame. The decoding step may be performed both locally at the same location as where the input image compression occurs as well as at the location where the decompression occurs. This allows the reconstructed image ˆ0to be available for later use by components of both the encoding and decoding sides of the pipeline. In contrast to I-frames, P-frames (and B-frames) do rely on information from other frames. Accordingly, the P-frame part 202 at the encoding side of the pipeline takes as input not only the input image ^^^^that is to be compressed (corresponding to a frame of a video stream at position t), but also one or more previously reconstructed images ˆ^^−1from an earlier frame t-1. As described above, the previously reconstructed ˆ^^−1is available at both the encode and decode side of the pipeline and can accordingly be used for various purposes at both the encode and decode sides. At the encode side, previously reconstructed images may be used for generating a flow maps containing information indicative of inter-frame movement of pixels between frames. In the example of Figure 3, both the image being compressed ^^^^and the previously reconstructed image from an earlier frame ˆ^^−1are passed into a flow module part 206 of the pipeline. The flow module part 206 comprises an autoencoder such as that of the autoencoder systems of Figures 1 and 2 but where the encoder neural network 207 has been trained to produce a latent representation of a flow map from inputs ˆ^^−1and ^^^^, which is indicative of inter-frame movement of pixels or pixel groups between ˆ^^−1and ^^^^. The latent representation of the flow map is quantised and entropy encoded to compress it and then transmitted as a bitstream 208. On the decode side, the bitstream is entropy decoded and passed to a decoder neural network 209 to produce a reconstructed flow map ^^ . The reconstructed flow map ^^ is applied to the previously reconstructed image ˆ^^−1to generate a warped image ˆ^^−1, ^^. It is envisaged that any suitable warping technique may be used, for example bi-linear or tri-linear warping, as is described in Agustsson, E., Minnen, D., Johnston, N., Balle, J., Hwang, S. J., and Toderici, G. (2020), Scale-space flow for end-to-end optimized video compression. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 8503-8512), which is hereby incorporated by reference. It is further envisaged that a scale-space flow approach as described in the above paper may also optionally be used. The warped image ˆ^^−1, ^^is a prediction of how the previously reconstructed image ˆ^^−1might have changed between frame positions t-1 and t, based on the output flow map produced by the flow module part 206 autoencoder system from the inputs of ^^^^and ˆ^^−1. As with the I-frame, the reconstructed flow map ^^ and corresponding warped image ˆ^^−1, ^^may be produced both on the encode side and the decode side of the pipeline so they are available for use by other components of the pipeline on both the encode and decode sides. In the example of Figure 3, both the image being compressed ^^^^and the ˆ^^−1, ^^are passed into a residual module part 210 of the pipeline. The residual module part 210 comprises an autoencoder system such as that of the autoencoder systems of Figures 1 and 2 but where the encoder neural network 211 has been trained to produce a latent representation of a residual map indicative of differences between the input mage ^^^^and the warped image ˆ^^−1, ^^. The latent representation of the residual map is then quantised and entropy encoded into a bitstream 212 and transmitted. The bitstream 212 is then entropy decoded and passed into a decoder neural network 213 which reconstructs a residual map ^^ from the decoded latent representation. Alternatively, a residual map may first be pre-calculated between ^^^^and the ˆ^^−1, ^^and the pre-calculated residual map may be passed into an autoencoder for compression only. This hand-crafted residual map approach is computationally simpler, but reduces the degrees of freedom with which the architecture may learn weights and parameters to achieve its goal during training of minimising the rate-distortion loss function. Finally, on the decode side, the residual map ^^ is applied (e.g. combined by addition, subtraction or a different operation) to the warped image to produce a reconstructed image ˆ^^which is a reconstruction of image ^^^^and accordingly corresponds to a P-frame at position t in a sequence of frames of a video stream. It will be appreciated that the reconstructed image ˆ^^can then be used to process the next frame. That is, it can be used to compress, transmit and decompress ^^^^+1, and so on until an entire video stream or chunk of a video stream has been processed. Thus, for a block of video frames comprising an I-frame and ^^ subsequent P-frames, the bitstream may contain (i) a quantised, entropy encoded latent representation of the I-frame image, and (ii) a quantised, entropy encoded latent representation of a flow map and residual map of each P-frame image. For completeness, whilst not illustrated in Figure 3, any of the autoencoder systems of Figure 3 may comprise hyper and hyper-hyper networks such as those described in connection with Figure 2. Accordingly, the bitstream may also contain hyper and hyper-hyper parameters, their latent quantised, entropy encoded latent representations and so on, of those networks as applicable. Finally, the above approach may generally also be extended to B-frames, for example as is described in Pourreza, R., and Cohen, T. (2021). Extending neural p-frame codecs for b-frame coding. In Proceedings of the IEEE / CVF International Conference on Computer Vision (pp. 6680-6689). The above-described flow and residual based approach is highly effective at reducing the amount of data that needs to be transmitted because, as long as at least one reconstructed frame (e.g. I-frame ˆ^^−1) is available, the encode side only needs to compress and transmit a flow map and a residual map (and any hyper or hyper-hyper parameter information, as applicable) to reconstruct a subsequent frame. Figure 4 shows an example of an AI image or video compression process such as that described above in connection with Figures 1-3 implemented in a video streaming system 400. The system 400 comprises a first device 401 and a second device 402. The first and second devices 401, 402 may be user devices such as smartphones, tablets, AR / VR headsets or other portable devices. In contrast to known systems which primarily perform inference on GPUs such as Nvidia A100, Geforce 3090, Gefore 4090 GPU cards, the system 400 of Figure 4 performs inference on a CPU (or NPU or TPU as applicable) of the first and second devices respectively. That is, compute for performing both encoding and decoding are performed by the respective CPUs of the first and second devices 401, 402. This places very different power usage, memory and runtime constraints on the implementation of the above methods than when implementing AI-based compression methods on GPUs. In one example, the CPU of first and second devices 401, 402 may comprise, for example, a Qualcomm Snapdragon CPU. The first device 401 comprises a media capture device 403, such as a camera, arranged to capture a plurality of images, referred to hereafter as a video stream 404, of a scene 404. The video stream 404 is passed to a pre-processing module 406 which splits the video stream into blocks of frames, various frames of which will be designated as I-frames, P-frames, and / or B-frames. The blocks of frames are then compressed by an AI-compression module 407 comprising the encode side of the AI-based video compression pipeline of Figure 3. The output of the AI-compression module is accordingly a bitstream 408a which is transmitted from the first device 401, for example via a communications channel, for example over one or more of a WiFi, 3G, 4G or 5G channel, which may comprise internet or cloud-based 409 communications. The second device 402 receives the communicated bitstream 408b which is passed to an AI-decompression module 410 comprising the decode side of the AI-based video compression pipeline of Figure 3. The output of the AI-decompression module 402 is the reconstructed I-frames, P-frames and / or B-frames which are passed to a post-processing module 411 where they can prepared, for example passed into a buffer, in preparation for streaming 412 to and rendering on a display device 413 of the second device 402. It is envisaged that the system 400 of Figure 4 may be used for live video streaming at 30fps of a 1080p video stream, which means a cumulative latency of both the encode and decode side is below substantially 50ms, for example substantially 30ms or less. Achieving this level of runtime performance with only CPU (or NPU or TPU) compute on user devices presents challenges which are not addressed by known methods and systems or in the wider AI-compression literature. For example, execution of different parts of the compression pipeline during inference may be optimized by adjusting the order in which operations are performed using one or more known CPU scheduling methods. Efficient scheduling can allow for operations to be performed in parallel, thereby reducing the total execution time. It is also envisaged that efficient management of memory resources may be implemented, including optimising caching methods such as storing frequently-accessed data in faster memory locations, and memory reuse, which minimizes memory allocation and deallocation operations. A number of concepts related to the AI compression processes and / or their implementation in a hardware system discussed above will now be described. Although each concept is described separately, one or more of the concepts described below may be applied in an AI based compression process as described above. Concept 1: Layer output clamping An example of a known neural network architecture comprises consecutive convolution layers and activation layers. For example, a convolution layer followed by an activation layer, followed by a convolution layer followed by an activation layer, and so on. The activation layers apply an activation function to their input, typically a rectified linear unit (ReLU) activation function. The outputs are then passed on to the next convolution layer, whose output is then passed onto the next activation layer and so on. In the context of an AI compression pipeline, this type of architecture may be used for various components. For example, on the encode side it may be the encoder that produces the latent representation of the input image, the hyper-encoder that produces the hyper-latent representation, the hyper-hyper-encoder, that produces the hyper-hyper-latent representation, and so on. On the decode side it may be the decoder, hyper-decoder, hyper-hyper-decoder and so on. In addition, in a flow residual model used for video compression, it may be used in the flow and / or residual encoder and / or decoders. When implementing these architectures in practice, in an ideal scenario, during inference, the weights and biases of the layers would be floating point data types of arbitrary precision (e.g. fp16, fp32 or fp64). Operating in floating point precision allows activation functions like the ReLU and others, such as leaky ReLU, to avoid becoming saturated and / or avoids ReLU death occurring. Indeed training is typically performed in floating point precision. However, when implementing AI compression pipelines where inference is intended to be run in real time on devices with limited memory, compute, and power available, real time with a focus on speed, operating on floating point data types results in an unacceptable draw on these memory, compute, and power resources. For example, running an AI compression pipeline on a mobile device such as a smartphone or a laptop in floating point precision can in many cases result in very high CPU (or NPU, or TPU) usage, very high power draw (reducing battery life times), and very high memory usage (preventing other applications from functioning). Instead, in known AI compression pipelines, the weights and biases of the layers, and the latent representation output by the encoder neural network are quantized to integer data type precision (also known as fixed-point precision). In the case of int8, this means quantising the weights, biases and latent representation to a discrete set of 256 values. Whilst it is possible for the operations used to apply the convolutions and activation function, and for their intermediate outputs (e.g. activations) to remain implemented in floating point precision this again results in memory, compute, and power resource difficulties. Instead, not only are the weights, biases and inputs quantized into the lower precision integer space (i.e. fixed point precision of e.g. int8), but the outputs and inputs of the intermediate layers are also quantized into the lower precision space. For example, an int8 input to a convolution and / or activation layer may result in a int32 output which is then requantised back into int8, which means the data flow through the network remains in the fixed point precision space. This allows inference to be run on devices with lower hardware overheads but there are a number of consequences of this. Firstly, an activation function, such as a ReLU, operating at floating point precision effectively has no maximum output (i.e. y-axis value). The upper right hand side of the ReLU can, for all intents and purposes, extend to the infinity (within the confines of floating point precision regime being used e.g. 264possible values for fp32 precision). This changes when operating in integer i.e. fixed point precision space, particularly in int16, int8 and int4 space. For example, in int8 space, the total number of discrete values available is only 256. This means that the upper right tail of the ReLU that can extend to infinity in floating point precision (e.g. up to 264in fp32 precision) is now confined to much less than that (e.g. 256 discrete values in int8 precision). The result of this is that, in inference, when operating in integer precision, the ReLU is not actually a ReLU at all. In fact, it loses many of its properties that rely on it being able to extend to infinity. The same applies to the related leaky ReLU. As both ReLU and leaky ReLU are the typical first choice of activation layer activation functions in AI compression, this contributes in part to the decrease in performance (e.g. less compression, more artefacts, and so on) when compression pipelines are implemented in integer data type precision space (E.g. int32, int16, int8, int4 and so on) on pipelines trained in floating point precision. More specifically, if training an AI compression pipeline is performed with floating point precision with ReLU or leaky ReLU activation functions, the pipeline in inference is expecting to take advantage of the properties of ReLUs or leaky ReLUs. However, when the activations are implemented in integer precision, the activation function is effectively converted into different activation function than that which was present in training as it extends across a very small or narrow range of discrete values (e.g. 256 for int8). In integer precision, this means the range of possible values cannot exceed the maximum value of the range. For example, if each value in int8 is a real number between 1 and 256, the maximum activation output of a ReLU in int8 precision is 256, which means the ReLU is not really a ReLU at all, which is expected to have a tail that can increase up to infinity. In other words, the activation functions during training in floating point precision have different properties to the activation functions during inference due to the implementation in integer precision. In order to overcome these and other problems of implementing the compression pipeline in integer precision, and particularly implementing the intermediate inputs and outputs of the individual layers of the neural networks used in the pipeline in integer precision, the inventors have realized that clamping the activations (i.e. the outputs of the activation layers) of one or more layers of the neural networks in the pipeline during training at or below a value simulates the behaviour of the activation layers implemented in integer precision during inference. In this way, the training more closely mirrors how the activations of the networks of the pipeline flow through the network when it is implemented in integer precision implemented for inference and this results in an inference performance increase (e.g. better compression and / or fewer artefacts). Not only does the inference performance improve, but clamping outputs of the layers is computationally cheap to implement compared to other approaches of quantisation aware training that try to take quantisation into account during training. As a result, implementing this technique in practice does not require extensive redesign of the architectures or training techniques used and does not negatively impact training run time. This allows clamping to be “dropped in” to the activation functions of the activation layers of existing training pipelines to provide a cheap and effective inference performance boost with next to no downside. Further, if the clamping is implemented with a function that can be backpropagated through, then it provides a cheaper and quicker way to implement quantisation aware training regime compared to other approaches (e.g. STE) that are used to overcome the inability to backpropagate through a rounding function. Examples of functions that may be used to implement clamping include tanh and hardtanh functions with minimum and maximum values set to the clamping range. The clamping range may be fixed, learned or estimated dynamically based on a property of the values being clamped or the inputs that resulted in them. In a specific example where the clamping is implemented in a language such as PyTorch, dynamically estimating the range may comprise using an observer object type or class to estimate one or more parameters or properties of the distribution being clamped, such as one or more scale or location parameters, and an approximation of corresponding min and max value between which the clamping is to be performed. Alternatively, to make clamping implementation even quicker and more efficient to implement without needing to otherwise alter an existing architecture, the activation functions of the existing architecture (e.g. that uses ReLU or leaky ReLU activation functions) can be supplemented with a hook that modifies any values greater than the maximum of the clamping range to be at or lower than that maximum. Effectively, this approach allows the ReLU or leaky ReLU to retain its properties while below the maximum, and only have different properties when at or above the maximum. It will be appreciated that a hook is a known approach in software engineering whereby a hook is a function that is automatically executed based on the occurrence of another event. In this case, the event may be the activation function successfully being applied to an input to produce an output from that layer. In a specific context of implementing AI compression pipelines using PyTorch, the hook may be implemented with a PyTorch hook class. The inventors have also realised that, as ReLU and leaky ReLU activation functions are widely used in the machine learning community, there exist highly efficient implementations of these activation functions on a wide variety of hardware and programming libraries, making them more readily compatible and faster "out of the box" across a wide variety of processor types at the machine level (e.g. in a neural accelerator) compared to hardtanh and / or leaky hardtanh activation functions which in same cases are not implemented efficiently at the machine level in known libraries. Accordingly, the software hook applied to an efficiently implemented ReLU or leaky ReLU typically results in much faster implementation of the activation function at the machine level than a corresponding hardtanh or leaky hardtanh activation function implementation. Figure 12 shows a toy example plot 1200 of two ReLU activation functions 1201, 1202 respectively implemented in fp32 precision and int8 precision. The output space as determined by f(x) of the non-zero gradient part of the fp32 ReLU 1201 can output values from a 264set which is, for all intents and purposes, an arbitrary or infinite level of precision. That is, infinitely thin bin sizes up to the precision limit of the fp32 values. The fp32 implementation 1201 of the ReLU accordingly retains the advantageous non-saturation property of the ReLU. In contrast, the int8 implementation 1202 can only output values f(x) from a set of 256 values. Stretching the 256 values out over an infinite range would result in such wide bin sizes that all or most information would be lost. Accordingly, a useable implementation of the int8 ReLU is capped 1203 at a maximum value, which corresponds to one of the 256 possible values. Effectively, any x that would result in f(x) greater than the maximum value of the 256 possible values is binned into that maximum value. The resulting int8 ReLU implementation is accordingly not a ReLU at all but more closely resembles a step type function that is bound in a range of 256 possible values between a minimum value and a maximum value. Figure 13 shows a toy example plot 1204 of an activation function 1205 where the output values are spread in a range 1208 between a minimum, lower value 1206, and a maximum 1207 value. In other words, the output f(x) of the activation function is clamped below a first value and / or clamped above a second value. As described above, the inventors have realized that training the neural networks of the compression pipeline with an activation function implemented in floating point precision such as the fp32 ReLU of Figure 12 but implementing that activation function in integer precision in inference results causing the behaviour of that function to change in inference results in underperformance of the networks during inference. Instead, the inventors have found that training with an activation function that behaves (e.g. has similar shape and / or properties) more closely to how an activation function implemented in integer precision behaves in inference results in an increase in performance of that trained network during inference (e.g. higher compression rates and lower distortion rates). An example of what such an activation function is shown in Figure 13. In Figure 13 the activation function is a hardtanh function. Alternatively, a tanh or other similar function may be used that has a similar shape to and / or has properties of a clamped function. In a further alternative, rather than using an activation function in training that inherently behaves like an int8 implemented activation function, it is also possible to take any activation function with different behaviour and properties, and to apply a post-processing step to its output whereby the output value is modified with, for example, a software hook that clamps the output below a value and that is applied responsive to the activation function successfully outputting a value. This approach provides the advantage that it is quick to implement as it does not require any significant other changes to an existing training pipeline. An example of this is shown in Figure 14 which illustrates a toy example plot 1209 of a ReLU activation function 1210 whose output is clamped in the range 1208 between a minimum value 1206 and a maximum value 1207 whereby the clamping is performed with a hook 1211 that is applied to the output to bring any output values above the maximum value 1207 back down to the maximum value 1207. It will also be appreciated, that the clamping need not be applied to all of the activation outputs. In some scenarios, clamping only one or some but not all of the activations achieves improved performance over clamping all activations. It will also be appreciated that the higher and / or lower values of the clamp can be learned parameters and vary across different layers. Further, applying the above clamping approach can in some cases result in the network learning weights that produce ever decreasing, smaller and smaller activations, which becomes detrimental to performance after a number of training epochs. Effectively, the network is trying to scale the values from the arbitrary floating point precision of e.g. fp32 into the much smaller clamped range defined by the int8 values by scaling the activations down into smaller and smaller values. This can decrease the performance of the trained network in inference if the activations end up tending to zero. To overcome this, the inventors realized that adding noise to one or more of the clamped activa- tions during training helps to prevent this phenomenon. More specifically, the performance of combining clamping with noise unintuitively results in a greater improvement in performance than the sum of the improvements of each approach individually. More specifically, if only clamping is performed, the model starts to emulate quantisation but then after a number of training iterations begins to "cheat" by simply minimising activation values as a rounding error of a very small value is a very small error. Conversely, if only noise is added to the activations without clamping, the model begins to learn to spread out and increase the dynamic range (e.g. the min and max values) of the activations in an attempt to drown out the effect of the added noise as the noise is effectively rounded away whenever the bins are wide and spread out over a very large range. By including both clamping and noise, the model pushed away from learning to (i) spread out the dynamic range by the clamping which pushes the values back down below the max value of the clamp, and (ii) minimise activations by the clamping which pushes values back above the min value of the clamp. In this way, the model is prevented from "cheating" and its training is kept on track enabling it to learn far better what the actual quantised environment looks like. Adding noise to activations is explained in more detail in concept 2 below. These concepts may be applied alone or together with any of the other concepts described herein with reference to any of the drawings. For example, the activation outputs may have noise applied to them first as described in concept 2 and then clamped, or clamped and then have noise applied to them. It is preferred to have noise applied first before the clamping. If it is the other way around, there is a risk that any added noise ends up exceeding the clamped bounds and this can result in an unstable model, especially if the quantisation parameters are learned. More specifically, there may be a cascading effect of the model learning to increase the values of the quantisation parameters of one or more layers to take into account of any noise leaking outside of the previous layers’ clamped range, which in turn would make the noise bigger, and so on. An effect of implementing the above-described clamping is that it can partially contribute to a reduction in the severeness of a an artefact type that is unique to end-to-end AI-based video compression pipelines: "ringing" around edges in image sequences with little to no movement. These artefacts occur when the neural networks of an AI-based compression pipeline - trained to handle motion between frames - hallucinate motion into frame sequences where no movement occurs. These hallucinated movements manifest themselves around edges of objects in images and appear as halo-like or outline-like effects around the edges. Once these artefacts appear, the networks can cause these to cascade, resulting in significantly worse rate and distortion scores as a sequence of frames progress from an initial I-frame to subsequent P-frames. The inventors have found that the above-described clamping, for example when applied to a final layer of a flow decoder neural network or residual decoder neural network when producing P-frames of a sequence, can in many cases prevents the cascade of the hallucinated movements across that sequence. This in turn results in a more stable rate distortion scores that do not get worse across the sequence. Concept 2: Adding noise to intermediate layer outputs Quantisation in respect of the latent representation has been described in detail above, for example in the context of learned bin sizes. However, something which is not discussed above is applying those concepts to the quantisation of the inputs and / or outputs to some or each of the layers of the neural networks of the compression pipeline. In other words, the quantisation of the activations and specifically how running a fully quantised pipeline during inference results in underperformance in terms of compression rate and / or distortion manifesting as artefacts. As with concept 1, to achieve acceptable encode and decode times during inference, the weights, biases, inputs and activations of the layers of the neural networks of the pipeline, for example the encoder, decoder, hypreencoder, hypderdecoder networks and so on, are quantised, for example to int16, int8 or int4 precision which results in some information destruction. In contrast, during training, these parameters are typically left in some arbitrary floating point precision, for example fp32 or fp16. The result of this difference is that these networks, specifically each layer within them, are trained to expect higher precision, floating point precision data as inputs but in inference are fed integer type data. More specifically, the inventors have realized that the quantisation of the activations flowing between network layers during inference disproportionately effects the overall performance of the compression pipeline during inference relative to the effect that quantisation of other data flows in the pipeline. The inventors have also realized that this and other problems may be overcome by making the networks activation of the networks during training quantisation aware by adding noise specifically to one or more activations as they are passed from one layer to the next (i.e. to the intermediate activations between the various layers, rather than just to the final output of that set of layers. This simulates the “error” that arises not just from the quanisation of e.g. the latent representation or overall output of a the encoder, decoder etc, but also from the quantisation specifically of the intermediate activations between layers. In the context of the clamping of concept 1 above, it also prevents the network from learning to produce smaller and smaller activations during training. In more detail, as described above, a typical neural network architecture of, for example, the encoder, decoder, hyperencoder, hyperdecoder, hyperhyperencoder, hyperhyperdecoder, and so on comprises consecutive convolution layers and activation layers. For example, a convolution layer followed by an activation layer, followed by a convolution layer followed by an activation layer, and so on. The activation layers apply an activation function to their input, typically a rectified linear unit (ReLU) activation function (but in our case it may be some other function, such as the hardtanh function described above). The outputs are then passed on to the next convolution layer, whose output is then passed onto the next activation layer and so on. In the context of an AI compression pipeline, this type of architecture may be used for various components. For example, on the encode side it may be the encoder that produces the latent representation of the input image, the hyper-encoder that produces the hyper-latent representation, the hyper-hyper-encoder, that produces the hyper-hyper-latent representation, and so on. On the decode side it may be the decoder, hyper-decoder, hyper-hyper-decoder and so on. In addition, in a flow residual model used for video compression, it may be used in the flow and / or residual encoder and / or decoders. The data that is passed from one convolution layer to the activation layer during training may be a floating point precision type data object, or some higher precision integer type data object such as an int32 object. The activation layer processes the input data object by applying an activation function to it resulting in an output data object which in training typically also has floating point precision or some high precision integer data type precision such as in32. To simulate the binning that would occur to corresponding data being output by that layer during inference that is running in lower precision (e.g. in int8 precision), noise is added to the higher precision output of the activation layer during training before it is passed on to the next convolution layer where the same process is performed, and so on. In this way, the network is trained to expect the data flow between the intermediate layers to approximate the quantised (i.e. binned) data that would flow between the layers during inference being run in lower precision, not just the final output of the network. It is also noted that this approach can be used in addition to (or instead of) other approaches to quantisation aware training, such as training a network in floating point precision for a first number of epochs, before quantising the weights to integer precision and training for a second number of epochs. It should also be noted that this approach can also or alternatively be applied to the output of a convolution layer or other layer in the network before it is passed to the activation layer, as well as to the weights themselves. In other words, noise can be added to any data as it flows between any layer in the network, or defines the network. When this approach is applied to weights specifically, it effectively allows for training of the weights in floating point space for at least part of the training epochs to be quantization aware, before the weights are later actually quantised and the training continues in integer space for a predetermined number of epochs (e.g. train for 500 epochs in floating point precision e.g. fp32 while adding noise to the weights between epochs, then quantise those weights into e.g. int8 and continue to train for 150 epochs). In practice, adding noise may be implemented by using a hook on the output of a given layer, for example on the output of an activation and / or convolution layer. It will be appreciated that a hook is a known approach in software engineering whereby a hook is a function that is automatically executed based on the occurrence of another event. In this case, the event may be the activation function successfully being applied to an input. In a specific context of implementing AI compression pipelines using PyTorch, the hook may be implemented with a PyTorch hook class. Figure 15 illustratively shows a sequence 1500 of convolution layers 1501, 1503 and activation layers 1502, 1504 from a neural network, such as the encoder or decoder networks of an AI compression pipeline. The dotted lines indicate the networks may have any number of such layers and / or other layers not shown. The first convolution layer 1501 receives input data, to which a convolution is applied, the output from 1505 from that layer is passed to an activation layer 1502, which may comprise an activation function such as a ReLU, leaky ReLU, hardtanh, or some other function such as a clamped function as described above in concept 1. The activation function is applied, producing an output 1506, to which noise 1507 is added. The noised output is then passed into the next convolution layer 1503, which in turn produces the next output 1508 which is passed into the next activation layer 1504 where another activation function is applied. Noise 1510 is then applied to the output 1509 of that activation layer, and so on. The dotted lines indicate this process may be applied to any number such layers in the sequence 1500. It is also noted that the noise may be applied to the output of the convolution layers in addition or instead of to the output of the activation layers. In more general terms, anywhere in the sequence of layers where input data is processed to produce an output, noise may be applied to that output. By applying the process of Figure 15 to the neural networks of an AI compression pipeline during training in higher precision space, the network layers each expect to receive noisy input. Accordingly, when quantisation is applied to data flows between layers during inference (which effectively corresponds to adding noise that pushes high precision values into lower precision bins), the networks have been trained to expect this and accordingly do not suffer from performance drops as a result of quantisation. The profile of the noise 1507, 1510 added may comprise, for example, uniform, Gaussian, Laplacian or some other noise 1511, for example having a predetermined or learned a scale, location and / or other parameter based on e.g. a profile of the quantisation bin width which may be naively based on the precision of the quantisation that is intended to be applied in inference, or on some learned quantisation bin width. In the case of a learned quantisation bin width, these may be learned as described in the general section above. Different noise profiles may be applied to different outputs within each sequence of layers, whereby noise may be applied to some layers but not to others. When the clamping of concept 1 is being used, one envisaged noise profile to use when a subsequent quantisation into int8 is expected is uniform noise with an amplitude of the quantisation range divided by 256 (i.e. the scale). In other words, the noise amplitude equals the quantisation bin, i.e. scale, divided by 2. Accordingly where the quantisation bin sizes are learned or different across layers, the noise profiles of each may be different. These concepts may be applied alone or together with any of the other concepts described herein with reference to any of the drawings. Concept 3: Quantisation parameter synchronisation At a fundamental implementation level, the AI compression pipeline described herein comprises a performing plurality of operations on input data. These operations may be, for example, various additions, concatenations, subtractions, multiplications (matrix or otherwise) and many others. For example, in the case of a flow residual model, estimating residuals may comprise subtracting one data output from another. In another example, applying convolution layers to data input to those layers may comprise performing matrix multiplication and / or to those inputs. In another example, implementing an activation function efficiently in a library such as PyTorch or similar may involve breaking it down into simpler operations such as additions, subtractions, multiplications, and so on, which can be efficiently performed by a processing unit. A known approach of doing this is the multiply-accumulate operation which can be efficiently implemented on neural network accelerators. In more general terms, implementing theoretical network architectures on hardware typically involves breaking them down into operations which can be performed quickly and efficiently on the hardware being used. Similarly, at implementation level when the weights, biases, inputs, activations, latent repre- sentation, and so on of a pipeline are quantised (something which is inherent to implementing neural network architectures on digital rather than analogue hardware) and the quantisation parameters (e.g. a scale and location parameters) for each of these are learned, the quantisation parameters for each of the weights, biases, inputs, activations, latent representation, and so on, are each unique with respect to each other. That is, for every one of these parameters and variables being quantised, there may be learned quantisation parameters (e.g. scale and location parameters) that are used to bin the parameter and / or variable into whatever quantised precision that the implementation is using (e.g. int8, and so on). Consider for example, the learned bin sizes of the latent representation defined by a scale and location parameter output by the hyperdecoder in Figure 6 that are indicative of the distribution of the latent representation and that are used to entropy decode the bitstream. In training, these quantisation parameters may be some arbitrary floating point precision. However, in inference, these location and scale parameters are quantised into bins of their own, defined by their own (optionally learned) location and scale parameter. Going one layer back in Figure 6, the weights, biases, inputs, activations, and so on of the hyperdecoder are also quantised. That is, a fp16 weight value in training may be quantised to some int8 value during inference. Going one layer forward in Figure 6, the weights, biases, inputs, activations, and so on of the decoder that decodes a latent representation back into an image are also quantised. Using the same example, a fp16 weight of some layer of the decoder may be quantised into int8 during inference. More generally, every numerical value that is in some way involved in the implementation and running of the AI compression pipeline on hardware will at some point be quantised according to some location and scale (and or other) quantisation parameters, which may be learned. In each case, the learned quantisation parameters are each unique. A complete compression pipeline is accordingly defined by a large number of unique and independent quantization parameters. When implementing such a pipeline, it means that each binning operation during quantisation is unique and must be defined independently of the others for example as functions, classes, hooks, and so on. This not only results in increased implementation code complexity, but also higher memory usage, and potentially higher error propagation through the pipeline during inference when a given learned bin size at a later layer learned during high precision floating point training, turns out to cause values to be distributed very differently during lower precision, quantized inference. The inventors have realized that this problem can be overcome by re-using quantisation parameters, learned or predetermined, from different parts of the pipeline. For example, this may comprise learning a set of quantisation parameters associated with one or more steps, operations and / or components of the compression pipeline, and then re-using those parameters to perform quantisation (i.e. bin a higher precision value into a lower precision data type) elsewhere in the network on similar steps, operations and / or components. Effectively, this limits the number of different quantisation parameters that are available to the pipeline during inference. An advantage that the inventors have identified is that re-using quantisation parameters associated in some way within the networks with one type of operation (e.g. addition, subtraction, multiplication, and so on) to quantise parameters that are involved in some way with that type of operation elsewhere in the network, results in improved performance (e.g. better compression rates, and fewer artefacts or errors) during inference. Accordingly, quantisation parameters may be stored in memory in association with an operation which is intended to be performed on any data objects quantised by those parameters. For example, if a residual estimation involves performing a subtraction and the input to that residual estimation is quantised using one or more quantisation parameters, then if a subtraction operation is to be performed to some other quantised data, those same quantisation parameters may be re-used. In more practical terms, if a pipeline is implemented in inference in PyTorch, one or more quantisation functions or objects that have unique quantisation parameter variables or values may be replaced with a common function or object that uses common quantisation parameter variables. Consider for example the following rounding quantisation function Q, and a simple toy neural network made up of a single activation layer sandwiched between two convolution layers. Here the rounding function Q may be defined as: (^^) ^^( ^^) = round + ^^ ^^ r is the input, S is the scale parameter, and Z is the zero point or location parameter, which may respectively be defined as: ^^ − ^^ ^^ = ^^^^− ^^^^ (^^) ^^ = − − ^^ ^^^^ Here, S is the ratio of the input range to the output range where the interval [ ^^, ^^] is the clipping ranges of the input, i.e. the boundaries of permissible inputs, and [ ^^^^, ^^^^] is the range in quantized output space that it is mapped to. For the specific 8-bit quantization example, the output range may defined as ^^^^− ^^^^≤ 28− 1. The location parameter Z acts as a bias to ensure that a 0 in input space maps to a 0 in the quantised space. In this toy example, the weights of the convolution and activation layers are learned in fp32 and are then quantised into int8 space. This quantisation of the learned model weights may be considered as a “static” quantisation operation in the sense that, once quantisation of the weights has been performed after training, the weights remains static (i.e. don’t change) thereafter. This type of quantisation typically reduces overall model size and thus reduces the memory bottleneck of loading the model into memory. In contrast, the output values from the convolution layer and the activation layer can not be pre-quantised as they only exist as data flows through the network (i.e. they depend on input) and are accordingly only able to be quantised after they have been calculated and output by the layer in question, using quantisation parameters that may change based on the input. In a known practical implementation of neural networks using PyTorch, observer functions or modules may be used to collect statistics on the input variables being quantised in order to dynamically calculate what quantisation parameters might best be used based on some predetermined scheme. This is considered to be “on-the-fly” (or dynamic) quantisation. Estimating the quantisation parameters to be used based on a given input and quantising the data or activations as it passes between layers “on the fly” is computationally intensive and adds to run time. Instead, the present approach envisages pre-calculating a set of different quantisation parameters and performing quantisation of outputs of layers during inference using one or more of those parameters either in a hardcoded manner, or based on a criteria associated with the layer, the input, and / or the output, and so on. For example, this might be based on what operation is to be subsequently performed on the value being quantised, on some property of the input that is expected, or on some other meta information such as layer position in the network, and so on. These pre-calculated quantisation parameters may then be stored and re-used for quantisation on any other data flows in the network. This partially mitigates the increase in run time of dynamic quantisation as the quantisation parameters need not be calculated on the fly, while at the same time it avoids a drop in performance because the stored quantisation parameters provide some degree of freedom and dynamic selection of quantisation parameter from the set that has been precalculated. In more general terms, this approach front-loads the on-the-fly quantisation computation to the pre-calculation of the set of different quantisation parameters, which can then be re-used elsewhere in the network as applicable. Quantisation of the outputs flowing between layers is thereby simplified to deciding which of the quantisation parameters from the pre-calculated set are to be used. Consider for example the calculation of flow in a flow residual model such as that shown in Figure 3. The flow encoder 207 may involve performing one or more concatenation operations to data flowing between layers (for example, if there are any skip connections, an output of one layer may be concatenated with the data obtained through the skip connection). The data being output by the layer is quantised with a location and scale parameter before the concatenation is performed. This location and scale parameter may be learned during training. For the sake of argument, assume that the residual encoder 211 also has skip connections and involves concatenating the outputs of a layer with the data flowing through the skip connections. Applying the present concept to this scenario, instead of learning different, unique quantisation parameters for both the flow encoder 207 and the residual encoder 211, and using them both separately during inference, we instead simply re-use the quantisation parameters learned for the flow encoder 207 for the residual encoder 211. The same approach can be applied to anywhere where quantisation of the data as it flows between layers or components of the network needs to be applied. For example, the quantisation of the latent representation output by encoder 203 before it is entropy encoded into a bitstream is performed using one or more quantisation parameters in module 201 in Figure 3. The operations involved in entropy encoding (e.g. addition, multiplication, division, and / or subtraction) also occur when the flow is entropy encoded into bitstream 208, and when the residual is encode into bitstream 212. The latent representation quantisation parameters can thus be re-used when quantisating the output of the flow encoder 207 and the residual encoder 211. In this way, a set of different quantisation parameters can be built up, which can be re-used in different parts of the compression pipeline. Figure 16a illustratively shows a flow residual compression pipeline corresponding to that of Figure 3. The same reference numerals are used as in Figure 3 for ease of reference. Figure 16a additionally shows quantisation operations 1600a, 1600b, 1600c, 1601a, 1601b. In Figure 16a, a set of different quantisation parameters have been pre-calculated: { ^^1, ^^1, ^^2, ^^2, ^^3, ^^3} In the example of Figure 16a, ^^1, ^^1are used to quantise the latent representation output by the encoder 203 and are then stored, and re-used to quantise the output of the flow encoder 207 and also re-used quantise the output of the residual encoder. This may be handcrafted, in the sense that, through experimentation, we can predetermine which quantisation parameters from our set we wish to use for quantisation and define these as part of the architecture, or they may be learned. More specifically, it may be determined that the distributions of the latent representation, the flow, and the residuals are generally distributed in a similar manner and we would accordingly like them to behave similarly when we entropy encode them. If they have different quantisation parameters, their behaviour in lower precision, quantisation space may diverge. Accordingly, choosing to use the same quantisation parameters 1600a, 1600b, 1600c for all three helps to mitigate this problem and avoids introducing errors or artefacts into the pipeline. At the same time, as these quantisation parameters have been pre-calculated, we do not suffer from an increase in run time. Alternatively, rather than pre-calculating all of the quantisation parameters, it is also envisaged that the synchronisation of the quantisation parameters across different parts or elements of the architecture may be dynamic in the sense that they may be calculated on the fly for one part or element such (for example using a PyTorch observer object or class if the architecture is implemented in PyTorch), and then the calculated quantisation parameters may then be propagated across the network. This may be done by, for example, storing the on-the-fly calculated parameters for the one part or element in a memory which may be retrieved for the quantisation of some subsequent layer output. The same reasoning applies to the quantisation of data flowing between layers of the encoder and / or decoder networks of the pipeline. Figure 16a shows the examples of the flow encoder 207 and residual encoder 211. In each case, one or more concatenations, multiplications as part of convolutions, and / or some other operation may be performed on the data output by a given layer. For the sake of argument, assume that the first layer of each encoder is a convolution layer and the second layer is an activation layer, ^^2, ^^2are used to quantise the output of the convolution layer of encoder 207, and ^^3, ^^3 used to quantise the output of the activation layer. These parameters are then used to quantise the outputs of the corresponding layers of the residual encoder 211. As above, which quantisation parameters are used for which layers and which encoders and / or decoders may be handcrafted or learned. More generally, the set of quantisation parameters may be stored in memory, e.g. in a look-up table 1602, as shown in Figure 16b, in association with an index (not shown) and / or in association with some meta information about what operation they are associated with in some way, for example at a high level of component generality (e.g. entropy encoding, convolution, activation, and so on) and / or at a low level in terms of what mathematical operation (not shown) is intended to be performed that outputs quantised by those quantisation parameters. Effectively, this approach synchronises quantisation parameters across the compression pipeline so that similarly behaving functions, data, operations, and so on are quantised with synchronized quantisation parameters. When implementing such an architecture in practice, for example in PyTorch, the function 1603 being used to implement quantisation may make a call 1604 to the look-up table to look for a given index and / or meta information to determine which quantisation parameters to retrieve 1605 to use for a given quantisation operation. For example, this may be set using a config file associated with each layer of the network which specifies whether or not the quantisation parameter is to be retrieved from the look-up table, calculated on-the-fly, or estimated using some other approach. As described above, this may be handcrafted (i.e. hard coded), whereby the quantisation parameters are fixed in the architecture of the model and a given quantisation function and its parameters for that layer or data object is hardcoded in the config file associated with it to always use the same approach to retrieving or calculating the quantisation parameters. Alternatively, the config file may be dynamic and change based meta information that changes based some other variable (such as changeable flag associated with the layer through which the data is flowing indicating some operation is to be performed to the output of that layer, or on some property identified associated with the input to that layer, and so on). Contrasting this approach to the observer modules of known implementations that collect statistics of an input, and dynamically calculate quantisation parameters on the fly, the look-up table approach need not perform any additional calculations, or collecting of input statistics, and accordingly runs much faster during inference, while retaining the improved performance that is typically associated with on the fly quantisation. Whilst not shown in Figure 16a, the same approaches of synchronized quantisation parameters may be used on any of the decoders, hyperencoders, hyperdecoders, and so on, and with any of the other components of the compression pipeline as described above in the general section where data is quantised at different stages as it flows through the compression pipeline. It will also be appreciated that this quantisation parameter synchronization concept may be used alone or together with the other concepts described herein. Concept 4a: Micro and macro flows encode side One problem of quantising data from a higher precision data type (such as fp32, fp16 and so on) into a lower precision data type (such as int16, int8, and so on) is that granular information is lost. This has a particularly significant impact on optical flow information where quantisation of the data that makes up a flow information destroys granular flow information which effects the accuracy of the eventual reconstruction of the image. Consider a toy example video stream of a large car moving across a view in the foreground with trees slowly and only slightly moving in the wind in the background. In fp16 precision, the movement of the trees is likely to be correctly picked up at granular or micro level in the flow map indicating differences between two images of the video stream. Assuming fp16 precision can’t be used during inference to calculate flows due to hardware, power, and bandwidth constraints, the input images, the activations / outputs flowing between layers, the estimated flows and residuals and so on are all quantised into int8. The subtle but accurate flow information indicative of the movement of the trees in the video stream are effectively rounded away during quantisation, resulting in the information being destroyed. The reconstructed images of the video stream produced by the int8 quantised compression pipeline in this case may not accurately reconstruct the movement of the trees, which results in artefacts and other errors. The inventors have realized that this problem of granular information destruction while remaining in int8 precision can be partially mitigated by estimating optical flow information at different levels of granularity, each in integer precision. More specifically, optical flow information, for example a flow map between two images or some other representation of optical flow between those images, may be calculated by: splitting the images into macro blocks and micro blocks which are sub blocks of the macro blocks all in the higher precision space, quantising the higher precision (e.g. fp16) macro blocks and sub blocks into a lower precision space (e.g. int8), estimating flow for the macro and the micro blocks in the lower precision. This provides an estimation of larger, non-granular (i.e. macro) flow information, as well as the granular (i.e. micro) flow information all done in lower precision, integer space which can be performed far more efficiently on CPUs, NPUs, TPUs and so on than in higher precision floating point space. In more general terms, this approach enables sub-pixel resolution of flow to be estimated using only int8 data. Figure 17 illustrates an example of a flow module, in this case a network 1700, configured to estimate information indicative of a difference between an image ^^^^−1and an image ^^^^, e.g. flow information. Figure 17 is provided as an example of how such flow information may be calculated between two input or output images, and accordingly which may be used when evaluating a loss function which incorporates a temporal consistency loss term. The flow module may be used in the flow module part 206 of the compression pipeline. An alternative approach to estimating flow is described in Mentzer, F., Agustsson, E., Ballé, J., Minnen, D., Johnston, N., and Toderici, G. (2022, November). Neural video compression using gans for detail synthesis and propagation. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVI (pp. 562-578), which is hereby incorporated by reference. The network 1700 in Figure 17 comprises a set of layers 1701a, 1701b respectively for an image ^^^^−1and an image ^^^^from respective times or positions in a sequence ^^−1 and ^^ of a sequence of image frames. The set of layers 1701a, 1701b may define one or more convolution operations and / or nonlinear activations in the layers to sequentially downsample the input images to produce a pyramid of feature maps for different levels of coarseness or spatial resolution. This may comprise performing ℎ / 2 ^^ / 2 downsampling in a first layer, ℎ / 4 ^^ / 4 downsampling in a second layer ℎ / 8 ^^ / 8 downsampling in a third layer, ℎ / 16 ^^ / 16 downsampling in a fourth layer, ℎ / 32 ^^ / 32 downsampling in a fifth layer, ℎ / 64 ^^ / 64 downsampling in a sixth layer, and so on. It will of course be appreciated that these downsampling operations and levels of coarseness or spatial resolution of a pyramid feature map are exemplary only and others levels are also envisaged. With the downsampling operations performed and the corresponding pyramid of feature maps generated, a first cost volume 1702 is calculated at the most course level between the pixels of the first image ^^^^−1and the corresponding pixels of in the second image ^^^^. Cost volumes define the matchmaking cost of matching the pixels in the initial image with the pixels in the later image. That is, the closeness of each pixel, or a subset of all pixels, in the initial image to one or more pixels in the later image is determined with a closeness metric such as a vector or dot product, a cosine similarity, a mean absolute difference, or some other closeness metric. This metric may be calculated against all pixels in the later image, or only for pixels in a predetermined search radius such as a 1-10 pixel radius (preferably a 1, 2, 3, or 4 pixel radius), or some other radius as described in connection with concept 4 below, around the pixel coordinate corresponding to the pixel against which the closeness metric is being calculated. It is this process which is computationally expensive in floating points space but which can be implemented efficiently in integer or fixed point space. Finally, a first flow 1703 can be estimated from the first cost volume 1702. This may be achieved using, for example a flow extractor network which may comprise a convolutional neural network comprising a plurality of layers trained to output a tensor defining a flow map from the input cost volumes. Other methods of calculating flow information from cost volumes will also be known to the skilled person. The same process is then repeated for the other levels of coarseness to calculate a second cost volume 1704 and second flow 1705, and so on for the cost volumes and flows associated with each of the levels of coarseness have been calculated, up to the final cost volume 1706 and flow 1707. The weights and / or biases of any activation layers in network 1700 (e.g. optionally in the downsampling convolution layers and / or in a flow extractor network that produces flow maps from the cost volumes) are trainable parameters and can accordingly be updated during training either alone, or in an end to end manner with the rest of the compression pipeline. The trainable nature of these parameters provides the network 1700 with flexibility to produce feature maps at each level of spatial resolution (i.e. pyramid feature maps) and / or at the flow outputs that are forced into a distribution that best allows the network to meet its training objective (e.g. better compression, better reconstruction accuracy, more accurate reconstruction of flow, and so on). For example, it allows the network 1700 to produce feature maps that, when cost volumes and / or flow are calculated therefrom, produce cost volumes or flows that are distributed roughly matching the latent representation distribution that would previously have been expected to be output by a dedicated flow encoder module. This effectively allows a dedicated flow encoder to be omitted entirely from the flow compression part of the pipeline. Optionally, for each level of coarseness or resolution, the flow of the previous level or levels of coarseness or resolution may be used to warp 1708, 1709, the feature maps before the cost volume is calculated. This has the effect of artificially reducing the amount of relative movement between the pixels of the t and t - 1 images or feature maps when calculating the cost volumes, reducing flow errors for high movement details. The inventors have realized that removing warping entirely or in some levels of coarseness or resolution can substantially reduce runtime of flow calculation while maintaining good levels of flow accuracy, something which goes against the teachings of known flow estimation literature. As the warping process uses inputs from different levels of coarseness or spatial resolution, the flow estimation output may be upsampled 1710, 1711 first to match the coarseness resolution of the feature map to which the flow is being applied in the warping process. The outputs of the flow module may accordingly be one or more cost volumes or some representation thereof, and / or one or more flows or some representation thereof). Figure 18a illustratively shows how the macro-micro flow concept may be implemented for one or more of the different resolution flow estimations of the flow module of Figure 17. It will be appreciated that the same concept may be implemented in other flow estimation techniques, such as those described in the papers incorporated herein by reference. Like-numbered features correspond to those of Figure 17. The input images ^^^^−1and ^^^^are input in a higher precision data type, e.g. fp32, fp16, int16, and so on, which are then optionally downsampled to various resolutions if the approach to flow estimation shown in Figure 17 is used. At one or more of the different resolutions, the images are divided into respective macro blocks 1800a, 1800b, 1800c, 1800d at the higher precision. Only two macro blocks are shown in Figure 18a but it is envisaged that this number may be any number depending e.g. on the resolution of the images being divided. Each of the macro blocks 1800a, 1800b, 1800c, 1800d is then subdivided further into micro blocks 1801a, 1801b, also at the higher precision. The macro blocks and micro blocks are then quantised into the lower precision space (e.g. int8, int4, and so on) using a quantisation function (such as a rounding function or some other function parameterized with one or more quantisation parameters: ^^^^, ^^^^, ^^^^, ^^^^, where m and n indicate an index indicating which quantisation parameters to use, which may be learned). The cost volumes or other difference metric such as a mean absolute difference, cosine difference and so on between ^^^^−1and ^^^^are then estimated 1702 for the macro blocks and the micro blocks. From these the flow information for each of the micro and macro blocks is estimated 1703. In the example of Figure 18a, there are two macro blocks each divided into nine micro blocks which means flow is estimated for 20 blocks (2+(2*9)). This is all done in the lower precision space which means that not only are the compute and memory overheads, and power draw, much smaller than if the same calculations were performed in the higher resolution space, but the granular flow details are retained within the micro block flow. This approach accordingly facilitates sub pixel resolution without needing to calculate cost volumes or mean absolute differences in floating points space. The numbers of blocks in Figure 18a are wholly illustrative and it is envisaged that other numbers can be used. For example, each macro block might only be divided into four micro blocks, or some other number, and there may be any number of macro blocks, as applicable for a given image resolution. Once the flows are obtained for each of the micro and macro blocks, they are combined 1802 into a single data object, which shifts the lower precision macro and micro information back into higher precision space (e.g. two combining two int8 objects into a fp16 or int16 object using a d-type or some other conversion process), and the output may be optionally upsampled 1710 if the approach of Figure 17 is being followed, before warping is performed in the manner as described above. Warping may be performed in the higher precision space such as fp16 or int16 and may be offloaded from a CPU, TPU or NPU that the other operations are being performed on, to a GPU where available. If the approach of Figure 17 is being followed, the same process may then be repeated for each of the different resolutions of the input image to obtain the cost volumes and flows up to a final flow estimation between the images, and / or a latent representation thereof which may be produced using a dedicated encoding network that receives the final flow as input and outputs a latent representation of that flow, or which may be produced directly by the network of Figure 17. In the extreme case, as the micro-blocks retain granular flow information, the process of Figure 17 may be modified to remove some or all of the different resolutions. That is, the macro and micro blocks are obtained for only the input image and / or for one or a few but not all of the different resolutions shown in Figure 17. The above concepts may also be applied more generally to any flow encoder such as that illustrated generally in Figure 3 where the flow module part 206 comprises a flow encoder neural network 207 that has been trained to produce a latent representation of flow based on inputs ˆ^^−1and ^^^^, which is indicative of inter-frame movement of pixels or pixel groups between ˆ^^−1and ^^^^, and whereby the produced latent representation of the flow is quantised and entropy encoded to compress it and then transmitted as a bitstream 208. On the decode side, the bitstream is entropy decoded and passed to a decoder neural network 209 to produce a reconstructed flow ^^ . Concept 4b: Micro and macro flows decode side Figure 18b schematically illustrates a method 1803 of reconstructing the flow on the decode side from the received latent representation of flow that takes advantage of the same concept of micro and macro flow described above but on the decode side, for example in the place of the flow decoder network 209 of Figure 3. It is envisaged that this method may be used together the micro and macro flow encode method described above, but need not be. More specifically, the final flow output by a flow encoder and / or its latent representation that the flow decoder ultimately receives as input is a data object of certain dimensions that contains flow information. The micro and macro flow approach of concept 4a helps to ensure as much granular flow information as possible is retained but ultimately the decoder is agnostic as to how the flow latent representation it receives was produced. Even more specifically, any flow can be broken down into macro and micro flow on the decode side. In Figure 18b, an image ^^^^and a previously decoded image ˆ^^−1are provided as input to a flow encoder network 207, such as that shown Figure 3, which produces flow information that is subsequently encoded into a latent representation ^^^^ ^^ ^^ ^^, or which produces a latent representation of flow information ^^^^ ^^ ^^ ^^directly, which is entropy encoded into a bitstream 1804. For completeness it is noted that the previously decoded image ˆ^^−1is available on both the encode side (for producing flow information) and the decode side (for producing a warped image for use in residual estimations). The bitstream 1804 is subsequently entropy decoded, producing the reconstruction of the latent representation of the flow information ^ˆ^^^ ^^ ^^ ^^on the decode side. The reconstruction of the latent representation of the flow information ^ˆ^^^ ^^ ^^ ^^is then fed into a first neural network 1805, referred to herein as a macro flow decoder, which produces a reconstruction of macro flow information^ˆ^ ^^ ^^ ^^ ^^. Next, the reconstruction of the latent representation of the flow ^ˆ^^^ ^^ ^^ ^^and the produced reconstruction of the macro flow information^ˆ^ ^^ ^^ ^^ ^^are fed into a second neural network 1806, referred to herein as a micro flow decoder, which produces a reconstruction of micro flow information^ˆ^ ^^ ^^ ^^ ^^. This may comprise concatenating or in some other way combining the the reconstruction of the latent representation of the flow information ^ˆ^^^ ^^ ^^ ^^and the macro flow information^ˆ^ ^^ ^^ ^^ ^^before the combined data is fed into the second neural network 1806. It is noted that the reconstruction of the latent representation of the flow ^ˆ^^^ ^^ ^^ ^^entropy decoded from the bitstream 1804, the weights of the micro and macro flow decoders 1805, 1806, and the output micro and macro flows^ˆ^ ^^ ^^ ^^ ^^and^ˆ^ ^^ ^^ ^^ ^^are (during inference) all defined in fixed point precision, such as int8, int16, and / or int32, preferably entirely in int8, as indicated by block 1807. Effectively, this approach allows flow information during inference to be reconstructed which includes both high level, macro flow information, while at the same time retaining granular low level, micro flow information (effectively sub pixel resolution), all while staying in fixed point precision such as int8 which can be very efficiently implemented on hardware compared floating point precision implementations allowing inference to be performed with reduced compute, power and memory overheads. Once the micro and macro flows have been reconstructed^ˆ^ ^^ ^^ ^^ ^^and^ˆ^ ^^ ^^ ^^ ^^, they are combined 1808, for example by addition or concatenation into a floating point precision data object, e.g. using d-type conversion, indicated by box 1809, into a combined flow^ˆ^ ^^ ^^ ^^ ^^ ^^ ^^ ^^that retains both high level macro flow information details as well as granular low level micro flow information details. This is then used to warp the previously decoded image ˆ^^−1into a warped image ˆ^^−1, ^^, in floating point precision, which can be passed onto the residual component of the compression pipeline, as shown in e.g. Figure 3. This approach can be generalised beyond micro and macro flows to various levels of flow reconstruction granularity by introducing additional flow decoders into block 1807 and passing the previous level of detail of flow into subsequent into the next level of granularity, in each case together with the latent representation of the flow ^ˆ^^^ ^^ ^^ ^^. An example of this is shown in Figure 18c, where an additional flow decoder 1810 is introduced between the macro and micro levels of granularity, here given the index "medium" to indicate it produces a reconstructed flow^ˆ^ ^^ ^^ ^^ ^^ ^^between the macro and micro levels of granularity, which is combined into ^ˆ^ ^^ ^^ ^^ ^^ ^^ ^^ ^^together with the reconstructed micro and macro flows^ˆ^ ^^ ^^ ^^ ^^and^ˆ^ ^^ ^^ ^^ ^^. It will be appreciated that any number of additional flow decoders may be introduced here to cover any level of granularity of flow, all implemented in fixed point precision such as int8 precision, this is visualised using the dotted lines. In order for the micro, macro and any granularity of flow decoders to learn to reconstruct the respective macro, micro and other granularity flows respectively, they effectively have to be made quantisation aware, not just in the sense that they become aware that they will be operating in a fixed point precision inference environment generally but also that they are expected to produce the different macro, micro and other granularity flows respectively. In other words, during training the macro flow decoder has to learn to produce macro flows, the micro flow has to learn to produce micro flows, and so on. Simply providing a flow decoder block with multiple decoders and training these using, e.g. a rate distortion loss, will not result in the separate decoders learning to be micro, macro, and so on, decoders that reconstruct the different flow granularities from the flow latent representation ^ˆ^^^ ^^ ^^ ^^. Instead, additional steps are taken during training to encourage the decoders to learn their respective micro and macro roles. One approach is to quantise the outputs of the respective flow decoders into different respective quantisation grids during training in floating point. This is illustrated schematically in Figure 18d, where like-numbered reference numerals refer to like-numbered features. That is, each of the flow decoders 1805, 1810, 1806 operate and output in floating point precision during training and the floating point outputs are then quantised using quantisation operations 1811, 1812, 1813 into fixed point precision onto first, second and third quantisation grids whereby the second quantisation grid is a sub-grid of the first quantisation grid, and, where there are three flow decoders, the third quantisation grid is a sub-grid of the second quantisation grid and so on. This approach effectively results in the macro flow decoder only being able to index in e.g. 8x8 blocks of the flow latent representation, the medium flow decoder only being able to index in sub-blocks within those 8x8 blocks, and the micro flow decoder only being able to index in sub-sub-blocks within the 8x8 sub-blocks, and so on for each layer of flow granularity. In this way, the macro flow decoder learns to produce macro flows from macro blocks of the flow latent representation, the micro flow decoder learns to produce micro flows from sub-blocks of those macro blocks, and so on. When the flow decoders trained in this way are converted into int8 for inference, they continue to produce the macro, micro and other granularity flows. In a specific implementation example, the micro and macro flow decoders 1805 and 1806 may be defined with the following pseudocode: Estimate flows in int8: ^ˆ^ ^^ ^^ ^^ ^^= ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ( ^ˆ^^^ ^^ ^^ ^^) ^ˆ^ ^^ ^^ ^^ ^^= ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ (^ˆ^ ^^ ^^ ^^ ^^, ^ˆ^^^ ^^ ^^ ^^) Convert to fp16 or fp32 and combine flows: ^ˆ^ ^^ ^^ ^^ ^^ ^^ ^^ ^^=^ˆ^ ^^ ^^ ^^ ^^+^ˆ^ ^^ ^^ ^^ ^^Warp previously decoded image: ˆ^^ ^^ ^^ ^^ ^^ ^^, ^^−1= ^^ ^^ ^^ ^^( ˆ^^−1,^ˆ^ ^^ ^^ ^^ ^^ ^^ ^^ ^^) The warped, previously decoded image ˆ^^ ^^ ^^ ^^ ^^ ^^, ^^−1is then passed on to the residual part of the compression pipeline as illustrated in e.g. Figure 3 where the final output image may be reconstructed. During training, where the MacroFlowDecoderNN and MicroFlowDecoderNN operate in floating point precision, e.g. fp16 or fp32, a quantisation step is applied to the outputs of each decoder, for example as defined below: ^ˆ^, ^^ ^^ ^^ ^^ ^^= 8 ∗ ^^ ^^ ^^ ^^ ^^( ^^ ^^ ^^ ^^ ^^ (^ˆ^ ^^ ^^ ^^ ^^, ^^ ^^ ^^ ^^ ^^ / 8),−128, 127) ^ˆ^, ^^ ^^ ^^ ^^ ^^= 8 / 256 ∗ ^^ ^^ ^^ ^^ ^^( ^^ ^^ ^^ ^^ ^^ (^ˆ^ ^^ ^^ ^^ ^^, ^^ ^^ ^^ ^^ ^^ / (8 / 256)),−128, 127) where the quantised macro flow is determined for 8x8 blocks of the flow latent representation pixels rounded to the grid of 256 values of a macro block of int8 space, and the quantised micro flow is determined for 8x8 sub blocks of those macro flow blocks, each also separately rounded to the grid of 256 values of a micro block of int8 space. The macro and micro flow decoders 1805, 1806 are according applied in turn to each block as applicable. For example, if the latent representation is divided into 64 macro blocks, the macro flow decoder will be applied to those 64 macro blocks, and if each macro block is divided into 64 micro blocks (i.e. sub blocks of the macro blocks), the micro flow decoder will be applied to each of the 64 micro blocks of the 64 macro blocks i.e. 4096 micro blocks. It will be appreciated that it padding, upsampling and / or downsampling may be applied to ensure that the inputs to each of the neural networks match the dimensions the neural networks are expecting to receive. After training, it is envisaged that the micro and macro flow decoders in inference will be fed with respective micro and macro blocks having the same dimensions and divisions as the blocks on which the decoders were trained, e.g. 8x8 macro blocks each divided into 8x8 micro blocks. In this way, the macro and micro flow decoders accurately output micro and macro flows in int8, which can accordingly be combined in fixed point precision and used for warping of the previously decoded image, which is sent to the residual part of the compression pipeline to reconstruct the image. It will be appreciated that this macro and micro flow approach can be used together with any of the other concepts described herein, or used in a standalone manner. Concept 5: Quantisation aware loss Training a network to be quantisation aware is a challenging task. Nagel, Markus, et al. "A white paper on neural network quantization." arXiv preprint arXiv:2106.08295 (2021), which is incorporated in its entirety herein by reference, proposes several different approaches to quantisation aware training (QAT) including using straight-through-estimation (STE), batch normalization folding, and weight initialization. The present inventors have realized that these approaches do not perform well in the data compression use case. More specifically and as described above, the typical rate distortion loss function used in compression is ^^ ^^ ^^ ^^ = ^^ + ^^ ∗ ^^, where ^^ is the distortion function, ^^ is a weighting factor, and ^^ is the rate loss. This loss does not directly take the effects of quantisation into account. It follows that the network, using the typical rate distortion loss function, is unable to directly optimize for the quantiastion of the various parameters, variables, inputs and outputs of the network at the same time, at the same time as the compression rate and the image distortion. To address this problem, at least in part, the loss function is modified to include a term which is based on a difference between an output of a layer of one or more of the networks in a first precision and that output quantised to a second precision: ^^ ^^ ^^ ^^ = ^^ + ^^ + ^^( ^^, ^^^^ ^^ ^^ ^^ ^^) where ^^ is the distortion function, ^^ is the rate loss, and ^^( ^^, ^^^^ ^^ ^^ ^^ ^^) is the newly introduced loss term based on a difference between an output ^^ and its quantised version ^^^^ ^^ ^^ ^^ ^^of one or more of the layers of the network or networks. In general terms, the effect of this new loss term can be intuitively understood to push the network away from learning parameters that result in a big difference between the output ^^ and its quantised counterpart ^^^^ ^^ ^^ ^^ ^^as the loss increases whenever this difference is big. As a result, the network instead learns parameters where this difference is small. In practice, this means that the network learns parameters that cause outputs of each of the layers to look and behave similarly to their quantised counterparts, even before the network has been quantised into a lower precision. The loss term may be implemented as a mean squared error between ^^ and its quantised version ^^^^ ^^ ^^ ^^ ^^or it may be some other loss term such as a cosine difference, and so on. Consider the toy example where a network (for example the encoder or decoder network of the compression pipeline described above) is implemented entirely in high precision fp32. The network begins it training. As inputs are convolved by convolution layers and the outputs of those layers are passed through activation layers producing activation outputs, each of these outputs are extracted in the full, high precision fp32 format to give us ^^. They are then quantised into some lower precision fixed point format such as int8. The loss term ^^( ^^, ^^^^ ^^ ^^ ^^ ^^) and any other loss terms present in the loss function are then evaluated to determine a loss, and the loss is backpropagates through the network to update the weights, and this process is repeated for a predetermined number of epochs. Figure 19 illustratively shows a sequence 1900 of convolution layers 1901, 1903 and activation layers 1902, 1904 from a neural network, such as the encoder or decoder networks of an AI compression pipeline. The dotted lines indicate the networks may have any number of such layers and / or other layers not shown. The first convolution layer 1901 receives input data, to which a convolution is applied, the output from 1905 from that layer is passed to an activation layer 1902 whose output 1906 in turn is passed to another convolution layer 1903 whose output 1908 is passed to another activation layer 1904 whose output 1909 is passed to the next layer, and so on. The outputs of each of the layers are envisaged to be floating point precision data such as fp32 during training of the network to which the sequence 1900 of layers belongs. In order to allow estimate the loss term ^^( ^^, ^^^^ ^^ ^^ ^^ ^^), one or both of the activation ouputs 1906, 1909 are copied 1910, 1911 from the data flow between layers, e.g. using an observer function when implemented in PyTorch in the floating point precision, quantised 1912, 1913 into a lower precision fixed point, integer space such as int8 using one or more quantisation parameters, which may be learned, providing the activation values ^^ and their quantised versions ^^^^ ^^ ^^ ^^ ^^. The mean squared error (MSE) between these values are then calculated to provide the contribution to the loss function 1914 from the effect of quantising the activations. The loss term may be evaluated using the activation outputs and their quantised versions on a per channel basis or across all channels together. Whilst not shown in Figure 19, this process may also or alternatively be applied to the outputs of the convolution layers 1901, 1903 to provide a contribution to the loss from the effect of quantising the outputs of the convolution layers. Finally, whilst not shown in Figure 19, the same process may also be applied to provide a contribution to the loss from the effect of quantising the weights of the layers, for example between training epochs. However, the inventors have counter-intuitively found that that applying this method to the weights of layers, as well as to the data flows between layers results in worse performance than when it is applied only to said data flows. Effectively, the inventors have realised that not including in the quantisation loss term any dependency on the weights’ quantisation results in unexpectedly improved performance compared to including in the quantisation loss term dependency on both the weights’ quantisation and the layer outputs’ quantisation. It is hypothesized that the quantisation of the outputs of the layers e.g. from the activation layers and / or convolution layers has a more significant effect on eventual network performance than the quantisation of the weights - something which is contrary to the general focus in the field of machine learning on quantisation of weights. This poor performance is is in part because the weights generally comprise larger values than the activations. Accordingly, quantisation errors and thus loss based on these larger values will drown out any signal from a quantisation loss term based only on the outputs of the layers. Further, the inventors have found that, even with a suitable regularisation term, including a quantisation loss term that depends on the quantisation of the weights still results in a trained network that underperforms one which was trained using a quantisation loss term that omits any dependency on the quantisation of the weights from the quantisation loss term.

Claims

CLAIMS 1. A method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first neural network to produce a latent representation; decoding the latent representation using a second neural network to produce an output image, wherein the output image is an approximation of the input image; evaluating a function based on a difference between the output image and the input image; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps using a first set of input images to produce a first trained neural network and a second trained neural network, wherein encoding the input image using the first neural network comprises at least one of clamping an output of at least one layer of the first neural network at or above a first value or clamping an output of at least one layer of the first neural network below a second value.

2. The method of claim 1, wherein encoding the input image using the first neural network comprises clamping the output of the at least one layer of the first neural network between the first value and the second value.

3. The method of claim 1 or 2, wherein decoding the latent representation using a second neural network comprises clamping at least one output of at least one layer of the second neural network below a first value and / or above the second value.

4. The method of claim 3, wherein one or more activation layers of the first or second neural network are defined by a hardtanh activation function.

5. The method of claim 4, wherein the hardtanh activation function has a minimum value corresponding to the first value, and wherein the hardtanh activation function has a maximum value corresponding to the second value.

6. The method of claim 3, wherein one or more activation layers of the first or second neural network are defined by a ReLu activation function and wherein said clamping comprises modifying the at least one output of the ReLu activation function to be at or below the second value.

7. The method of any preceding claim, wherein said encoding the input image using the first neural network comprises clamping each output of each layer of the first neural network below a first value and / or above a second value.

8. The method of claim 2, wherein said decoding the latent representation using the second neural network comprises clamping each output of each layer of the second neural network below a first value and / or above a second value.

9. The method of any preceding claim, wherein encoding the input image using the first neural network or decoding the latent representation with the second neural network comprises adding first noise to the output of at least one layer of the first or second neural network.

10. The method of claim 19, comprising adding second noise, different to the first noise, to the output of at least one other layer of the first or second neural network.

11. The method of claim 9 or 10, wherein encoding the input image using the first neural network or decoding the latent representation with the second neural network comprises adding noise to the output of each layer of the first and second neural network.

12. The method of any preceding claim, wherein said encoding comprises applying a predetermined number of convolution layers and activation layers of the first neural network to the input image, and wherein said clamping is performed to the output of one or more of the activation layers.

13. A method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first neural network to produce a latent representation; decoding the latent representation using a second neural network to produce an output image, wherein the output image is an approximation of the input image; evaluating a function based on a difference between the output image and the inputimage; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps using a first set of input images to produce a first trained neural network and a second trained neural network, wherein decoding the latent representation using a second neural network comprises at least one of clamping an output of at least one layer of the first neural network at or above a first value or clamping an output of at least one layer of the first neural network below a second value.

14. The method of claim 13, wherein one or more activation layers of the second neural network are defined by a hardtanh activation function.

15. The method of claim 14, wherein the hardtanh activation function has a minimum value corresponding to the first value, and wherein the hardtanh activation function has a maximum value corresponding to the second value.

16. The method of any of claims 13 to 15, wherein said decoding comprises applying a predetermined number of convolution layers and activation layers of the second neural network to the latent representation, and wherein said clamping is performed to the output of one or more of the activation layers.

17. The method of any preceding claim, wherein encoding the input image using the first neural network or decoding the latent representation with the second neural network comprises adding first noise to the output of at least one layer of the first or second neural network.

18. The method of claim 17, comprising adding second noise, different to the first noise, to the output of at least one other layer of the first or second neural network.

19. A method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first trained neural network to produce a latent representation; transmitting the latent representation to a second computer system; and decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image, wherein encoding and / or decoding the input image using the first neural network comprises clamping at least one output of at least one layer of the first and / or neural network below a first value and / or above a second value.

20. A method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first neural network to produce a latent representation;decoding the latent representation using a second neural network to produce an output image, wherein the output image is an approximation of the input image; evaluating a function based on a difference between the output image and the input image; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps using a first set of input images to produce a first trained neural network and a second trained neural network, wherein encoding the input image using the first neural network or decoding the latent representation with the second neural network comprises adding first noise to the output of at least one layer of the first or second neural network.

21. The method of claim 20, comprising adding second noise, different to the first noise, to the output of at least one other layer of the first or second neural network.

22. The method of claim 20 or 21, wherein said encoding comprises applying a predetermined number of convolution layers and activation layers of the first neural network to the input image, wherein the noise is added to the output of one or more of the activation layers.

23. The method of any of claims 20 to 22, wherein said decoding comprises applying a predetermined number of convolution layers and activation layers of the second neural network to the latent representation, wherein the noise is added to the output of one or more of the activation layers.

24. The method of any of claims 20 to 23, wherein encoding the input image using the first neural network or decoding the latent representation with the second neural network comprises adding noise to the output of each layer of the first or second neural network.

25. The method of any of claims 20 to 24, wherein the first noise comprises Gaussian or Laplacian noise.

26. The method of any of claims 20 to 24, wherein the first noise comprises uniform noise.

27. The method of any of claims 20 to 26, wherein, during said repeating of the steps, the first noise added in a first step is different to the first noise added in a second step.

28. The method of any of claims 20 to 27 wherein the first noise and / or the second noise are based on a property of the output of the at least one layer from a previous step during said repeating of the steps.

29. A data processing system configured to perform the method of any one of claims 20 to 28.

30. A method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first neural network to produce a latent representation; transmitting the latent representation to a second computer system; decoding the latent representation using a second neural network to produce an output image, wherein the output image is an approximation of the input image,wherein said encoding and / or decoding comprises performing a quantisation operation on an output of one layer of the first or second neural network using one or more quantisation parameters to produce a quantized output of the one layer; and wherein the method comprises using the one or more quantisation parameters to perform a quantisation operation on an output of a different layer of the first or second neural network to produce a quantized output of the different layer.

31. The method of claim 29, wherein the one or more quantisation parameters comprise a scale parameter and / or a location parameter.

32. The method of claim 29, wherein the one or more quantisation parameters comprise a quantisation bin size value and / or a zero point value.

33. The method of claim 31 or 32, wherein the quantisation operation comprises a rounding operation.

34. The method of any preceding claim, wherein said encoding and / or decoding comprises processing the quantized output of the one layer by performing one or more of an addition, subtraction, multiplication, and / or concatenation operations using the quantized output of the one layer.

35. The method of claim 34, wherein the one or more quantisation parameters used to perform the quantisation operation on the output of the different layer are based on the processing type performed to the quantized output of the one layer.

36. The method of claim 35, comprising storing the one or more quantisation parameters used to perform the quantisation operation on the output of the one layer in association with the processing type performed on said output.

37. The method of claim 36, comprising identifying a processing type to be performed on the output of the different layer, retrieving the stored one or more quantisation parameters based on the identified processing type, and performing said quantisation operation on the output of the different layer using the retrieved one or more quantisation parameters.

38. The method of claim 36, further comprising the steps of: performing a quantisation operation on the latent representation using one or more latent quantisation parameters; encoding the quantized latent representation using a third neural network to produce a hyper- latent representation; transmitting the hyper-latent representation to the second computer system; and decoding the quantized hyper-latent representation using a fourth trained neural network to obtain the latent representation, processing the quantized latent representation to obtain the one or more latent quantisation parameters; comparing the one or more latent quantisation parameters to the stored one or more quantisation parameters, retrieving the stored one or more quantisation parameters based on a result of said comparing; and performing said quantisation operation on the output of the different layer using the retrieved one or more quantisation parameters.

39. The method of any of claims 36 to 38, wherein said storing is performed at the first computer system and at the second computer system before receiving, encoding, transmitting and decoding.

40. The method of any preceding claim, wherein the quantized output of the one layer and / or the different layer comprises integer type data.

41. The method of claim 40, wherein a result of performing said processing to said quantized output of the one layer and / or the different layer comprises integer type data.

42. The method of any of claims 30 to 34, wherein the one or more quantisation parameters used to perform a quantisation operation on the output of the different are based on a position of the one layer in the first or second neural network.

43. A method for lossy image or video encoding and transmission, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first neural network to produce a latent representation; and transmitting the latent representation, wherein said encoding comprises performing a quantisation operation on an output of one layer of the first neural network using one or more quantisation parameters to produce a quantized output of the one layer; and wherein the method comprises using the one or more quantisation parameters to perform a quantisation operation on an output of a different layer of the first neural network to produce a quantized output of the different layer.

44. The method of claim 43, wherein said encoding comprises processing the quantized output of the one layer by performing one or more of an addition, subtraction, multiplication, and / or concatenation operations using the quantized output of the one layer.

45. The method of claim 44, comprising storing the one or more quantisation parameters used to perform the quantisation operation on the output of the one layer in association with the processing type performed on said output.

46. The method of claim 44, comprising identifying a processing type to be performed on the output of the different layer, retrieving the stored one or more quantisation parameters based on the identified processing type, and performing said quantisation operation on the output of the different layer using the retrieved one or more quantisation parameters.

47. A method for lossy image or video receipt and decoding, the method comprising the steps of: receiving a latent representation transmitted by a first computer system at a second computer system, the latent representation corresponding to an input image; and decoding the latent representation using a second neural network to produce an output image, wherein the output image is an approximation of the input image wherein said decoding comprises performing a quantisation operation on an output of one layer of the second neural network using one or more quantisation parameters to produce a quantized output of the one layer; and wherein the method comprises using the one or more quantisation parameters to performa quantisation operation on an output of a different layer of the second neural network to produce a quantized output of the different layer.

48. The method of claim 47, wherein said decoding comprises processing the quantized output of the one layer by performing one or more of an addition, subtraction, multiplication, and / or concatenation operations using the quantized output of the one layer.

49. The method of claim 48, comprising storing the one or more quantisation parameters used to perform the quantisation operation on the output of the one layer in association with the processing type performed on said output.

50. The method of claim 49, comprising identifying a processing type to be performed on the output of the different layer, retrieving the stored one or more quantisation parameters based on the identified processing type, and performing said quantisation operation on the output of the different layer using the retrieved one or more quantisation parameters.

51. A method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image and a previously decoded image at a first computer system; estimating a first block and a second block of optical flow information using a first neural network, the first block of optical flow information being indicative of a difference between a block of the previously decoded image and a block of the input image, and the second block of optical flow information being a sub-block of the first block; transmitting the optical flow information to a second computer system;decoding the optical flow information using a second neural network; and using the previously decoded image and the decoded optical flow information to produce an output image, wherein the output image is an approximation of the input image.

52. The method of claim 51, comprising performing a quantisation operation on the first and on second block using one or more quantisation parameters to produce quantized first and second blocks of flow information.

53. The method of claim 52, wherein the one or more quantisation parameters used to perform the quantisation operation on the first block are different to the one or more quantisation parameters used to perform the quantisation operation on the second block.

54. The method of claim 52 or 53, wherein the one or more quantisation parameters comprise a scale parameter and / or a location parameter.

55. The method of any of claims 52 to 54, wherein the quantized first and second blocks of flow information comprise fixed point precision type data.

56. The method of claim 55, wherein the integer type data comprises 4-bit, 8-bit or 16-bit integer type data.

57. The method of claim 55 or 56, wherein using the previously decoded image and the decoded optical flow information comprises combining the quantized first block and second block of integer type optical flow information as floating point type data.

58. The method of claim 57, comprising warping the previously decoded image using the floating point type data.

59. The method of claim 55 or 56, wherein using the previously decoded image and the decoded optical flow information comprises combining the quantized first block and second block of integer type optical flow information as 16-bit integer type data.

60. The method of claim 59, comprising warping the previously decoded image using the 16-bit integer type data.

61. The method of claim 58, 59 or 60, wherein said decoding the optical flow information is performed with a central processing unit (CPU), neural processing unit (NPU) or tensor processing unit (TPU), and wherein said warping is performed with a graphical processing unit (GPU).

62. The method of any of claims 51 to 61, wherein the second block of optical flow information provides optical flow information at a quarter pixel resolution of the optical flow information of the first block.

63. A data processing system configured to perform the method of any one of claims 51 to 62.

64. The data processing system of claim 63, comprising at least one of a CPU, NPU or TPU, and comprising a GPU.

65. A method for lossy image or video encoding and transmission, the method comprising the steps of: receiving an input image and a previously decoded image at a first computer system; estimating a first block and a second block of optical flow information using a first neural network, the first block of optical flow information being indicative of a difference between a block of the previously decoded image and a block of the input image, and the second block of optical flow information being a sub-block of the first block; transmitting the optical flow information to a second computer system.

66. A method for lossy image or video receipt and decoding, the method comprising the steps of: receiving a first block and a second block of optical flow information transmitted by a first computer system at a second computer system, the first block of optical flow information being indicative of a difference between a block of a previously decoded image and a block of an input image, and the second block of optical flow information being a sub-block of the first block; and decoding the first and second blocks of optical flow information using a second neural network; and using a previously decoded image and the decoded optical flow information to produce an output image, wherein the output image is an approximation of the input image.

67. A data processing apparatus configured to perform the method of claims 65 or 66.

68. A method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image and a previously decoded image at a first computer system; estimating optical flow information using a first neural network, the optical flow information being indicative of a difference between the previously decoded image and the input image; producing a latent representation of the optical flow information; transmitting the latent representation of the optical flow information to a second computer system; decoding the latent representation of the optical flow information using a second neural network to produce first decoded optical flow information; decoding the latent representation of the optical flow information using a third neural network and the first decoded optical flow information to produce second decoded optical flow information; using the previously decoded image and the first and second decoded optical flow information to produce an output image, wherein the output image is an approximation of the input image.

69. The method of claim 68, wherein the first decoded optical flow information comprises optical flow information associated with a first block of the latent representation, and wherein the second decoded optical flow information comprises optical flow information associated with a sub-block of the first block.

70. The method of claim 68 or 69, comprising splitting the latent representation into a plurality of blocks, producing the first decoded optical flow information with the second neural network using a first block of the plurality of blocks, and producing the second decoded optical flow information with the third neural network using: (i) the first decoded optical flow information and (ii) a sub-block of the first block.

71. The method of claim 79, comprising combining the first decoded optical flow information and the sub-block by concatenation before inputting the concatenated first decoded optical flow information and the sub-block of the first block into the third neural network.

72. The method of any preceding claim, comprising decoding the latent representation of the optical flow information using: a fourth neural network, the first decoded optical flow information, and the second decoded optical flow information to produce third decoded optical flow information.

73. The method of claim 5, comprising splitting the latent representation into a plurality of blocks, producing the first decoded optical flow information with the second neural network using a first block of the plurality of blocks, producing the second decoded optical flow information with the third neural network using the first decoded optical flow information and a sub-block of the first block, and producing the third optical flow information with the fourth neural network using: (i) the first decoded optical flow information, (ii) the second optical flow information, and (iii) a sub-block of the sub-block.

74. Themethod of claim 73, comprising combining the second decoded optical flow information and the sub-sub-block by concatenation before inputting the concatenated second decoded optical flow information and the sub-sub-block into the fourth neural network.

75. The method of any of claims 68 to 74, wherein the first and second decoded optical flow information are defined in fixed point precision.

76. The method of any preceding claim, comprising combining the first and the second decoded optical flow information to produce combined optical flow information; and using the combined optical flow information to produce the output image.

77. the method of claim 76, wherein the combined optical flow information is defined in floating point precision.

78. A method for lossy image or video receipt and decoding, the method comprising the steps of: receiving a latent representation transmitted by a first computer system at a second computer system, the latent representation corresponding to optical flow information being indicative of a difference between a previously decoded image and a input image; and decoding the latent representation using a second neural network to produce first decoded optical flow information; decoding the latent representation using a third neural network and using the first decoded optical flow information to produce second decoded optical flow information, using the previously decoded image and the first and second decoded optical flowinformation to produce an output image, wherein the output image is an approximation of the input image.

79. The method of claim 78, wherein the first decoded optical flow information comprises optical flow information associated with a first block of the latent representation, and wherein the second decoded optical flow information comprises optical flow information associated with a sub-block of the first block.

80. The method of claim 78 or 79, comprising splitting the latent representation into a plurality of blocks, producing the first decoded optical flow information with the second neural network using a first block of the plurality of blocks, and producing the second decoded optical flow information with the third neural network using: (i) the first decoded optical flow information and (ii) a sub-block of the first block.

81. A method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image and a previously decoded image at a first computer system; estimating optical flow information using a first neural network, the optical flow information being indicative of a difference between the previously decoded image and the input image; producing a latent representation of the optical flow information; transmitting the latent representation of the optical flow information to a second computer system;decoding the latent representation of the optical flow information using a second neural network to produce first decoded optical flow information; decoding the latent representation of the optical optical flow information using a third neural network and the first decoded optical flow information to produce second decoded optical flow information; using the previously decoded image and the first and second decoded optical flow information to produce an output image, wherein the output image is an approximation of the input image; evaluating a function based on a difference between the input image and the output image; updating the parameters of the first, second, and third neural networks based on the evaluated function; and repeating the above steps using a first set of input images to produce first, second and third trained neural networks.

82. The method of claim 81, wherein the first decoded optical flow information comprises optical flow information associated with a first block of the latent representation, and wherein the second decoded optical flow information comprises optical flow information associated with a sub-block of the first block.

83. The method of any of claims 68 top 82, comprising splitting the latent representation into a plurality of blocks, producing the first decoded optical flow information with the second neural network using a first block of the plurality of blocks, and producing the second decodedoptical flow information with the third neural network using: (i) the first decoded optical flow information and (ii) a sub-block of the first block.

84. The method of any of claims 81 to 83, wherein the latent representation and the output of the second neural network are defined in floating point precision, and wherein the method comprising performing a quantisation operation on the output of the second neural network to produce the first decoded optical flow information.

85. The method of claim 84, wherein the output of the third neural network is defined in floating point precision, and wherein the method comprises performing a quantisation operation on the output of the third neural network to produce the second optical flow information.

86. A method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first neural network to produce a latent representation; decoding the latent representation using a second neural network to produce an output image, wherein the output image is an approximation of the input image; evaluating a function based on a difference between an output of a layer of the first or second neural network in a first precision and said output quantised to a second precision; updating the parameters of the first neural network and the second neural network based on the evaluated function; andrepeating the above steps using a first set of input images to produce a first trained neural network and a second trained neural network.

87. The method of claim 86, wherein the first precision comprises floating point precision and wherein the second precision comprises integer precision.

88. The method of claim 86 or 87, wherein the layer of the first or second neural network comprises an activation layer, and wherein the output comprises an output of the activation layer.

89. The method of claim 88, comprising quantising the output from the first precision to the second precision.

90. The method of claim 89, wherein said quantising comprises applying a hook to the activation.

91. The method of claim 86 or 87, wherein the layer of the first or second neural network comprises a convolution layer.

92. The method of claim 91, comprising quantising the tensor output by the convolution layer from the first precision to the second precision.

93. The method of claim 92, wherein said quantising comprises applying a hook to the tensor.

94. The method of any preceding claim, wherein the parameters of the first neural network and the second neural network are stored in the first precision for a first predetermined number of said steps, and wherein the method comprises: quantising the parameters of the first neural network and the second neural network into the second precision after said predetermined number of said steps, and performing said steps for a second predetermined number of said steps.

95. The method of claim 94, wherein when performing said second predetermined number of steps, the function is based on a difference between the input image and the output image.

96. The method of of claims 86 to 95, wherein said difference comprises a mean squared error between the output in the first precision and the output quantised to the second precision.

97. The method of any of claims 86 to 96, wherein the output of the layer of the first or second neural network comprises an output of a channel of the output.

98. The method of any claims 86 to 97, comprising: entropy encoding the latent representation into a bitstream, and entropy decoding the bitstream to produce the latent representation, wherein the function is further based on the size of the bitstream.

99. The method of any of claims 86 to 98, wherein the function is further based on a difference between the input image and the output image.

100. A method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprisingthe steps of: receiving an input image at a first computer system; encoding the input image using a first neural network to produce a latent representation; decoding the latent representation using a second neural network to produce an output image, wherein the output image is an approximation of the input image; evaluating a function; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps using a first set of input images to produce a first trained neural network and a second trained neural network wherein the function is based on a difference between one or more of the parameters of the first or second neural networks in a first precision and said one or more parameters quantised to a second precision.

101. The method of claim 100, wherein the first precision comprises floating point precision and wherein the second precision comprises integer precision.

102. The method of claim 100 or 101, comprising, between said steps, quantising the one or more parameters from the first precision into the second precision.

103. The method of claim 102, wherein said quantising comprises, between said steps, applying a hook to one or more parameters.

104. A data processing system configured to perform the method of any one of claims 86 to 103.

Images