Method and data processing system for lossy image or video encoding, transmission and decoding

Abstract

A method for lossy image or video encoding, the method comprising the steps of: performing a first lossy image or video encoding process, the process comprising the steps of: receiving an input image at a first computer system; and encoding the input image using a first trained neural network to produce a latent representation; wherein, at a step of the first lossy image or video encoding process, a result of a determination is retrieved; and based on the result of the determination an additional process is performed, wherein the additional process is configured to, when an output image that is an approximation of the input image is obtained by decoding the latent representation, result in an improved approximation of the output image to the input image.

Description

[0001] Method and data processing system for lossy image or video encoding, transmission and decoding

[0002] BACKGROUND

[0003] This invention relates to a method and system for lossy image or video encoding, transmission and decoding, a method, apparatus, computer program and computer readable storage medium for lossy image or video encoding and transmission, and a method, apparatus, computer program and computer readable storage medium for lossy image or video receipt and decoding.

[0004] There is increasing demand from users of communications networks for images and video content. Demand is increasing not just for the number of images viewed, and for the playing time of video; demand is also increasing for higher resolution content. This places increasing demand on communications networks and increases their energy use because of the larger amount of data being transmitted.

[0005] To reduce the impact of these issues, image and video content is compressed for transmission across the network. The compression of image and video content can be lossless or lossy compression. In lossless compression, the image or video is compressed such that all of the original information in the content can be recovered on decompression. However, when using lossless compression there is a limit to the reduction in data quantity that can be achieved. In lossy compression, some information is lost from the image or video during the compression process. Known compression techniques attempt to minimise the apparent loss of information by the removal of information that results in changes to the decompressed image or video that is not particularly noticeable to the human visual system. JPEG, JPEG2000, AVC, HEVC and AVI are examples of compression processes for image and / or video files.

[0006] In general terms, known lossy image compression techniques use the spatial correlations between pixels in images to remove redundant information during compression. For example, in an image of a blue sky, if a given pixel is blue, there is a high likelihood that the neighbouring pixels, and their neighbouring pixels, and so on, are also blue. There is accordingly no need to retain all the raw pixel data. Instead, we can retain only a subset of the pixels which take up fewer bits and infer the pixel values of the other pixels using information derived from spatial correlations.

[0007] A similar approach is applied in known lossy video compression techniques. That is, spatial correlations between pixels allow the removal of redundant information during compression. However, in video compression, there is further information redundancy in the form of temporal correlations. For example, in a video of an aircraft flying across a blue-sky background, most of the pixels of the blue sky do not change at all between frames of the video. The most of the blue sky pixel data for the frame at position t = 0 in the video is identical to that at position t = 10. Storing this identical, temporally correlated, information is inefficient. Instead, only the blue sky pixel data for a subset of the frames is stored and the rest are inferred from information derived from temporal correlations.

[0008] In the realm of lossy video compression in particular, the removal of redundant temporally correlated information in a video sequence is known inter-frame redundancy. One technique using inter-frame redundancy that is widely used in standard video compression algorithms involves the categorization of video frames into three types: I-frames, P-frames, and B-frames. Each frame type carries distinct properties concerning their encoding and decoding process, playing different roles in achieving high compression ratios while maintaining acceptable visual quality.

[0009] I-frames, or intra-coded frames, serve as the foundation of the video sequence. These frames are self-contained, each one encoding a complete image without reference to any other frame. In terms of compression, 1-frames are least compressed among all frame types, thus carrying the most data. However, their independence provides several benefits, including being the starting point for decompression and enabling random access, crucial for functionalities like fast-forwarding or rewinding the video.

[0010] P-frames, or predictive frames, utilize temporal redundancy in video sequences to achieve greater compression. Instead of encoding an entire image like an I-frame, a P-frame represents the difference between itself and the closest preceding I- or P-frame. The process, known as motion compensation, identifies and encodes only the changes that have occurred, thereby significantly reducing the amount of data transmitted. Nonetheless, P-frames are dependent on previous frames for decoding. Consequently, any error during the encoding or transmission process may propagate to subsequent frames, impacting the overall video quality.

[0011] B-frames, or bidirectionally predictive frames, represent the highest level of compression. Unlike P-frames, B-frames use both the preceding and following frames as references in their encoding process. By predicting motion both forwards and backwards in time, B-frames encode only the differences that cannot be accurately anticipated from the previous and next frames, leading to substantial data reduction. Although this bidirectional prediction makes B-frames more complex to generate and decode, it does not propagate decoding errors since they are not used as references for other frames. Artificial intelligence (Al) based compression techniques achieve compression and decompression of images and videos through the use of trained neural networks in the compression and decompression process. Typically, during training of the neutral networks, the difference between the original image and video and the compressed and decompressed image and video is analyzed and the parameters of the neural networks are modified to reduce this difference while minimizing the data required to transmit the content. However, Al based compression methods may achieve poor compression results in terms of the appearance of the compressed image or video or the amount of information required to be transmitted.

[0012] An example of an Al based image compression process comprising a hyper-network is described in Balle, Johannes, et al. “Variational image compression with a scale hyperprior.” arXiv preprint arXiv: 1802.01436 (2018), which is hereby incorporated by reference.

[0013] An example of an Al based video compression approach is shown in Agustsson, E., Minnen, D., Johnston, N., Balle, J., Hwang, S. J., and Toderici, G. (2020), Scale-space flow for end-to-end optimized video compression. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 8503-8512), which is hereby incorporated by reference.

[0014] A further example of an Al based video compression approach is shown in Mentzer, F., Agustsson, E., Balle, J., Minnen, D., Johnston, N., and Toderici, G. (2022, November). Neural video compression using gans for detail synthesis and propagation. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVI (pp. 562-578), which is hereby incorporated by reference.

[0015] SUMMARY

[0016] According to the present invention there is provided a method for lossy image or video encoding, the method comprising the steps of: performing a Erst lossy image or video encoding process, the process comprising the steps of: receiving an input image at a first computer system; and encoding the input image using a first trained neural network to produce a latent representation; wherein, at a step of the first lossy image or video encoding process, a result of a determination is retrieved; and based on the result of the determination an additional process is performed, wherein the additional process is configured to, when an output image that is an approximation of the input image is obtained by decoding the latent representation, result in an improved approximation of the output image to the input image.

[0017] According to the present invention there is provided a method for lossy image or video decoding, the method comprising the steps of: performing a first lossy image or video decoding process, the process comprising the steps of: receiving a latent representation at a second computer system, wherein the latent representation corresponds to an input image; and decoding the latent representation using a fourth trained neural network to produce an output image, wherein the output image is an approximation of the input image; wherein, at a step of the first lossy image or video decoding process, a result of a determination is retrieved; and based on the result of the determination an additional process is performed, wherein the additional process is configured to result in an improved approximation of the output image to the input image.

[0018] According to the present invention there is provided a method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: performing a first lossy image or video encoding, transmission and decoding process, the process comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first trained neural network to produce a latent representation; transmitting the latent representation to a second computer system; and decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image; wherein, at a step of the first lossy image or video encoding, transmission and decoding process, a determination is performed at at least one of the first computer system and the second computer system; and based on the result of the determination, an additional process is performed, wherein the additional process is configured to result in an improved approximation of the output image to the input image.

[0019] According to the present invention there is provided a method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first neural network to produce a latent representation; decoding the latent representation using a second neural network to produce an output image, wherein the output image is an approximation of the input image; retrieving the result of a determination; based on the result of the determination, performing an additional process, wherein the additional process is configured to result in an improved approximation of the output image to the input image; evaluating a function based on a difference between the output image and the input image; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps using a first set of input images to produce a first trained neural network and a second trained neural network.

[0020] According to the present invention there is provided a method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first trained neural network to produce a latent representation; decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image; retrieving the result of a determination; based on the result of the determination, performing an additional process, wherein the additional process is configured to result in an improved approximation of the output image to the input image; wherein the additional process comprises using a third neural network; evaluating a function based on a difference between the output image and the input image; updating the parameters of the third neural network based on the evaluated function; and repeating the above steps using a first set of input images to produce a third trained neural network.

[0021] According to the present invention there is provided a method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input video at a first computer system; encoding an input frame of the input video using a first trained neural network to produce a latent representation; determining one or more neural network parameters based on one or more frames of the input video; transmitting the latent representation and the one or more neural network parameters to a second computer system; decoding the latent representation using a second trained neural network to produce an output frame, wherein the output frame is an approximation of the input frame; and applying the one or more neural network parameters to the second trained neural network for the decoding of subsequent output frames corresponding to subsequent input frames of the input video.

[0022] According to the present invention there is provided a method of training one or more neural network parameter sets, the one or more neural network parameter sets being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input video at a first computer system; encoding an input frame of the input video using a first neural network to produce a latent representation; determining one or more neural network parameters based on one or more frames of the input video; transmitting the latent representation and the one or more neural network parameters to a second computer system; decoding the latent representation using a second neural network to produce an output frame, wherein the output frame is an approximation of the input frame; and applying the one or more neural network parameters to the second neural network for the decoding of subsequent output frames corresponding to subsequent input frames of the input video; evaluating a function based on a difference between the output frame and the input frame; updating the one or more neural network parameters based on the evaluated function; and repeating the above steps using a first set of input videos to produce one or more trained neural network parameter sets. According to the present invention there is provided a method for lossy image or video encoding and transmission, the method comprising the steps of: receiving an input video at a first computer system; encoding an input frame of the input video using a first trained neural network to produce a latent representation; determining one or more neural network parameters based on one or more frames of the input video; transmitting the latent representation and the one or more neural network parameters.

[0023] According to the present invention there is provided a method for lossy image or video receipt and decoding, the method comprising the steps of: receiving a latent representation and one or more neural network parameters transmitted by a first computer system at a second computer system, the latent representation corresponding to an input frame of an input video; decoding the latent representation using a second trained neural network to produce an output frame, wherein the output frame is an approximation of the input frame; and applying the one or more neural network parameters to the second trained neural network for the decoding of subsequent output frames corresponding to subsequent input frames of the input video.

[0024] According to the present invention there is provided a method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first neural network to produce a latent representation; decoding the latent representation using a second neural network to produce an output image, wherein the output image is an approximation of the input image; evaluating a first function based on a difference between the output image and the input image; updating the parameters of the first neural network and the second neural network based on the evaluated function; repeating the above steps using a first set of input images to produce a first trained neural network and a second trained neural network; sending one or more parameters of the first trained neural network or the second trained neural network to a second computer system; and at the second computer system, modifying one or more parameters of a third trained neural network for use in lossy image or video encoding, transmission and decoding with the one or more parameters of the first trained neural network or the second trained neural network.

[0025] According to the present invention there is provided a method for lossy image or video encoding and transmission, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first trained neural network to produce a latent representation; transmitting the latent representation; sending one or more parameters of a third trained neural network; receiving a second input image at the first computer system; encoding the second input image using the first trained neural network to produce a second latent representation; and transmitting the second latent representation.

[0026] According to the present invention there is provided a method for lossy image or video receipt and decoding, the method comprising the steps of: receiving a latent representation transmitted by a first computer system at a second computer system, the latent representation corresponding to an input image; decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image; receiving one or more parameters of a third trained neural network; modifying one or more parameters of the second trained neural network with the one or more parameters of the third trained neural network; receiving a second latent representation transmitted by a first computer system, the latent representation corresponding to a second input image; and decoding the second latent representation using the second trained neural network to produce a second output image, wherein the second output image is an approximation of the second input image.

[0027] According to the present invention there is provided a method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; selecting a first trained neural network from a first plurality of neural networks; encoding the input image using the first trained neural network to produce a latent representation; transmitting the latent representation to a second computer system; selecting a second trained neural network from a second plurality of neural networks; decoding the latent representation using the second trained neural network to produce an output image, wherein the output image is an approximation of the input image.

[0028] According to the present invention there is provided a method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; selecting a first neural network from a first plurality of neural networks; encoding the input image using the first trained neural network to produce a latent representation; selecting a second neural network from a second plurality of neural networks; decoding the latent representation using the second trained neural network to produce an output image, wherein the output image is an approximation of the input image; evaluating a function based on a difference between the output image and the input image; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps using a first set of input images to produce a first plurality of trained neural network and a second plurality of trained neural networks.

[0029] According to the present invention there is provided a method for lossy image or video encoding and transmission, the method comprising the steps of: receiving an input image at a first computer system; selecting a first trained neural network from a first plurality of neural networks; encoding the input image using the first trained neural network to produce a latent representation; and transmitting the latent representation.

[0030] According to the present invention there is provided a method for lossy image or video receipt and decoding, the method comprising the steps of: receiving a latent representation transmitted by a first computer system at a second computer system, the latent representation corresponding to an input image; selecting a second trained neural network from a second plurality of neural networks; and decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image.

[0031] According to the present invention there is provided a method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first trained neural network to produce a latent representation; transmitting the latent representation to a second computer system; decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image; wherein at least one of the first trained neural network and the second trained neural network comprises a convolution comprising a plurality of values set to zero.

[0032] According to the present invention there is provided a method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first neural network to produce a latent representation; decoding the latent representation using a second neural network to produce an output image, wherein the output image is an approximation of the input image; evaluating a function based on a difference between the output image and the input image; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps using a first set of input images to produce a first trained neural network and a second trained neural network; wherein at least one of the first neural network and the second neural network comprises a convolution comprising a plurality of values set to zero.

[0033] According to the present invention there is provided a method for lossy image or video encoding and transmission, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first trained neural network to produce a latent representation; and transmitting the latent representation; wherein the first trained neural network comprises a convolution comprising a plurality of values set to zero.

[0034] According to the present invention there is provided a method for lossy image or video receipt and decoding, the method comprising the steps of: receiving a latent representation transmitted by a first computer system at a second computer system, the latent representation corresponding to an input image; and decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image; wherein the second trained neural network comprises a convolution comprising a plurality of values set to zero.

[0035] According to the present invention there is provided a method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input frame at a first computer system; encoding the input frame using a first trained neural network to produce a latent representation; determining one or more target values of the latent representation transmitting the latent representation and information relating to the target values to a second computer system; modifying the received latent representation based on the information relating to the target values; and decoding the latent representation using a second trained neural network to produce an output frame, wherein the output frame is an approximation of the input frame.

[0036] According to the present invention there is provided a method for lossy frame or video encoding, transmission and decoding, the method comprising the steps of: receiving an input frame at a first computer system; encoding the input frame using a first trained neural network to produce a latent representation; encoding the latent representation using a second trained neural network to obtain a hyper-latent representation; determining one or more target values of the hyper-latent representation; transmitting the latent representation, the hyper-latent representation and information relating to the target values to a second computer system; decoding the hyper-latent representation using a third trained neural network to obtain a decoded hyper-latent; modifying the decoded hyper-latent based on the information relating to the target values; and decoding the latent representation using a fourth trained neural network and the decoded hyper-latent to produce an output frame, wherein the output frame is an approximation of the input frame.

[0037] According to the present invention there is provided a method for lossy frame or video encoding, transmission and decoding, the method comprising the steps of: receiving an input frame at a first computer system; encoding the input frame using a first trained neural network to produce a latent representation; encoding the latent representation using a second trained neural network to obtain a hyper-latent representation; encoding the latent representation using a third trained neural network to obtain a hyper-hyper-latent representation; determining one or more target values of the hyper-hyper-latent representation; transmitting the latent representation, the hyper-latent representation, the hyper-hyper-representation and information relating to the target values to a second computer system; decoding the hyper-latent representation using a fourth trained neural network to obtain a decoded hyper-hyper-latent; modifying the decoded hyper-hyper-latent based on the information relating to the target values; decoding the hyper- latent representation using a fifth trained neural network and the decoded hyper-hyper-latent to obtain a decoded hyper-latent; and decoding the latent representation using a sixth trained neural network and the decoded hyper-latent to produce an output frame, wherein the output frame is an approximation of the input frame. According to the present invention there is provided a method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input frame at a first computer system; determining a flow between a previous input frame and the input frame; encoding the input frame using a first trained neural network to produce a latent representation; encoding the flow using a second trained neural network to produce a latent flow representation; determining one or more target values of the latent flow representation; transmitting the latent representation, the latent flow representation and information relating to the target values to a second computer system; modifying the received latent flow representation based on the information relating to the target values; decoding the latent flow representation using a third trained neural network to obtain a received flow; and decoding the latent representation using a second trained neural network and the received flow to produce an output frame, wherein the output frame is an approximation of the input frame.

[0038] According to the present invention there is provided a method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input frame at a first computer system; encoding the input frame using a first trained neural network to produce a latent representation; determining one or more target values of the latent representation; decoding the latent representation using a second trained neural network to produce an output frame, wherein the output frame is an approximation of the input frame.; evaluating a function based on a difference between the output frame and the input frame; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps using a first set of input frames to produce a first trained neural network and a second trained neural network.

[0039] According to the present invention there is provided a method for lossy image or video encoding and transmission, the method comprising the steps of: receiving an input frame at a first computer system; encoding the input frame using a first trained neural network to produce a latent representation; determining one or more target values of the latent representation transmitting the latent representation and information relating to the target values.

[0040] According to the present invention there is provided a method for lossy image or video receipt and decoding, the method comprising the steps of: receiving a latent representation and information relating to one or more target values transmitted by a first computer system at a second computer system, the latent representation corresponding to an input frame; modifying the received latent representation based on the information relating to the target values; and decoding the latent representation using a second trained neural network to produce an output frame, wherein the output frame is an approximation of the input frame.

[0041] According to the present invention there is provided a method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input frame at a first computer system; encoding the input frame using a first trained neural network to produce a latent representation; wherein during the encoding step, a correction is applied to at least one of the values associated with the input frame; transmitting the latent representation to a second computer system; and decoding the latent representation using a second trained neural network to produce an output frame, wherein the output frame is an approximation of the input frame. According to the present invention there is provided a method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input frame at a first computer system; encoding the input frame using a first trained neural network to produce a latent representation; transmitting the latent representation to a second computer system; and decoding the latent representation using a second trained neural network to produce an output frame, wherein the output frame is an approximation of the input frame; wherein during the decoding step, a correction is applied to at least one of the values associated with the latent representation.

[0042] According to the present invention there is provided a method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input frame at a first computer system; encoding the input frame using a first trained neural network to produce a latent representation; transmitting the latent representation to a second computer system; and decoding the latent representation using a second trained neural network to produce an output frame, wherein the output frame is an approximation of the input frame wherein a flag from the second computer system is additionally received at the first computer system prior to the encoding step; and based on the flag, at least one of the transformations performed by the first trained neural network is modified before the encoding step is performed.

[0043] According to the present invention there is provided a method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input frame at a first computer system; encoding the input frame using a first trained neural network to produce a latent representation; transmitting the latent representation to a second computer system; and decoding the latent representation using a second trained neural network to produce an output frame, wherein the output frame is an approximation of the input frame wherein a flag from the first computer system is additionally received at the second computer system prior to the decoding step; and based on the flag, at least one of the transformations performed by the second trained neural network is modified before the decoding step is performed.

[0044] According to the present invention there is provided a method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input frame at a first computer system; encoding the input frame using a first trained neural network to produce a latent representation; wherein during the encoding step, a correction is applied to at least one of the values associated with the input frame; transmitting the latent representation to a second computer system; and decoding the latent representation using a second trained neural network to produce an output frame, wherein the output frame is an approximation of the input frame; evaluating a function based on a difference between the output frame and the input frame; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps using a first set of input frames to produce a first trained neural network and a second trained neural network.

[0045] According to the present invention there is provided a method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input frame at a first computer system; encoding the input frame using a first trained neural network to produce a latent representation; transmitting the latent representation to a second computer system; and decoding the latent representation using a second trained neural network to produce an output frame, wherein the output frame is an approximation of the input frame; wherein during the decoding step, a correction is applied to at least one of the values associated with the latent representation; evaluating a function based on a difference between the output frame and the input frame; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps using a first set of input frames to produce a first trained neural network and a second trained neural network.

[0046] According to the present invention there is provided a method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input frame at a first computer system; encoding the input frame using a first trained neural network to produce a latent representation; transmitting the latent representation to a second computer system; and decoding the latent representation using a second trained neural network to produce an output frame, wherein the output frame is an approximation of the input frame wherein a flag from the second computer system is additionally received at the first computer system prior to the encoding step; and based on the flag, at least one of the transformations performed by the first trained neural network is modified before the encoding step is performed; evaluating a function based on a difference between the output frame and the input frame; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps using a first set of input frames to produce a first trained neural network and a second trained neural network. According to the present invention there is provided a method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input frame at a first computer system; encoding the input frame using a first trained neural network to produce a latent representation; transmitting the latent representation to a second computer system; and decoding the latent representation using a second trained neural network to produce an output frame, wherein the output frame is an approximation of the input frame wherein a flag from the first computer system is additionally received at the second computer system prior to the decoding step; and based on the flag, at least one of the transformations performed by the second trained neural network is modified before the decoding step is performed; evaluating a function based on a difference between the output frame and the input frame; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps using a first set of input frames to produce a first trained neural network and a second trained neural network.

[0047] According to the present invention there is provided a method for lossy image or video encoding and transmission, the method comprising the steps of: receiving an input frame at a first computer system; encoding the input frame using a first trained neural network to produce a latent representation; wherein during the encoding step, a correction is applied to at least one of the values associated with the input frame; and transmitting the latent representation.

[0048] According to the present invention there is provided a method for lossy image or video receipt and decoding, the method comprising the steps of: receiving a latent representation at a second computer system; and decoding the latent representation using a second trained neural network to produce an output frame, wherein the output frame is an approximation of the input frame; wherein during the decoding step, a correction is applied to at least one of the values associated with the latent representation.

[0049] According to the present invention there is provided a method for lossy image or video encoding and transmission, the method comprising the steps of: receiving an input frame at a first computer system; and encoding the input frame using a first trained neural network to produce a latent representation; transmitting the latent representation; wherein a flag from a second computer system is additionally received at the first computer system prior to the encoding step; and based on the flag, at least one of the transformations performed by the first trained neural network is modified before the encoding step is performed.

[0050] According to the present invention there is provided a method for lossy image or video receipt and decoding, the method comprising the steps of: receiving the latent representation at a second computer system; and decoding the latent representation using a second trained neural network to produce an output frame, wherein the output frame is an approximation of the input frame; wherein a flag from a first computer system is additionally received at the second computer system prior to the decoding step; and based on the flag, at least one of the transformations performed by the second trained neural network is modified before the decoding step is performed.

[0051] According to the present invention there is provided a data processing apparatus configured to perform any of the methods above. According to the present invention there is provided a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out any of the methods above.

[0052] According to the present invention there is provided a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out any of the methods above.

[0053] BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Aspects of the invention will now be described by way of examples, with reference to the following figures in which: Figure 1 illustrates an example of an image or video compression, transmission and decompression pipeline.

[0055] Figure 2 illustrates a further example of an image or video compression, transmission and decompression pipeline including a hyper-network.

[0056] Figure 3 illustrates an example of a video compression, transmission and decompression pipeline.

[0057] Figure 4 illustrates an example of a video compression, transmission and decompression system. Figure 5 illustrates an example of an image or video compression process comprising an additional process.

[0058] Figure 6 illustrates examples of additional processes.

[0059] Figure 7 illustrates an example of an image or video decompression process comprising an additional process.

[0060] Figure 8 illustrates further examples of additional processes.

[0061] Figure 9 illustrates an example of an image or video compression, transmission and decompression process comprising an additional process.

[0062] Figure 10 illustrates an example of an image or video compression, transmission and decom- pression process comprising obtaining one or more parameters.

[0063] Figure 11 illustrates the transmission of one or more parameters for a input video, a group of frames and a sub-group of frames.

[0064] Figure 12 illustrates an example of a training process for an image or video compression, transmission and decompression process. Figure 13 illustrates an example of component and parameter transmission from different computer systems. Figure 14 illustrates an example of a first and second plurality of trained neural networks.

[0065] Figure 15 illustrates an example of a selection step based on the output of a plurality of trained neural networks.

[0066] Figure 16 illustrates an example of a sparse convolution. Figure 17 illustrates an example of the selection of a convolution pattern.

[0067] Figure 18 illustrates an example of an image or video compression, transmission and decompression process comprising a determination step.

[0068] Figure 19 illustrates a further example of an image or video compression, transmission and decompression process comprising a determination step. Figure 20 illustrates an example of an image or video encoding process comprising a correction step.

[0069] Figure 21 illustrates an example of an image or video decoding process comprising a correction step.

[0070] DETAILED DESCRIPTION OF THE DRAWINGS Compression processes may be applied to any form of information to reduce the amount of data, or file size, required to store that information. Image and video information is an example of information that may be compressed. The Hie size required to store the information, particularly during a compression process when referring to the compressed file, may be referred to as the rate. In general, compression can be lossless or lossy. In both forms of compression, the file size is reduced. However, in lossless compression, no information is lost when the information is compressed and subsequently decompressed. This means that the original file storing the information is fully reconstructed during the decompression process. In contrast to this, in lossy compression information may be lost in the compression and decompression process and the reconstructed file may differ from the original file. Image and video files containing image and video data are common targets for compression.

[0071] In a compression process involving an image, the input image may be represented as x. The data representing the image may be stored in a tensor of dimensions H x W x C, where H represents the height of the image, W represents the width of the image and C represents the number of channels of the image. Each H x W data point of the image represents a pixel value of the image at the corresponding location. Each channel C of the image represents a different component of the image for each pixel which are combined when the image file is displayed by a device. For example, an image file may have 3 channels with the channels representing the red, green and blue component of the image respectively. In this case, the image information is stored in the RGB colour space, which may also be referred to as a model or a format. Other examples of colour spaces or formats include the CMKY and the YCbCr colour models. However, the channels of an image file are not limited to storing colour information and other information may be represented in the channels. As a video may be considered a series of images in sequence, any compression process that may be applied to an image may also be applied to a video. Each image making up a video may be referred to as a frame of the video.

[0072] The output image may differ from the input image and may be represented by x. The difference between the input image and the output image may be referred to as distortion or a difference in image quality. The distortion can be measured using any distortion function which receives the input image and the output image and provides an output which represents the difference between input image and the output image in a numerical way. An example of such a method is using the mean square error (MSE) between the pixels of the input image and the output image, but there are many other ways of measuring distortion, as will be known to the person skilled in the art. The distortion function may comprise a trained neural network.

[0073] Typically, the rate and distortion of a lossy compression process are related. An increase in the rate may result in a decrease in the distortion, and a decrease in the rate may result in an increase in the distortion. Changes to the distortion may affect the rate in a corresponding manner. A relation between these quantities for a given compression technique may be defined by a rate-distortion equation.

[0074] Al based compression processes may involve the use of neural networks. A neural network is an operation that can be performed on an input to produce an output. A neural network may be made up of a plurality of layers. The first layer of the network receives the input. One or more operations may be performed on the input by the layer to produce an output of the first layer. The output of the first layer is then passed to the next layer of the network which may perform one or more operations in a similar way. The output of the final layer is the output of the neural network.

[0075] Each layer of the neural network may be divided into nodes. Each node may receive at least part of the input from the previous layer and provide an output to one or more nodes in a subsequent layer. Each node of a layer may perform the one or more operations of the layer on at least part of the input to the layer. For example, a node may receive an input from one or more nodes of the previous layer. The one or more operations may include a convolution, a weight, a bias and an activation function. Convolution operations are used in convolutional neural networks. When a convolution operation is present, the convolution may be performed across the entire input to a layer. Alternatively, the convolution may be performed on at least part of the input to the layer.

[0076] Each of the one or more operations is defined by one or more parameters that are associated with each operation. For example, the weight operation may be defined by a weight matrix defining the weight to be applied to each input from each node in the previous layer to each node in the present layer. In this example, each of the values in the weight matrix is a parameter of the neural network. The convolution may be defined by a convolution matrix, also known as a kernel. In this example, one or more of the values in the convolution matrix may be a parameter of the neural network. The activation function may also be defined by values which may be parameters of the neural network. The parameters of the network may be varied during training of the network. Other features of the neural network may be predetermined and therefore not varied during training of the network. For example, the number of layers of the network, the number of nodes of the network, the one or more operations performed in each layer and the connections between the layers may be predetermined and therefore fixed before the training process takes place. These features that are predetermined may be referred to as the hyperparameters of the network. These features are sometimes referred to as the architecture of the network.

[0077] To train the neural network, a training set of inputs may be used for which the expected output, sometimes referred to as the ground truth, is known. The initial parameters of the neural network are randomized and the first training input is provided to the network. The output of the network is compared to the expected output, and based on a difference between the output and the expected output the parameters of the network are varied such that the difference between the output of the network and the expected output is reduced. This process is then repeated for a plurality of training inputs to train the network. The difference between the output of the network and the expected output may be defined by a loss function. The result of the loss function may be calculated using the difference between the output of the network and the expected output to determine the gradient of the loss function. Back-propagation of the gradient descent of the loss function may be used to update the parameters of the neural network using the gradients dL / dy of the loss function. A plurality of neural networks in a system may be trained simultaneously through back-propagation of the gradient of the loss function to each network. In the context of image or video compression, this type of system, where simultaneous training with back-propagation through each element or the whole network architecture may be referred to as end-to-end, learned image or video compression. Unlike in traditional compression algorithms that use primarily handcrafted, manually constructed steps, an end-to-end learned system learns itself during training what combination of parameters best achieves the goal of minimising the loss function. This approach is advantageous compared to systems that are not end-to-end learned because an end-to-end system has a greater flexibility to learn weights and parameters that might be counter-intuitive to someone handcrafting features.

[0078] It will be appreciated that the term "training" or "learning" as used herein means the process of optimizing an artificial intelligence or machine learning model, based on a given set of data. This involves iteratively adjusting the parameters of the model to minimize the discrepancy between the model’s predictions and the actual data, represented by the above-described rate-distortion loss function.

[0079] The training process may comprise multiple epochs. An epoch refers to one complete pass of the entire training dataset through the machine learning algorithm. During an epoch, the model’s parameters are updated in an effort to minimize the loss function. It is envisaged that multiple epochs may be used to train a model, with the exact number depending on various factors including the complexity of the model and the diversity of the training data.

[0080] Within each epoch, the training data may be divided into smaller subsets known as batches. The size of a batch, referred to as the batch size, may influence the training process. A smaller batch size can lead to more frequent updates to the model’s parameters, potentially leading to faster convergence to the optimal solution, but at the cost of increased computational resources. Conversely, a larger batch size involves fewer updates, which can be more computationally cllicicnt but might converge slower or even fail to converge to the optimal solution.

[0081] The learnable parameters are updated by a specified amount each time, determined by the learning rate. The learning rate is a hyperparameter that decides how much the parameters are adjusted during the training process. A smaller learning rate implies smaller steps in the parameter space and a potentially more accurate solution, but it may require more epochs to reach that solution. On the other hand, a larger learning rate can expedite the training process but may risk overshooting the optimal solution or causing the training process to diverge.

[0082] The training described herein may involve use of a validation set, which is a portion of the data not used in the initial training, which is used to evaluate the model’s performance and to prevent overfitting. Overfitting occurs when a model learns the training data too well, to the point that it fails to generalize to unseen data. Regularization techniques, such as dropout or L1 / L2 regularization, can also be used to mitigate overfitting.

[0083] It will be appreciated that training a machine learning model is an iterative process that may comprise selection and tuning of various parameters and hyperparameters. As will be appreciated, the specific details, such as hyper parameters and so on, of the training process may vary and it is envisaged that producing a trained model in this way may achieved in a number of different ways with different epochs, batch sizes, learning rates, regularisations, and so on, the details of which are not essential to enabling the advantages and effects of the present disclosure, except where stated otherwise. The point at which an “untrained” neural network is considered be “trained” is envisaged to be case specific and depend on, for example, on a number of epochs, a plateauing of any further learning, or some other metric and is not considered to be essential in achieving the advantages described herein.

[0084] More details of an end-to-end, learned compression process will now be described. It will be appreciated that in some cases, end-to-end, learned compression processes may be combined with one or more components that are handcrafted or trained separately.

[0085] In the case of Al based image or video compression, the loss function may be defined by the rate distortion equation. The rate distortion equation may be represented by Loss = D + A * R, where D is the distortion function, A is a weighting factor, and R is the rate loss. A may be referred to as a lagrange multiplier. The langrange multiplier provides as weight for a particular term of the loss function in relation to each other term and can be used to control which terms of the loss function are favoured when training the network.

[0086] In the case of Al based image or video compression, a training set of input images may be used. An example training set of input images is the KODAK image set (for example at www.cs.albany.edu / xypan / research / snr / Kodak.html). An example training set of input images is the IMAX image set. An example training set of input images is the Imagenet dataset (for example at www.image-net.org / download). An example training set of input images is the CLIC Training Dataset P (“professional”) and M (“mobile”) (for example at http : / / challenge . compression.cc / tasks / ) . An example of an Al based compression, transmission and decompression process 100 is shown in Figure 1. As a first step in the Al based compression process, an input image 5 is provided. The input image 5 is provided to a trained neural network 110 characterized by a function f acting as an encoder. The encoder neural network 110 produces an output based on the input image. This output is referred to as a latent representation of the input image 5. In a second step, the latent representation is quantised in a quantisation process 140 characterised by the operation Q- resulting in a quantized latent. The quantisation process transforms the continuous latent representation into a discrete quantized latent. An example of a quantization process is a rounding function.

[0087] In a third step, the quantized latent is entropy encoded in an entropy encoding process 150 to produce a bitstream 130. The entropy encoding process may be for example, range or arithmetic encoding. In a fourth step, the bitstream 130 may be transmitted across a communication network.

[0088] In a fifth step, the bitstream is entropy decoded in an entropy decoding process 160. The quantized latent is provided to another trained neural network 120 characterized by a function go acting as a decoder, which decodes the quantized latent. The trained neural network 120 produces an output based on the quantized latent. The output may be the output image of the Al based compression process 100. The encoder-decoder system may be referred to as an autoencoder.

[0089] Entropy encoding processes such as range or arithmetic encoding are typically able to losslessly compress given input data up to close to the fundamental entropy limit of that data, as determined by the total entropy of the distribution of that data. Accordingly, one way in which end-to-end, learned compression can minimise the rate loss term of the rate-distortion loss function and thereby increase compression effectiveness is to learn autoencoder parameter values that produce low entropy latent representation distributions. Producing latent representations distributed with as low an entropy as possible allows entropy encoding to compress the latent distributions as close to or to the fundamental entropy limit for that distribution. The lower the entropy of the distribution, the more entropy encoding can losslessly compress it and the lower the amount of data in the corresponding bitstream. In some cases where the latent representation is distributed according to a gaussian or Laplacian distribution, this learning may comprise learning optimal location and scale parameters of the gaussian or Laplacian distributions, in other cases, it allows the learning of more flexible latent representation distributions which can further help to achieve the minimising of the rate-distortion loss function in ways that are not intuitive or possible to do with handcrafted features. Examples of these and other advantages are described in W02021 / 220008A1, which is incorporated in its entirety by reference.

[0090] Something which is closely linked to the entropy encoding of the latent distribution and which accordingly also has an effect on the effectiveness of compression of end-to-end learned approaches is the quantisation step. During inference, a rounding function may be used to quantise a latent representation distribution into bins of given sizes, a rounding function is not differentiable everywhere. Rather, a rounding function is effectively one or more step functions whose gradient is either zero (at the top of the steps) or infinity (at the boundary between steps). Back propagating a gradient of a loss function through a rounding function is challenging. Instead, during training, quantisation by rounding function is replaced by one or more other approaches. For example, the functions of a noise quantisation model are differentiable everywhere and accordingly do allow backpropagation of the gradient of the loss function through the quantisation parts of the end-to-end, learned system. Alternatively, a straight-through estimator (STE) quantisation model or one other quantisation models may be used. It is also envisaged that different quantisation models may be used for during evaluation of different term of the loss function. For example, noise quantisation may used to evaluate the rate or entropy loss term of the rate-distortion loss function while STE quantisation may be used to evaluate the distortion term.

[0091] In a similar manner to how learning parameters top produce certain distributions of the latent representation facilitates achieving better rate loss term minimisation, end-to-end learning of the quantisation process achieves a similar effect. That is, learnable quantisation parameters provide the architecture with a further degree of freedom to achieve the goal of minimising the loss function. For example, parameters corresponding to quantisation bin sizes may be learned which is likely to result in an improved rate-distortion loss outcome compared to approaches using hand-crafted quantisation bin sizes.

[0092] Further, as the rate-distortion loss function constantly has to balance a rate loss term against a distortion loss term, it has been found that the more degrees of freedom the system has during training, the better the architecture is at achieving optimal rate and distortion trade off.

[0093] The system described above may be distributed across multiple locations and / or devices. For example, the encoder 110 may be located on a device such as a laptop computer, desktop computer, smart phone or server. The decoder 120 may be located on a separate device which may be referred to as a recipient device. The system used to encode, transmit and decode the input image 5 to obtain the output image 6 may be referred to as a compression pipeline.

[0094] The Al based compression process may further comprise a hyper-network 105 for the transmission of meta-information that improves the compression process. The hyper-network 105 comprises a trained neural network 115 acting as a hyper-encoder and a trained neural network 125 acting as a hyper-decoder gg. An example of such a system is shown in Figure 2. Components of the system not further discussed may be assumed to be the same as discussed above. The neural network 115 acting as a hyper-decoder receives the latent that is the output of the encoder 110. The hyper-encoder 115 produces an output based on the latent representation that may be referred to as a hyper-latent representation. The hyper-latent is then quantized in a quantization process 145 characterised by Qhto produce a quantized hyper-latent. The quantization process 145 characterised by Qhmay be the same as the quantisation process 140 characterised by Q discussed above.

[0095] In a similar manner as discussed above for the quantized latent, the quantized hyper-latent is then entropy encoded in an entropy encoding process 155 to produce a bitstream 135. The bitstream 135 may be entropy decoded in an entropy decoding process 165 to retrieve the quantized hyper-latent. The quantized hyper-latent is then used as an input to trained neural network 125 acting as a hyper-decoder. However, in contrast to the compression pipeline 100, the output of the hyper-decoder may not be an approximation of the input to the hyper-decoder 115. Instead, the output of the hyper-decoder is used to provide parameters for use in the entropy encoding process 150 and entropy decoding process 160 in the main compression process 100. For example, the output of the hyper-decoder 125 can include one or more of the mean, standard deviation, variance or any other parameter used to describe a probability model for the entropy encoding process 150 and entropy decoding process 160 of the latent representation. In the example shown in Figure 2, only a single entropy decoding process 165 and hyper-decoder 125 is shown for simplicity. However, in practice, as the decompression process usually takes place on a separate device, duplicates of these processes will be present on the device used for encoding to provide the parameters to be used in the entropy encoding process 150.

[0096] Further transformations may be applied to at least one of the latent and the hyper-latent at any stage in the Al based compression process 100. For example, at least one of the latent and the hyper latent may be converted to a residual value before the entropy encoding process 150,155 is performed. The residual value may be determined by subtracting the mean value of the distribution of latents or hyper-latents from each latent or hyper latent. The residual values may also be normalised.

[0097] To perform training of the Al based compression process described above, a training set of input images may be used as described above. During the training process, the parameters of both the encoder 110 and the decoder 120 may be simultaneously updated in each training step. If a hyper- network 105 is also present, the parameters of both the hyper-encoder 115 and the hyper-decoder 125 may additionally be simultaneously updated in each training step. The training process may further include a generative adversarial network (GAN). When applied to an Al based compression process, in addition to the compression pipeline described above, an additional neutral network acting as a discriminator is included in the system. The discriminator receives an input and outputs a score based on the input providing an indication of whether the discriminator considers the input to be ground truth or fake. For example, the indicator may be a score, with a high score associated with a ground truth input and a low score associated with a fake input. For training of a discriminator, a loss function is used that maximizes the difference in the output indication between an input ground truth and input fake.

[0098] When a GAN is incorporated into the training of the compression process, the output image 6 may be provided to the discriminator. The output of the discriminator may then be used in the loss function of the compression process as a measure of the distortion of the compression process. Alternatively, the discriminator may receive both the input image 5 and the output image 6 and the difference in output indication may then be used in the loss function of the compression process as a measure of the distortion of the compression process. Training of the neural network acting as a discriminator and the other neutral networks in the compression process may be performed simultaneously. During use of the trained compression pipeline for the compression and transmission of images or video, the discriminator neural network is removed from the system and the output of the compression pipeline is the output image 6.

[0099] Incorporation of a GAN into the training process may cause the decoder 120 to perform hallucination. Hallucination is the process of adding information in the output image 6 that was not present in the input image 5. In an example, hallucination may add fine detail to the output image 6 that was not present in the input image 5 or received by the decoder 120. The hallucination performed may be based on information in the quantized latent received by decoder 120.

[0100] Details of a video compression process will now be described. As discussed above, a video is made up of a series of images arranged in sequential order. Al based compression process 100 described above may be applied multiple times to perform compression, transmission and decompression of a video. For example, each frame of the video may be compressed, transmitted and decompressed individually. The received frames may then be grouped to obtain the original video.

[0101] The frames in a video may be labelled based on the information from other frames that is used to decode the frame in a video compression, transmission and decompression process. As described above, frames which are decoded using no information from other frames may be referred to as I-frames. Frames which are decoded using information from past frames may be referred to as P-frames. Frames which are decoded using information from past frames and future frames may be referred to as B-frames. Frames may not be encoded and / or decoded in the order that they appear in the video. For example, a frame at a later time step in the video may be decoded before a frame at an earlier time.

[0102] The images represented by each frame of a video may be related. For example, a number of frames in a video may show the same scene. In this case, a number of different parts of the scene may be shown in more than one of the frames. For example, objects or people in a scene may be shown in more than one of the frames. The background of the scene may also be shown in more than one of the frames. If an object or the perspective is in motion in the video, the position of the object or background in one frame may change relative to the position of the object or background in another frame. The transformation of a part of the image from a first position in a first frame to a second position in a second frame may be referred to as flow, warping or motion compensation. The flow may be represented by a vector. One or more flows that represent the transformation of at least part of one frame to another frame may be referred to as a flow map.

[0103] An example Al based video compression, transmission, and decompression process 200 is shown in Figure 3. The process 200 shown in Figure 3 is divided into an I- frame part 201 for decompressing I-frames, and a P-frame part 202 for decompressing P-frames. It will be understood that these divisions into different parts are arbitrary and the process 200 may be also be considered as a single, end-to-end pipeline.

[0104] As described above, 1-frames do not rely on information from other frames so the I-frame part 201 corresponds to the compression, transmission, and decompression process illustrated in Figures 1 or 2. The specific details will not be repeated here but, in summary, an input image XQ is passed into an encoder neural network 203 producing a latent representation which is quantised and entropy encoded into a bitstream 204. The subscript 0 in o indicates the input image corresponds to a frame of a video stream at position t = 0. This may be the first frame of an entire video stream or the first frame of a chunk of a video stream made up of, for example, an I-frame and a plurality of subsequent P-frames and / or B-frames. The bitstream 204 is then entropy decoded and passed into a decoder neural network 205 to reproduce a reconstructed image XQ which in this case is an I-frame. The decoding step may be performed both locally at the same location as where the input image compression occurs as well as at the location where the decompression occurs. This allows the reconstructed image XQ to be available for later use by components of both the encoding and decoding sides of the pipeline.

[0105] In contrast to I-frames, P-frames (and B-frames) do rely on information from other frames. Accordingly, the P-frame part 202 at the encoding side of the pipeline takes as input not only the input image xtthat is to be compressed (corresponding to a frame of a video stream at position t), but also one or more previously reconstructed images xt-i from an earlier frame t-1. As described above, the previously reconstructed xt-l is available at both the encode and decode side of the pipeline and can accordingly be used for various purposes at both the encode and decode sides.

[0106] At the encode side, previously reconstructed images may be used for generating a flow maps containing information indicative of inter-frame movement of pixels between frames. In the example of Figure 3, both the image being compressed xtand the previously reconstructed image from an earlier frame xt- are passed into a flow module part 206 of the pipeline. The flow module part 206 comprises an autoencoder such as that of the autoencoder systems of Figures 1 and 2 but where the encoder neural network 207 has been trained to produce a latent representation of a flow map from inputs xt-l and xt, which is indicative of inter- frame movement of pixels or pixel groups between xt- and xt. The latent representation of the flow map is quantised and entropy encoded to compress it and then transmitted as a bitstream 208. On the decode side, the bitstream is entropy decoded and passed to a decoder neural network 209 to produce a reconstructed flow map f. The reconstructed flow map f is applied to the previously reconstructed image xf_i to generate a warped image xt- ,w. It is envisaged that any suitable warping technique may be used, for example bi-linear or tri-linear warping, as is described in Agustsson, E., Minnen, D., Johnston, N., Balle, J., Hwang, S. J., and Toderici, G. (2020), Scale-space flow for end-to-end optimized video compression. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 8503-8512), which is hereby incorporated by reference. It is further envisaged that a scale-space flow approach as described in the above paper may also optionally be used. The warped image xt- ,wis a prediction of how the previously reconstructed image Xt-i might have changed between frame positions t-1 and t, based on the output flow map produced by the flow module part 206 autoencoder system from the inputs of xtand xt- .

[0107] As with the I-frame, the reconstructed flow map f and corresponding warped image xt- ,wmay be produced both on the encode side and the decode side of the pipeline so they are available for use by other components of the pipeline on both the encode and decode sides.

[0108] In the example of Figure 3, both the image being compressed xtand the xt- ,ware passed into a residual module part 210 of the pipeline. The residual module part 210 comprises an autoencoder system such as that of the autoencoder systems of Figures 1 and 2 but where the encoder neural network 211 has been trained to produce a latent representation of a residual map indicative of differences between the input mage xtand the warped image xt- ,w. The latent representation of the residual map is then quantised and entropy encoded into a bitstream 212 and transmitted. The bitstream 212 is then entropy decoded and passed into a decoder neural network 213 which reconstructs a residual map r from the decoded latent representation. Alternatively, a residual map may first be pre-calculated between xtand the xt-i,wand the pre-calculated residual map may be passed into an autoencoder for compression only. This hand-crafted residual map approach is computationally simpler, but reduces the degrees of freedom with which the architecture may learn weights and parameters to achieve its goal during training of minimising the rate-distortion loss function.

[0109] Finally, on the decode side, the residual map r is applied (e.g. combined by addition, subtraction or a different operation) to the warped image to produce a reconstructed image xtwhich is a reconstruction of image xtand accordingly corresponds to a P-frame at position t in a sequence of frames of a video stream. It will be appreciated that the reconstructed image xtcan then be used to process the next frame. That is, it can be used to compress, transmit and decompress / +i, and so on until an entire video stream or chunk of a video stream has been processed.

[0110] Thus, for a block of video frames comprising an I-frame and n subsequent P-frames, the bitstream may contain (i) a quantised, entropy encoded latent representation of the I-frame image, and (ii) a quantised, entropy encoded latent representation of a flow map and residual map of each P-frame image. For completeness, whilst not illustrated in Figure 3, any of the autoencoder systems of Figure 3 may comprise hyper and hyper-hyper networks such as those described in connection with Figure 2. Accordingly, the bitstream may also contain hyper and hyper-hyper parameters, their latent quantised, entropy encoded latent representations and so on, of those networks as applicable.

[0111] Finally, the above approach may generally also be extended to B-frames, for example as is described in Pourreza, R., and Cohen, T. (2021). Extending neural p-frame codecs for b-frame coding. In Proceedings of the IEEE / CVF International Conference on Computer Vision (pp. 6680-6689) which is hereby incorporated by reference.

[0112] The above-described flow and residual based approach is highly effective at reducing the amount of data that needs to be transmitted because, as long as at least one reconstructed frame (e.g. I-frame xt- ) is available, the encode side only needs to compress and transmit a flow map and a residual map (and any hyper or hyper-hyper parameter information, as applicable) to reconstruct a subsequent frame.

[0113] Figure 4 shows an example of an Al image or video compression process such as that described above in connection with Figures 1-3 implemented in a video streaming system 400. The system 400 comprises a first device 401 and a second device 402. The first and second devices 401, 402 may be user devices such as smartphones, tablets, AR / VR headsets or other portable devices. In contrast to known systems which primarily perform inference on GPUs such as Nvidia A 100, Geforce 3090, Gefore 4090 GPU cards, the system 400 of Figure 4 performs inference on a CPU of the first and second devices respectively. That is, compute for performing both encoding and decoding are performed by the respective CPUs of the first and second devices 401, 402. This places very different power usage, memory and runtime constraints on the implementation of the above methods than when implementing Al-based compression methods on GPUs. In one example, the CPU of first and second devices 401, 402 may comprise a Qualcomm Snapdragon CPU.

[0114] The first device 401 comprises a media capture device 403, such as a camera, arranged to capture a plurality of images, referred to hereafter as a video stream 404, of a scene 404. The video stream 404 is passed to a pre-processing module 406 which splits the video stream into blocks of frames, various frames of which will be designated as I-frames, P-frames, and / or B-frames. The blocks of frames are then compressed by an Al-compression module 407 comprising the encode side of the Al-based video compression pipeline of Figure 3. The output of the Al-compression module is accordingly a bitstream 408a which is transmitted from the first device 401, for example via a communications channel, for example over one or more of a WiFi, 3G, 4G or 5G channel, which may comprise internet or cloud-based 409 communications .

[0115] The second device 402 receives the communicated bitstream 408b which is passed to an Al-decompression module 410 comprising the decode side of the Al-based video compression pipeline of Figure 3. The output of the Al-decompression module 402 is the reconstructed I-frames, P-frames and / or B-frames which are passed to a post-processing module 411 where they can prepared, for example passed into a buffer, in preparation for streaming 412 to and rendering on a display device 413 of the second device 402.

[0116] It is envisaged that the system 400 of Figure 4 may be used for live video streaming at 30fps of a 1080p video stream, which means a cumulative latency of both the encode and decode side is below substantially 50ms, for example substantially 30ms or less. Achieving this level of runtime performance with only CPU compute on user devices presents challenges which are not addressed by known methods and systems or in the wider Al-compression literature.

[0117] For example, execution of different parts of the compression pipeline during inference may be optimized by adjusting the order in which operations are performed using one or more known CPU scheduling methods. Efficient scheduling can allow for operations to be performed in parallel, thereby reducing the total execution time. It is also envisaged that efficient management of memory resources may be implemented, including optimising caching methods such as storing frequently-accessed data in faster memory locations, and memory reuse, which minimizes memory allocation and deallocation operations.

[0118] Additional process

[0119] As discussed above, the Al based compression process may be divided into an encode process and a decode process. Figure 5 shows an example of an Al based encoding process 101 comprising the steps of receiving the input image 5 and encoding the input image 5 using a trained neural network 110 to obtain a latent representation. The latent representation may be quantized in the quantization process 140 as discussed above. The latent representation may then be entropy encoded in the entropy encoding process 150 as discussed above and subsequently transmitted as a bitstream.

[0120] Figure 7 shows an example of a corresponding Al based decoding process 104 comprising the steps of receiving a latent representation corresponding to an input image 5 and decoding the latent using a trained neural network 120 to obtain an output image 5, wherein the output image 6 is an approximation of the input image 5. Figure 9 shows an Al based compression process 100 comprising both encode and decode processes.

[0121] The Al based encoding process 101 and / or the Al based decoding process 104 may comprise two further steps of receiving the result of a determination 102 and performing an additional process 103 based on the result of the determination 102, wherein the additional process 103 is configured to, when an output image 6 that is an approximation of the input image 5 is obtained by decoding the latent representation, improve the approximation of the output image 6 to the input image 5. For example, the distortion of the improved output image 6 with respect to the input image 5 may be lower than the distortion of output image 6 to the input image 5 when the additional process 103 is not performed.

[0122] The step of receiving the result of the determination 102 may be performed between or at the same time as any of the other steps of the Al based encoding process 101 and / or the Al based decoding process 104. For example, the result of the determination 102 may be received before any other step of the Al based encoding process 101 or the Al based decoding process 104 is performed. Alternatively, the result of the determination 102 may be received after each other step of the Al based encoding process 101 or the Al based decoding process 104 has been performed. The additional process 103 may be performed between or at the same time as any of the other steps of the Al based encoding process 101 and / or the Al based decoding process 104 in a similar manner. The result of the determination 102 determines whether the additional process 103 is performed.

[0123] The result of the determination 102 may be received from the same computer system on which the Al based encoding process 101 and / or the Al based decoding process 104 is being performed. In the case, the result of the determination 102 may be referred to as being retrieved from the computer system performing the Al based encoding process 101 or the Al based decoding process 104. Alternatively, the result of the determination 102 may be received from a separate computer system. The separate computer system may be the computer system which will perform the other of the Al based encoding process 101 or the Al based decoding process 104. Alternatively, the separate computer system may not be performing either of the encoding or decoding processes.

[0124] The received result of the determination 102 may be an indicator or flag which indicates whether the additional process 103 should be performed. Based on the indicator or flag, the computer system performing the Al based encoding process 101 and / or the Al based decoding process 104 may perform the additional process.

[0125] Alternatively or additionally, the result of the determination 102 may comprise additional information. The additional information may include information about the Al based encoding process 101 and / or the corresponding Al based decoding process 104. For example, the additional information may include one or more of the runtime of the encoding and / or decoding process, the power consumption of the encoding and / or decoding process, the bitrate of the encoding and decoding process and the distortion of the encode and decode process. For each of these quantities, the determination may be a threshold. This means that for each quantity, the additional process may be performed if the quantity is above or below a predetermined value. For example, if the power consumption of the encoding or decoding process is below a predetermined value, the additional process may be performed.

[0126] Alternatively or additionally, the additional information may include information about the computer system performing the Al based encoding process 101 and / or the corresponding Al based decoding process 104. For example the additional information may include one or more of the compute available at a computer system, the hardware available at a computer system such as the presence of a graphical processing unit (GPU) or a neural processing unit (NPU) and / or the compute available at one or more pieces of hardware available at the computer system. For each of these quantities, the determination may be a threshold. Alternatively or additionally, the determination may be based on the presence of particular hardware at the computer system performing the encoding process 101 and / or the separate computer system performing the corresponding decoding process. For example, if an NPU is present, the result may be that the additional process is performed.

[0127] The received result of the determination may be predetermined. This may mean that the determination may be preset. The received result may also not depend on any information obtained as the Al based encoding process 101 and / or the corresponding Al based decoding process 104 is being performed. This also may mean that the determination does not change during the Al based encoding process 101 and / or the corresponding Al decoding process 104. For example, the result of the determination may be set at the time that the program which may perform the Al based encoding process 101 and / or the corresponding Al based decoding process 104 is installed on the computer system. Alternatively, the result of the determination may be set when the program which may perform the Al based encoding process 101 and / or the corresponding Al based decoding process 104 is transmitted to one or more computer systems for the purpose of installing the program.

[0128] The additional process 103 may comprise the activation of one or more components or sub-components of the Al based encoding process 101 and / or the corresponding Al based decoding process 104 that are not activated when the corresponding result of the determination is not received.

[0129] During the Al based encoding process 101, the additional process 103 may comprise activating at least one additional neural network layer associated with the trained neural network 110. This means that the trained neural network 110 may comprise one or more layers that are optionally activated. In the case where the one or more layers are activated, the latent representation produced by the trained neural network 110 with a particular input image will be different to the latent representation produced when the one or more layers are not activated. In the case where one or more initial layers are the additional layers, the input image will be provided to the subsequent layers when the additional layers are not active. In the case where one or more of the final layers are the additional layers, the latent representation will be obtained from the last layer before the one or more final layers. In the case where one or more intermediate layers are the additional layers, the feature map will be passed from the final layer before the one or more intermediate layers to the first layer after the one or more intermediate layers. An example of the additional process as at least one additional final layer of the network is shown in the top left image of figure 6.

[0130] Alternatively or additionally, the additional process 103 may comprise activating at least one further trained neural network. This means that a further trained neural network may receive the input image and output a further latent representation. The further latent representation may then be encoded into a bitstream and transmitted with the bitstream corresponding to the latent representation. When the additional process comprises activating at least one further trained neural network, one or more of the layers of the further trained neural network may additionally receive the output, for example the feature map, of one or more layers of the trained neural network 110. An example of this arrangement is shown in the top right image of figure 6. The architecture of the trained neural network and the further trained neural network may be the same.

[0131] Alternatively or additionally, the additional process 103 may comprise processing one or more of the tensors used or produced during the Al based encoding process 101. For example, at least one of the input image 5 or the latent representation may be processed. An example of this is shown in the bottom image of figure 6. In this example, the input image 5 undergoes the additional process 103 to produce an intermediate image 5000. The intermediate image 5000 may then be encoded by the trained neural network to obtain a corresponding latent representation. The processing may be performed by an additional trained neural network.

[0132] During the Al based decoding process 104, the additional process 103 may comprise activating at least one additional neural network layer associated with the trained neural network 120. This means that the trained neural network 120 may comprise one or more layers that are optionally activated. In the case where the one or more layers are activated, the output image produced by the trained neural network 110 with a particular latent representation will be different to the output image produced when the one or more layers are not activated. In the case where one or more initial layers are the additional layers, the latent representation will be provided to the subsequent layers when the additional layers are not active. In the case where one or more of the final layers are the additional layers, the output image will be obtained from the last layer before the one or more final layers. In the case where one or more intermediate layers are the additional layers, the feature map will be passed from the final layer before the one or more intermediate layers to the first layer after the one or more intermediate layers. An example of the additional process as at least one additional final layer of the network is shown in the top left image of figure 8.

[0133] Alternatively or additionally, the additional process 103 may comprise activating at least one further trained neural network. This means that a further trained neural network may receive a second latent representation and output a refined output image. The refined output image may be combined with the output image to improve the approximation of the output image. For example, the pixel values of the output image may be averaged with the pixel values of the refined output image. When the additional process comprises activating at least one further trained neural network, one or more of the layers of the further trained neural network may additionally receive the output, for example the feature map, of one or more layers of the trained neural network 120. An example of this arrangement is shown in the top right image of figure 8. The architecture of the trained neural network and the further trained neural network may be the same. The further trained neural network may additionally receive the output image output by the trained neural network 130 as an input. The output image may be input to any layer of the further trained neural network.

[0134] Alternatively or additionally, the additional process 103 may comprise processing one or more of the tensors used or produced during the Al based decoding process 103. For example, at least one of the received latent representation or the output image may be processed. An example of this is shown in the bottom image of figure 8. In this example, the output image 6 undergoes the additional process 103 to produce a improved output image 6000. The processing may be performed by an additional trained neural network. In this case, the additional trained neural network may receive a previously produced output image as an input.

[0135] Flow information may additionally be used in the additional process 103 to improve the approximation of the output image 6 to the input image 5. During the Al based decoding process 104, when the at least one further trained neural network is activated, the output of the at least one further trained neural network may be flow information such as a flow map that may be used to warp a previous frame to obtain a refined output image. Alternatively or additionally, flow information may be obtained from elsewhere, for example from the trained neural network used to decode the latent representation. This flow information may be combined with flow information obtained by the additional process 103 to improve the approximation of the output image 6 to the input image 5.

[0136] The Al based encoding process may be performed on the central processing unit (CPU) of the computer system performing the Al based encoding process 101 and / or the corresponding Al decoding process 104. The additional process 103 may also be performed by the CPU. Alternatively, the additional process 103 may be performed by the GPU or NPU of the computer system performing the Al based encoding process 101 and / or the corresponding Al decoding process 104 if these components are present.

[0137] An Al training process as described above may be used to obtain the Al based compression process comprising retrieving the result of a determination 102 and performing an additional process 103 described above. The basic principles of training an Al based compression process are described above. The evaluated function may additionally include terms based on either or both of a difference between the output image and the input image without the additional process being performed and with the additional process 103 being performed. In the case where the additional process 103 comprises a further neural network, the parameters of the further neural network may be updated in addition to the other neural networks of the Al based compression process. Alternatively, the other neural networks of Al based compression process may be trained first and their parameters fixed for the training of the further neural network.

[0138] Transmitting parameters

[0139] An Al based compression process such as those described above may additionally comprise the step 501 of determining one or more neural network parameters 121 based on one or more frames of the input video 5, for example at the computer system performing the encode process. The one or more neural network parameters 121 are transmitted to the second computer system in addition to the bitstream corresponding to the latent representation. At the computer system performing the decode process, after decoding the latent representation to produce at least one output frame 6, the one or more neural network parameters 121 may be applied to the trained neural network 120 used to decode the latent representation. This process may be referred to as updating the trained neural network 120. Subsequent output frames may be decoded using the updated trained neural network 120. An example of an Al based compression process including these steps is shown in Figure 10. The transmission of the one or more neural network parameters 121 may occur in parallel or after the transmission of at least the first frame of the input video. This means that one or more output frames may have been decoded and displayed at decoding computer system before receipt of the one or more neural network parameters 121.

[0140] The one or more neural network parameters 121 may comprise one or more of neural network weights, convolution or filter parameters or activation function parameters. The one or more neural network parameters 121 may replace one or more of the neural network. Alternatively or additionally, the one or more neural network parameters 121 may comprise perturbations of parameters. This means that the one or more neural network parameters 121 represent a difference between the original parameters and the parameters desired after the update. In this case, the one or more neural network parameters 121 may be applied by adding or combining the parameters with the corresponding parameters of the neural network to be updated.

[0141] The one or more neural network parameters 121 may be compressed before being transmitted to the computer system performing the decode process. The compression of the one or more neural network parameters 121 may be performed by a trained neural network.

[0142] The one or more neural network parameters 121 are determined based on one or more frames of the input video 5. The one or more neural network parameters 121 may be selected from a plurality of predetermined sets of neural network parameters. For example, each of the the frames of the video may be determined to be a particular kind of video content. Examples of video content include animation, sports, screen recording and video conference. Based on the type of content determined, the one or more neural network parameters 121 may be selected such that the compression process of the remaining input video is improved. For example, a particular set of parameters may provide improved compression performance for a video comprising animated content. The type of video content may be indicated by a tag associated with one or more frames of the input video and the one or more neural network parameters 121 may be selected based on the tag.

[0143] Alternatively or additionally, the one or more neural network parameters 121 may be selected based on particular properties of the frames of the input video 5. For example, it may be determined that the frames are one or more of high or low quality, high or low visual noise, high or low numbers of video artifacts, high or low light levels, or high or low motion. Based on this determination, the one or more neural network parameters 121 may be selected such that the compression process of the remaining input video is improved.

[0144] Alternatively or additionally, the one or more neural network parameters 121 may be the output of a trained neural network that receives one or more frames of the input video 5 as an input.

[0145] The one or more neural network parameters 121 may be selected based on all of the frames of the input video 5. In this case, the one or more neural network parameters 121 may be transmitted only once during the encoding and decoding of the video. An example of this process is shown in the top image of figure 11.

[0146] Alternatively, the one or more neural network parameters 121 may be selected based on a group of frames 50 comprising an I-frame and one or more P-frames. This group of frames 50 may be referred to as a GOP. In this case, the one or more neural network parameters 121 corresponding to each GOP may be transmitted during the encoding and decoding process of the previous GOP. An example of this process is shown in the middle image of figure 11. In this case, multiple sets of one or more neural network parameters 121 are transmitted and applied.

[0147] Alternatively, the one or more neural network parameters 121 may be selected based on a sub-group of frames comprising one or more P-frames. In this case, the one or more neural network parameters 121 corresponding to each GOP may be transmitted multiple times during the encoding and decoding process of a GOP. An example of this process is shown in the bottom image of figure 11. In this case, multiple sets of one or more neural network parameters 121 are transmitted and applied.

[0148] One or more of the one or more neural network parameters 121 may additionally be applied to the neural network 110 encoding the input frame to obtain the latent representation. In the case where the one or more neural network parameters 121 are selected from a plurality of predetermined sets of neural network parameters, the predetermined sets may comprise pairs of sets of parameters corresponding to the trained neural network used for encoding and the trained neural network used for decoding respectively.

[0149] An Al training process as described above may be used to obtain the Al based compression process comprising retrieving the result of a determination 102 and performing an additional process 103 described above. The basic principles of training an Al based compression process are described above. However, in this case it may be the one or more neural network parameters 121 that are updated based on the evaluated function. The evaluated function may additionally comprise a term based on the bitrate of the one or more neural network parameters 121. When the one or more neural network parameters 121 are selected from a plurality of sets of one or more neural network parameters 121, the evaluated function may comprise a term based on the bitrate of each of the plurality of sets of one or more neural network parameters 121. Alternatively or additionally, the parameters of the neural network used to encode the input image and / or the neural network used to decode the latent representation may also be updated in the training process.

[0150] Parameter update

[0151] As discussed above, a training process is used to obtain trained neural networks suitable for use in an Al based compression process. A modification to the known training processes will now be discussed in this section.

[0152] The training process to obtain neural networks for use in an Al based compression process requires an image or video data set. However, the image or video data set used for the training process may vary. For example, over time the type of images or videos being sent or received, and therefore being compressed may change. This may result in a previous training process no longer resulting in optimal compression because the image or video data set used for the previous training process no longer corresponds to the current images or videos being sent and received.

[0153] In this case, it may be advantageous for further training and an update to the Al based compression process to be performed. In an example method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprises the steps of receiving an input image at a first computer system, encoding the input image using a first neural network to produce a latent representation, decoding the latent representation using a second neural network to produce an output image, wherein the output image is an approximation of the input image, evaluating a first function based on a difference between the output image and the input image, updating the parameters of the first neural network and the second neural network based on the evaluated function, repeating the above steps using a first set of input images to produce a first trained neural network and a second trained neural network, sending one or more parameters of the first trained neural network or the second trained neural network to a second computer system, and, at the second computer system, modifying one or more parameters of a third trained neural network for use in lossy image or video encoding, transmission and decoding with the one or more parameters of the first trained neural network or the second trained neural network.

[0154] An example of such a method is shown in figure 12. In this example, the first neural network 110 and second neural network 120 undergo a training process in which the first function is evaluated in the evaluation step 170. This process is repeated to obtain trained neural networks. The one or more parameters of the second trained neural network go* may then be transmitted to the second computer system and used to modify the parameters of the third trained neural network 131.

[0155] Through the use of this training method, the lossy image or video encoding, transmission and decoding process in which the third trained neural network is used may be improved. In this example, the modified third trained neural network may be used in either the encode or the decode process. For example, the third trained neural network may be the network used to encode the input image into a latent representation as discussed above. Alternatively, the third trained neural network may be the neural network used to decode a received latent representation to obtain an output image. Both of the encode and decode trained neural networks may be updated during the modification step. Furthermore, if other neural networks are present in the Al based compression process, for example hyper-networks as discussed above, these other neural networks may also be updated during modification step.

[0156] The one or more parameters may be neural network weights. The modified weights may correspond to a particular layer of the neural network. Alternatively or additionally, all of the weights in the third trained neural network may be modified. In this case, the modification of the weights of the third neural network may comprise replacing the previous weights with the received weights. The modification step may therefore comprise modification of the memory or computer storage which contains the parameters of the third neural network where these are stored in memory or longer term storage. This memory or storage location may be in a different location to the computer apparatus on which the third trained neural network is operated as part of the Al based compression process.

[0157] Alternatively or additionally, the one or more parameters may be perturbations of neural network weights. In this case, the modification step comprises perturbing the previous weights of the third trained neural network with the received perturbations. For example, the perturbations may be an addition or subtraction to be made to the previous weights of the third trained neural network to obtain new weights. In this case, the operation to obtain the new weights for the third training neural network may be performed by the same computer apparatus at which the parameters are stored.

[0158] The previous one or more parameters of the third trained neural network may be deleted after the modification step. Alternatively, the previous one or more parameters may be stored at the second computer system after the modification step. For example, if the previous parameters are stored, a further communication may be received at the second computer system indicating to perform a further modification step to revert the parameters of the third trained neural network to the previous values. The new parameters may represent an additional contribution to a database of parameters that may be selected depending on the content of the image or video to be sent or received in the Al based compression process. Previous parameters may be deleted or stored based on the storage requirements at the second computer system. For example, if memory or storage is not available the previous parameters may be discarded.

[0159] In the case where the previous one or more parameters are stored, the previous one or more parameters may be compressed before being stored at the second computer system. Neural network parameter sets, for example neural network weights, may have particular distributions which allow for efficient compression to be used. The previous parameters may also be stored as perturbations of the new parameters.

[0160] Alternatively or additionally, the one or more parameters may be compressed before the step of sending the one or more parameters to the second computer system. If the parameters are not compressed for transmission, the one or more parameters may compressed after receipt at the second computer system. For example, a more efficient but slower compression process may be performed once the parameters have been received such that the parameters can be stored until the modification step is performed. The modification step may be delayed until the Al based compression process comprising the third trained neural network finishes the compression or decompression of a particular image video, or set of videos. Alternatively, the modification step may be delayed until a scheduled update time.

[0161] As discussed above, a neural network comprises a plurality of layers, each layer comprising at least one operation to be applied to input data and a set of weights to be applied to the output of the operation to produce output data that is passed to the next layer. The number of layers, particular functions within a layer, the connections of one layer to another and other features that are not the weights may be referred to as the architecture of a neural network. The arrangement and connections between a plurality of neural networks in a neural network system such as an Al based compression process may also be referred to as the architecture of the system. The connections between different neural networks or neural network components may also be described as a mapping or a flow. At least one of the architecture, flow and mapping of a system for lossy image or video encoding, transmission and decoding comprising the third neural network may not be changed during the modifying step. This means that the arrangement of neural networks within the system and the structure of each network may be retained during the modification step.

[0162] The method may further comprise the step of, prior to the step of receiving the input image at the first computer system, sending one or more components of a system for lossy image or video encoding, transmission and decoding comprising the third neural network to the second computer system. Components of the system may include one or more of the trained neural networks, the entropy encoding components, or any other part of an Al based compression process as described above. Transmission of components may include the transmission of one or more architectures, flow whose, or mappings of a neural network or a neural network system.

[0163] The one or more components of a system for lossy image or video encoding, transmission and decoding comprising the third neural network may be sent from a third computer system. This means that the one or more parameters are received from a different computer system to the computer system which the components are received from. Typically, the flow, architecture or mapping of the Al based compression process may require significantly less storage space than the weights of the neural networks of the the Al based compression process. Therefore, the components may be stored at a system with smaller storage capabilities but which can transmit data more quickly. The weights may be stored at a system which has a larger storage capacity but may transmit data more slowly. When performing a modification step, a user may access the system containing the components to quickly obtain one or more of the architecture, flow or mapping of the Al based compression process. The user may then subsequently access the system containing the parameters such as weights of the neural networks of the Al based compression process. An example of this process is shown in figure 13. The lossy image and video encoding or decoding takes place at the second computer system 2000. The components of the process may be received from the third computer system 3000. The parameters of the process such as weights may be recieved from the first computer system 1000. The one or more parameters of the third trained neural network have been obtained using a training process based on a second set of input images which is different to the first set of input images. This means that the previous parameters of the third trained neural network before the modification step have been obtained using a different training process to that used to obtain the one or more parameters. As discussed above, developments in training processes may be the reason to perform the modification step, for example, the previous one or more parameters of the third trained neural network have been obtained using a training process based on a second function which is different to the first function. The first and the second function may both be loss functions. For example, the first function may comprise a term which is not present in the second loss function. A training process based on such a modified loss function may have the same architecture, mapping or flow as the second loss function. However, the use of a different loss function results in a different set of parameters such as weights. When used in the third trained neural network after the modification step, the modified third neural network may result in better compression performance.

[0164] Alternatively or additionally, the image or video data set used to obtain the one or more parameters may be different. For example, the image or video data set may comprise a different type of image or video, such as webcam, video, cartoon, animated or any other image or video format. The image or video data set may comprise different proportions of these formats. Alternatively or additionally, the image or video data set may have undergone processing, for example an up scaling or a down scaling process. The training process using the image or video data set may have the same architecture mapping or flow as the previous training process. However, the use of a different image for video data set to obtain the one or more parameters may result in better compression performance.

[0165] The one or more parameters used for the modification may be transmitted by the same computer system which is used to perform the encoding process. For example, an encoding process may be a method for lossy image or video encoding and transmission, the method comprising the steps of: receiving an input image at a first computer system, encoding the input image using a first trained neural network to produce a latent representation, transmitting the latent representation; sending one or more parameters of a third trained neural network, receiving a second input image at the first computer system, encoding the second input image using the first trained neural network to produce a second latent representation; and transmitting the second latent representation.

[0166] In a corresponding manner, thee one or more parameters used for the modification may be received by the same computer system which is used to perform the decoding process. For example, a decoding process may be a method for lossy image or video receipt and decoding, the method comprising the steps of: receiving a latent representation transmitted by a first computer system at a second computer system, the latent representation corresponding to an input image; decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image; receiving one or more parameters of a third trained neural network; modifying one or more parameters of the second trained neural network with the one or more parameters of the third trained neural network; receiving a second latent representation transmitted by a first computer system, the latent representation corresponding to a second input image; and decoding the second latent representation using the second trained neural network to produce a second output image, wherein the second output image is an approximation of the second input image. The decoding process may further comprise the step of receiving a flag at the second computer system; and based on the flag, selecting either the unmodified or modified second trained neural network for the decoding of subsequently received latent representations.

[0167] As discussed above, the encode and decode processes may be performed by a data processing apparatus. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method.

[0168] Parameter Selection

[0169] A modification of the Al based compression process described above will now be discussed. A method for lossy image or video encoding, transmission and decoding, may comprise the steps of receiving an input image at a first computer system, selecting a first trained neural network from a first plurality of neural networks; encoding the input image using the first trained neural network to produce a latent representation; transmitting the latent representation to a second computer system; selecting a second trained neural network from a second plurality of neural networks; decoding the latent representation using the second trained neural network to produce an output image, wherein the output image is an approximation of the input image. An example of the plurality of trained neural networks which may be selected in shown in figure 14. This example shows three possible encoding neural networks 110a, 110b, 110c which may be selected as an encoder and three possible decoding neural networks 120a, 120b, 120c which may be selected as a decoder.

[0170] By selecting a particular first neural network and a particular second neural network during the encoding and decoding process, the compression and decompression processes may be improved. For example, particular neural networks or neural network systems may have different power consumptions when run. Depending on the image or frames that will be compressed, different levels of complexity of neural network or neural network system may perform compression and decompression at an acceptable level of quality and bitrate. Therefore, selecting an appropriate neural network or system of neural networks can result in reduce power consumption. This is because a particular neural network will use the same power when compressing or decompressing an image or frame regardless of the content of that frame. To reduce the power consumption of the process it is therefore preferable to select an appropriate neural network or neural network system for the encoding or decoding process.

[0171] The first plurality of neural networks and the second plurality of neural networks may comprise corresponding pairs of neural networks and a corresponding pair of neural networks are selected during the step of selecting a first trained neural network and the step of selecting a second trained neural network. When the first trained neural network performs and encoding process and the second trained neural network performs a decoding process, the first and second trained neural networks may have been trained simultaneously in a single end to end compression. In this case, the first and second trained neural networks may need to be selected together for optimal compression performance. Alternatively, one of the first plurality of neural networks may correspond to a plurality of networks in the second plurality. In this case, a single encoding network may be paired with a number of different decoding networks when compressing an image or frame. In a corresponding manner, one of the second plurality of neural networks may correspond to a plurality of networks in the first plurality. In this case, a single decoding network may be paired with a number of different encoding networks when compressing an image or frame.

[0172] The method may further comprise receiving, encoding, transmitting and decoding a further plurality of input images wherein the first and second trained neural networks used for the encoding and decoding of each of the plurality of input images are selected based on a predetermined schedule. This means that the particular neural networks used are selected in a repeating pattern. For example, in the case where there are two different neural networks A and B which may be selected as the first trained neural network, the neural networks may be used in an alternating ABAB pattern. In this case, the predetermined schedule comprises the use of different pairs of first and second neural networks for alternating input images. If the neural networks A and B have different power consumptions, the use of this schedule will result in an average power consumption that is between the average power consumption of each individual network. This is an example of a schedule that repeats every two steps. If three neural networks A, B and C are available, the neural networks may be used in a repeating ABCB pattern. A corresponding schedule may be used for the second trained neural network. The predetermined schedule may comprise the use of a first pair of first and second neural networks when the input image is an I-frame and a second pair of first and second neural networks when the input image is a not an I-frame. In this case, the schedule used will depend on the frame structure of the input video of the compression process. For example, if the frame structure of the input video is an I-frame followed by 7 P-frames (a group of picture (GOP) size of 8) the corresponding schedule would be ABBBBBBB.

[0173] The system on which the encoding or decoding process is operating may have particular power limitations or a maximum power that may be allocated to the compression process. This allocation may be used to obtain a target value. The predetermined schedule may be selected such that the average power consumption associated with the operation of at least one of the first trained neural network and the second trained neural network is below the target value. For example, the amount of frames for which a higher power network is used may be reduced to bring the average power consumption of the encoding or decoding process below the target value.

[0174] At least one of the first trained neural network and the second trained neural network is selected based on one or more of the following: encoding or decoding complexity, runtime, output image quality, available hardware, quality of service metrics, network conditions and power consumption. Each of these quantities, information or values may be determined prior to operation of the compression process and passed to the system or apparatus operating the compression process so that the information be used in the determination. The encoding complexity, runtime, output image quality and power consumption required for operation of each of the possible neural networks available for selection may have been predetermined. Alternatively, the steps of selecting the first trained neural network from the first plurality of neural networks and encoding the input image using the first trained neural network to produce a latent representation may comprise encoding the input image using each of the first plurality of neural networks; and selecting one of the produced latent representations for transmission to the second computer system. In this case, the selection of the first neural network is effectively made by selecting the associated latent representation produced as an output of the network. The selection of the produced latent representations may be based on one or more of the encoding complexity, runtime, output image quality and power consumption required for operation of the corresponding first neural network. For example, the output image quality resulting from a particular encoding network can depend on the input image or frame used. In this case, it may be beneficial to perform the encoding and decoding process using each available network so that the actual output image quality can be determined instead of using a predetermined estimate of the quality of the network. An example of such a process is shown in figure 15. In this example, the input image 5 is provided to each of the first plurality of trained neural networks. The output latent representation from each network is then used in the selection step 333 to determine the produced latent representation for transmission.

[0175] Each of the first plurality of neural networks and / or each of the second plurality of neural networks differ in one or more of: the number of layers, the number of filters applied in a particular layer, the hyperparameters and the number and / or location of skip connections. For example, a network with lower power consumption may have a reduced number of layers when compared to a network with higher power consumption. The network with higher power consumption may have the same layers as the network with lower power consumption but further comprise additional layers. The higher power consumption network may have additional filters in one or more layers when compared to the lower power consumption network.

[0176] The method discussed may also make use of flow information. The method may further comprising the step of receiving flow information associated with a previous input image;, wherein at least one of the first trained neural network and the second trained neural network is selected based on the received flow information. For example, if the flow information indicates a large amount of motion occurring between the input frame and the previous input frame. In this case, a more powerful neural network capable of maintaining the quality of the output image may be selected. Alternatively, if a small amount of motion is occurring, a less powerful neural network with low power consumption may be selected.

[0177] The first and the second neural network may be used for the transmission of flow information. In this case, the first trained neural network may additionally receive an input based on a previous input image, the output of the second trained neural network may be used to obtain a flow information; and the flow information may be used to obtain the output image. The first and second plurality of neural networks may comprise corresponding pairs of neural networks for flow information compression; and the corresponding pairs compress flow information at different quality levels. The method discussed may also make use of residual information. The method may further comprise the step of receiving residual information associated with a previous input image, wherein at least one of the first trained neural network and the second trained neural network is selected based on the received residual information. For example, if the residual information indicates that the size of the residual is large, a more powerful neural network capable of maintaining the quality of the output image may be selected to encode and / or decode the residual information. Alternatively, if a small residual is indicated, a less powerful neural network with low power consumption may be selected.

[0178] The first and the second neural network may be used for the transmission of flow information. In this case, the method may further comprise the step of receiving flow information between the input image and a previous input image, wherein the first trained neural network encodes a residual between the input image and a warped image obtained using the flow information, the second trained neural network decodes an output residual; and the output residual is used to obtain the output image. The first and second plurality of neural networks comprise corresponding pairs of neural networks for residual compression; and the corresponding pairs compress residuals at different quality levels.

[0179] One or more neural networks may be trained for use in a system as described above. A method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, may comprise the steps of: receiving an input image at a first computer system; selecting a first neural network from a first plurality of neural networks; encoding the input image using the first trained neural network to produce a latent representation; selecting a second neural network from a second plurality of neural networks; decoding the latent representation using the second trained neural network to produce an output image, wherein the output image is an approximation of the input image; evaluating a function based on a difference between the output image and the input image; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps using a first set of input images to produce a first plurality of trained neural network and a second plurality of trained neural networks.

[0180] As discussed above, the neural network selected may be used in either of an encoding or decoding process. For example, an encoding process may be a method for lossy image or video encoding and transmission, the method comprising the steps of: receiving an input image at a first computer system, selecting a first trained neural network from a first plurality of neural networks, encoding the input image using the first trained neural network to produce a latent representation; and transmitting the latent representation. A decoding process may be a method for lossy image or video receipt and decoding, the method comprising the steps of: receiving a latent representation transmitted by a first computer system at a second computer system, the latent representation corresponding to an input image, selecting a second trained neural network from a second plurality of neural networks, and decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image.

[0181] As discussed above, the compression process may be performed by a data processing system. The encode and decode processes may be performed by a data processing apparatus. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the encode and decode processes. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the encode and decode processes.

[0182] Convolution patterns

[0183] A modification of the Al based compression process discussed above will now be described. As discussed above, Al based compression processes make use of neural networks comprising convolutional layers. Each convolutional layer applies one or more convolutional filters to the input data to obtain the output. Each convolution operation is typically represented by a matrix containing a plurality of values that are applied to the input data in the convolution operation. Typical convolution filter sizes in Al based compression are 3 x 3 or 5 x 5 but a convolution operation can be of any size. At least or one of the values in a convolution operation may be a zero value. Such a zero value may also be referred to as a null value. A zero value may be exactly zero. Alternatively, a zero value may be a minimal value that is negligible in comparison to other values of the convolution operation or the input data. A convolution operation that comprises a number of zero values may be referred to as a sparse convolution. A sparse convolution may be a convolution operation in which the majority, or at least 50%, of the values are zero values. Alternatively, a sparse convolution may be defined as a convolution operation where at least 95% of the values are zero values. An example of a sparse convolution operation 550 is shown in figure 16. Alternatively or additionally, sparse convolutions may also be defined in the context of the hardware on which the convolution operation is being performed on. For example, a particular processing unit may define a sparse convolution as a convolution operation comprising at least 60% zero values. A different processing unit may define a sparse convolution as a convolution operation comprising at least 90% zero values. In this case, a sparse convolution may be defined as a convolution for which the operating hardware performs the convolution in a different way to the standard or general method for performing a convolution operation. For example, specific approximations may be applied to perform the convolution operation. For example, a lookup table of all the possible calculations necessary to perform the convolution operation may be prepared and this lookup table may be used to perform the sparse convolution operation.

[0184] Sparse convolutions may be used in an Al based compression process such as those discussed in the introduction. In a method for lossy image or video encoding transmission and decoding, the method may comprise the steps of receiving an image input image at a first computer system, encoding the input image using a first change neuron network to produce a latent representation, transmitting the laser representation to a second computer system, decoding and latent representation using a second train network to produce an output image, or in the output image is an approximation of the input image, wherein at least one of the first trained neural network and the second trained neural network comprise at least one convolution operation comprising a plurality of values set to 0. The at least one convolution operation comprising a plurality of values set to 0 may be considered a sparse convolution. Either or both of the first trained neural network and the second trained neural network may comprise a plurality of convolution operations comprising a plurality of values set to 0. The plurality of convolution operations may be present in a single layer of the trained neural network. For example, a single layer may comprise at least 50% of these convolution operations. Further to this, at least 80% of the convolution operations in a single layer may be these convolution operations. Alternately, all of the convolution operations in a particular layer may be these convolution operations. A significant proportion of convolution operations comprising a plurality of values set to 0 may be present in a particular layer due to the function or purpose of that particular layer within the trained neural network.

[0185] Alternatively or additionally, either or both of the first trained neural network and the second trained neural network may comprise at least 50% convolution operations comprising a plurality of values set to 0. Further to this, at least 80% of the convolution operations in the neural network may be these convolution operations. Alternately, all of the convolution operations in a particular neural network may be these convolution operation. A signification proportion these convolution operations may be present in a particular neural network in the Al based compression process.

[0186] The plurality of values set to 0 may be in a particular pattern. The particular pattern may be spatial. This means the locations of the zero values within the matrix defining the convolution has a particular spatial pattern. An example of a spatial pattern in a sparse convolution is a diagonal matrix, in which all of the non-diagonal values of the matrix are 0 values. Another example of a spatial pattern is where only a single row or column of the matrix has non zero values. These patterns may be considered pre-determined. This means that the patterns have been set in advance of the training process of the Al based compression process or that the patterns have been set to achieve a particular result instead of being learned.

[0187] During an Al based compression process, the parameters being used in a particular convolution operation may be modified. For example, a number of pre-determined patterns may be available when the Al based compression process is being performed. Prior to the operation of either of the first trained neural network or the second trained neural network during a particular iteration, the pattern of the plurality of values set to 0 in at least one convolution operation in these networks may be selected from a group of predetermined patterns. An example of this process is shown in figure 17. A pattern is selected from the group of patterns 550a, 550b, 550c in a selection step 555 and applied to a convolution operation. This may mean that the convolution operation performed varies between the compression of different frames or images.

[0188] The group of predetermined patterns may be created based on the different types of hardware apparatus that the Al based compression process will be performed. For example, the group of pre-determined patterns may include a one or more patterns that may be computed efficiently on a first type of computer hardware. The group of patterns may include one or more patterns that may be performed more efficiently on a different, second type of computer hardware.

[0189] The group of patterns may include a first group of patterns which perform efficient compression on a particular type of video content such as webcam data, streaming data, video game data or film data. Efficient compression may mean performing a compression process at a lower power consumption for an equivalent quality and bitrate. Additionally or alternatively, the group of patterns may include a first group of patterns which perform efficient compression for one or more quality levels, frame rates or target power consumptions. The group of patterns may be stored at both the first computer system and the second computer system. In this case, the group of patterns at each system may correspond and be identical to each other. Alternatively the group of patterns at each computer system may differ.

[0190] The selection of the pattern for use in a particular compression process may be based on one or more of a flag indicating the content type of the input image, a quality level, a target runtime and a target power consumption. For example, the content type of the input image or frame may be provided to the first computer apparatus before the compression processes begins. A pattern that is particularly efficient for this content type may then be selected. When more than one factor is to be considered, a particular ranking may be applied to the factors when selecting an appropriate pattern. For example, the quality level selected may take precedence over other factors. Alternatively or additionally, groups of pattern type that correspond to more than one factor may be present in the group of predetermined patterns. These patterns may be selected when the plurality of factors are present. Conversely, there may also be patterns which are known to be inefficient for particular outcomes.

[0191] Where the convolution comprising a plurality of value set to 0 is present in the second trained neural network, the selection of the pattern may be based on a flag received from the first computer system. This means that the determination discussed above may have been made at the first computer system and is transmitted to the second computer system along with the bitstream corresponding to any compressed data. The flag may be determined based on the latent representation produced by the first trained neural network. This means that the flag, and corresponding pattern selected for the convolution, is determined based on the latent representation to be decoded by the second trained neural network.

[0192] The pattern may also be selected based on information from a previously encoded or decoded input image or frame. For example, the pattern may be selected based on flow information determined between a the previously encoded frame and the present input frame.

[0193] During the operation of the Al based compression process on a particular computer apparatus, convolution operations of that process will be performed by particular hardware within the computer apparatus. Hardware within the computer apparatus may be more or less optimised for performing convolution operations comprising a plurality of values set to zero. The more optimised hardware may be able to perform the relevant operation more quickly, with a fewer number of calculations and with a lower power consumption than hardware that is not optimised. For example, a graphical processing unit (GPU) or neural processing unit (NPU) may be more optimised at performing the relevant convolution than a central processing unit (CPU). When the Al based compression process discussed above is performed, convolution operations comprising a plurality of values set to zero may be selected to be performed on optimised hardware. Additionally, convolution operations that do not fall within this definition may be selected to be performed on non-optimised hardware.

[0194] As discussed in the introduction above, Al based compression processes undergo training to arrive at an Al based compression process that is suitable for performing compression. During such a training process, one or more of the convolution operations performed in the Al based compression process may have a plurality of values set to 0.

[0195] This may mean that the particular values of the convolution operation are set to 0 at a particular point in the training process. For example, the plurality of values may be set to 0 at the beginning of the training process and not subject to updates during the training process. This means that the plurality of values are not updated during the backpropagation step of the training process. Alternatively, the plurality of values may be set to 0 during or at the end of the training process. The plurality of values set to 0 may be in a predetermined pattern in the convolution operation. Alternatively or additionally, the convolution operation comprising the plurality of values may be in a predetermined layer of a neural network in the Al based compression process.

[0196] Alternatively or additionally, the plurality of values set to zero in the convolution operation may arise as a consequence of other factors of the training process. For example, neural network training processes may use regularisation techniques, which penalize large values from occurring in the operations being performed by the network. The training of an Al based compression process using regularisation techniques may therefore result in convolution operations comprising a plurality of values at 0 at the end of the training process.

[0197] As discussed above, the neural network selected may be used in either of an encoding or decoding process. For example, an encoding process may be a method for lossy image or video encoding and transmission, the method comprising the steps of: receiving an input image at a first computer system, selecting a first trained neural network from a first plurality of neural networks, encoding the input image using the first trained neural network to produce a latent representation and transmitting the latent representation. A decoding process may be a method for lossy image or video receipt and decoding, the method comprising the steps of: receiving a latent representation transmitted by a first computer system at a second computer system, the latent representation corresponding to an input image, selecting a second trained neural network from a second plurality of neural networks, and decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image.

[0198] As discussed above, the compression process may be performed by a data processing system. The encode and decode processes may be performed by a data processing apparatus. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the encode and decode processes. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the encode and decode processes.

[0199] Hardware variation

[0200] A modification of the Al based compression process discussed in the introduction will now be described. An Al based compression process may be performed across more than one computer system. For example, the encoding or compression part of the process may be performed at a first computer system to obtain a bitstream. The bitstream is transmitted to a second computer system, at which the decoding or decompression part of the process is performed to obtain the original data. Operation of the compression process at each computer system may involve the operation of one or more functions, for example convolution or quantization functions as described in more detail above. However, these operations may be performed in different ways depending on the computer system that the operation is being performed on.

[0201] For example, a generalised quantisation function performs an operation in which a set of continuous or floating point values are mapped to one or more discrete or integer values (which may be referred to as bins). However, the implementation of the quantisation function may vary. For example, quantisation may be performed by a ceiling function, which maps each continuous value to the next higher integer value. Alternatively, quantisation may be performed by a floor function, which maps each continuous value to the next lower integer value. Alternatively, quantisation may be performed by a rounding function, which maps each continuous value above a threshold value to the next higher integer value and each below the threshold value to the next lower integer value. A first computer system might use a floor function to perform a quantisation operation and a second computer system might use a ceiling operation. Alternatively, the threshold value used when performing a rounding operation may be different between a first and second computer system.

[0202] When these differences in operation are present, the operation of an Al based compression process performed across such two computer systems may be degraded. Particular problems may occur when an Al based compression process uses a hyper-network as discussed in the introduction. This is because compression performance in an Al based compression process may be particular sensitive to the probability distribution parameters predicted by the hyper-network that are used to entropy encode and decode the bitstream. Small differences in these parameters between the encode and decode process may lead to large errors in the output image or frame when compared to the input image or frame.

[0203] An Al based compression process that may mitigate these issues will now be discussed. In particular, a method for lossy image or video encoding, transmission and decoding may comprise the steps of receiving an input frame at a first computer system, encoding the input frame using a first trained neural network to produce a latent representation, determining one or more target values of the latent representation, transmitting the latent representation and information relating to the target values to a second computer system, modifying the received latent representation based on the information relating to the target values and decoding the latent representation using a second trained neural network to produce an output frame, wherein the output frame is an approximation of the input frame. An example of such a method is shown in figure 18. In a determination step 601 the one or more target values are determined. On receipt, the one or more target values are used to modify the received latent representation in a modifying step.

[0204] The target values may be one or more values which are expected to cause a degradation of compression performance if the way in which operations are performed at the first computer system differs from the way in which operations are performed at the second computer system. The target values may be determined based on known values associated with particular operations. For example, it may be known that the threshold value of the rounding function performed by the first computer system is slightly different to the threshold value of the rounding function performed by the second computer system. In this case, the target values may be one or more of the values of the latent representation that are close to the threshold value. This may be determined by identifying values of the latent representation that differ by less than a predetermined value from the threshold value of the function.

[0205] The information relating to the target values may comprise the values of the target values. Alternatively or additionally, the information may comprise the location of the target values. This may be the coordinate location of the target value within the array or other storage defining the latent representation.

[0206] The method may comprise the step of modifying the latent representation based on the information relating to the target values. This modification may be performed prior to the transmission or entropy coding of the latent representation. For example, the modification may comprise adding or subtracting a correction value to one or more of the target values. The modification may comprise replacing one or more of the target values with a correction value.

[0207] As discussed above, various transformations may be performed on the data in an Al based compression pipeline. For example, after the input image has been encoded into a latent representation, the latent representation may be quantized. As part of this quantization, the latent representation may be scaled or divided by a scaling value representing the quantization bins of the quantization process. When the latent representation is obtained at the second computer system, a corresponding transformation may be performed. This corresponding transformation may be the inverse of the transformation performed on the encode side. For example, the received latent may be multiplied by the scaling value used during the quantization process on the encode side.

[0208] As discussed above, certain operations may be known to be problematic when performed on different computer systems. In the case where the transformation performed on the latent representation is one of these operations, the modification of the latent representation at either the first or second computer system may be based on the transformation or corresponding transformation. For example, if the transformation is a quantization operation the modification may be to change the threshold value of the quantization operation.

[0209] Alternatively or additionally, the modification may comprise modifying the transformation performed on the latent representation at the first computer system or received latent representation at the second computer system. For example, if the transformation comprises a quantization operation, the modification may be made to the quantization operation when applied to target values of the latent representation. The modification to the quantization operation maybe the change of one or more parameters of the quantized operation. For example, the threshold values of the quantization operation may be changed. Alternatively or additionally, the bin size of the quantization operation may be changed.

[0210] In another example, the form of the quantization operation may be changed. For example, the quantization process may initially be one of a round, floor, or ceiling function. In this case, the modification may comprise changing the quantization operation to another of the round, floor or ceiling functions. Alternatively, the quantization operation may have been performed in an alternative way and the modification process comprises changing the quantization operation to one of a round, floor or ceiling function.

[0211] The concepts discussed in this section may also be applied when a hyper network is used in the Al based compression process. For example, a method for lossy frame or video encoding, transmission and decoding may comprise the steps of receiving an input frame at a first computer system, encoding the input frame using a first trained neural network to produce a latent representation, encoding the latent representation using a second trained neural network to obtain a hyper-latent representation, determining one or more target values of the hyper- latent representation, transmitting the latent representation, the hyper-latent representation and information relating to the target values to a second computer system, decoding the hyper-latent representation using a third trained neural network to obtain a decoded hyper-latent, modifying the decoded hyper-latent based on the information relating to the target values and decoding the latent representation using a fourth trained neural network and the decoded hyper-latent to produce an output frame, wherein the output frame is an approximation of the input frame. An example of such a method is shown in figure 19. In a determination step 611 the one or more target values are determined. On receipt, the one or more target values are used to modify the received hyper-latent representation in a modifying step.

[0212] This may be particularly beneficial in the case where the performance of the Al based compression process is particularly sensitive to values that are transmitted or predicted by the hyper network. This may be the case when the output of the hyper network comprises one or more parameters associated with the probability distribution of the latent representation. For example, the output of the hyper network decoder may comprise one or more mean values of the probability distribution of the latent representation. In this case the modification may comprise changing one or more of these mean values. Alternatively or additionally, the output may comprise one or more scale values associated with the probability distribution of the latent representation and the modification may comprise changing one or more of these scale values. The concept may be applied in a similar way when a further hyper-hyper network is present in the Al based compression process that may be used to predict parameters of the probability distribution of the hyper-latent. The concept may also be applied to a flow-residual Al based compression pipeline as described in the introduction, in particular to the latent representation associated with flow information. The concept described in this section may be applied in the training process of the neural networks to be used in the Al based compression pipeline. In particular, the function used to determine the performance of the network may include additional terms beyond the rate and distortion terms discussed in the introduction.

[0213] For example, the function may be additionally based on a property of the one or more target values of the latent representation. For example, the function may be based on one or more of the number or quantity of target values identified, the magnitude of one or more of the target values and the difference between the one more target values and one or more set values. Incorporating terms based on one or more of these factors into the loss function may allow the network to be trained in such a way that reduces the likelihood of errors occurring due to differences in operation between a first computer system and a second computer system. As discussed in the introduction, the relevant portions of the Al based compression process may be performed individually at each of the first computer system and the second computer system. In this case, the encoding and transmission of the input image or frame and determining off the target information takes place at the first computer system. Separately, the target information and latent representation may be received at a second computer system and decoded to obtain an output image or frame.

[0214] An alternative concept which may improve the performance of an Al based compression system operating over multiple computer systems will now be described. A method for lossy image or video encoding, transmission and decoding may comprise the steps of receiving an input frame at a first computer system, encoding the input frame using a first trained neural network to produce a latent representation, wherein during the encoding step, a correction is applied to at least one of the values associated with the input frame, transmitting the latent representation to a second computer system and decoding the latent representation using a second trained neural network to produce an output frame, wherein the output frame is an approximation of the input frame. In this way, values which may cause problems when the encoding and decoding processes are taking place on different computer systems may be identified and corrected for.

[0215] An example of this method is shown in figure 20. During the encoding process, a correction step 710 is performed where a correction is applied to at least one of the values associated with the input frame being encoded by the first trained neural network.

[0216] The correction may be applied to one or more feature maps associated with the input frame. The correction may be performed for every feature map associated with the input frame. For example, if a particular problematic operation is performed in each layer of the neural network performing the encoding process, a correction may be applied to the feature map after the operation in each layer.

[0217] The correction to be performed may be based on one or more of the input to a layer of a neural network, the output of that layer, the input frame, a previously received input frame, the latent representation, a previously received latent representation, the output frame and a previously decoded output frame. The correction may also be based on one or more probability distribution parameters obtained from the input frame and slash or a previous input frame. For example, the probability distribution parameters may be associated with the latent representation.

[0218] The correction may comprise modifying the at least one values associated with the input frame. For example, the correction may comprise adding a correction value to at least one of the values. Alternatively or additionally, the correction may comprise replacing at least one of the values with a correction value. The correction may be performed at either of the first computer system during the encoding part of the Al based compression process or the second computer system during the decoding part of the Al based compression process. An example of the correction being performed in the decoding part is shown in figure 21. During the decoding process, a correction step 720 is performed where a correction is applied to at least one of the values associated with the latent representation being decoded by the second trained neural network.

[0219] The correction value maybe obtained as the output of a further training neural network. For example, the trained neural network may receive one or more of the following as an input, the input frame, a previously received input frame, the latent representation, a previously produced latent representation, the output frame, a previously decoded output frame, one or more probability distribution parameters obtained from the input frame and / or a previous input frame, and one or more of the feature maps associated with the input frame and / or a previous input frame.

[0220] The method may further comprise the step of transmitting a flag associated with the correction to be performed. For example, before compression of an input frame takes place, the second computer system may transmit a flag to the first computer system. Alternatively, the first computer system may transmit a flag to the second computer system together with the latent representation. The flag may contain information about the nature of the correction to be performed. For example, the flag may indicate the hardware used at the first or second computer system to perform the encoding or decoding step of the process. Alternatively or additionally, the flag may indicate a particular operation that will be performed at either of the first or second computer system. The flag may be used to determine which values are to be corrected or the nature of the correction to be performed.

[0221] Alternatively or additionally, the flag may indicate that at least one of these transformations performed by the first or second trained neural network is modified before the encoding step is performed. The modified transformation may be replaced by a different transformation. For example, the quantization process may be replaced as discussed above. Alternatively, the modified transformation may be a convolution operation. For example, the size of the convolution or the nature of the multiplication performed within the convolution operation may be changed. Alternatively, the modified transformation may be an activation function. For example, one type of activation function may be replaced with a different type of activation function.

[0222] The concept described in this section may be applied in the training process of the neural networks to be used in the Al based compression pipeline.

[0223] The subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. The subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory program carrier for execution by, or to control the operation of, data processing apparatus. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. The computer storage medium is not, however, a propagated signal.

[0224] The term “data processing apparatus” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0225] A computer program (which may also be referred to or described as a program, software, a software application, a module, a software module, a script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a Hie in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0226] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0227] Computers suitable for the execution of a computer program include, by way of example, can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a VR headset, a game console, a Global Positioning System (GPS) receiver, a server, a mobile phones, a tablet computer, a notebook computer, a music player, an e-book reader, a laptop or desktop computer, a PDAs, a smart phone, or other stationary or portable devices, that includes one or more processors and computer readable media, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0228] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0229] The subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), e.g., the Internet.

[0230] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0231] While this specification contains many specific implementation details, these should be construed as descriptions of features that may be specific to particular examples of particular inventions. Certain features that are described in this specification in the context of separate examples can also be implemented in combination in a single example. Conversely, various features that are described in the context of a single example can also be implemented in multiple examples separately or in any suitable subcombination. Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the examples described above should not be understood as requiring such separation in all examples, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

Claims

CLAIMS1. A method for lossy image or video encoding, the method comprising the steps of: performing a first lossy image or video encoding process, the process comprising the steps of: receiving an input image at a first computer system; and encoding the input image using a first trained neural network to produce a latent representation; wherein, at a step of the first lossy image or video encoding process, a result of a determination is retrieved; and based on the result of the determination an additional process is performed, wherein the additional process is configured to, when an output image that is an approximation of the input image is obtained by decoding the latent representation, result in an improved approximation of the output image to the input image.

2. The method of claim 1, wherein the additional process comprises activating at least one additional neural network layer associated with the first trained neural network during the step of encoding the input image using the first trained neural network.

3. The method of claims 1 or 2, wherein the additional process comprises the step of: encoding the input image using a second trained neural network to produce a second latent representation.

4. The method of claim 3, wherein at least one feature map of the first trained neural network is provided as an additional input to at least one layer of the second trained neural network.

5. The method of any one of claims 1 to 4, wherein the additional process comprises the steps of: processing one of more of the input image prior to encoding by the first trained neural network and the latent representation after encoding by the first trained neural network.

6. The method of claim 5, wherein the processing is performed by a third trained neural network.

7. The method of any one of claims 1 to 6, wherein the first lossy image or video encoding process is performed on a first processing unit of the first computer system; and the additional process is performed on a second processing unit of the first computer system.

8. The method of any one of claims 1 to 7, wherein the additional process is performed by a graphics processing unit of the first computer system.

9. The method of any one of claims 1 to 7, wherein the additional process is performed by a neural processing unit of the first computer system.

10. The method of any one of claims 1 to 9, wherein the first lossy image or video encoding process is performed by a central processing unit of the first computer system.9711. The method of any one of claims 1 to 10, wherein the determination is performed by the first computer system.

12. The method of any one of claims 1 to 10, wherein the determination is performed by a second computer system configured to receive the latent representation from the first computer system.

13. A method for lossy image or video decoding, the method comprising the steps of: performing a first lossy image or video decoding process, the process comprising the steps of: receiving a latent representation at a second computer system, wherein the latent representation corresponds to an input image; and decoding the latent representation using a fourth trained neural network to produce an output image, wherein the output image is an approximation of the input image; wherein, at a step of the first lossy image or video decoding process, a result of a determination is retrieved; and based on the result of the determination an additional process is performed, wherein the additional process is configured to result in an improved approximation of the output image to the input image.

14. The method of claim 13, wherein the additional process comprises activating additional neural network layers associated with the fourth trained neural network during the step of decoding the latent representation using the fourth trained neural network.

15. The method of claims 13 or 14, wherein the additional process comprises the steps of: receiving a second latent representation at the second computer system, wherein the second latent representation corresponds to the input image; decoding the second latent representation using a fifth trained neural network to produce a refined output image.

16. The method of claim 15, wherein at least one feature map of the fourth trained neural network is provided as an additional input to at least one layer of the fifth trained neural network.

17. The method of claim 15 or 16, wherein the output image is provided as an additional input to at least one layer of the fifth trained neural network.

18. The method of any one of claims 15 to 17, wherein the step of decoding the second latent representation comprises obtaining flow information from the output of the fifth trained neural network, where the flow information corresponds to a flow between the input image and a previous input image; and the flow information is used to obtain the refined output image.

19. The method of any one of claims 15 to 18, wherein the additional process comprises the step of: retrieving flow information obtained from the output of the fourth trained neural network, the flow information corresponding to a flow between the input image and a previous input image; andusing the flow information obtained from the output of the third neural network to refine the flow information obtained from the output of the fifth neural network.

20. The method of any one of claims 13 to 19, wherein the additional process comprises processing at least one of the latent representation prior to decoding by the fourth trained neural network and the output image after decoding by the fourth trained neural network.

21. The method of claim 20, wherein the processing is performed using a sixth trained neural network.

22. The method of claim 21, wherein the sixth neural network additionally receives at least one of a previously produced output image and a previously produced processed output image as an input.

23. The method of any one of claims 13 to 22, wherein the first lossy image or video decoding process is performed on a first processing unit of the second computer system; and the additional process is performed on a second processing unit of the second computer system.

24. The method of any one of claims 13 to 23, wherein the additional process is performed by a graphics processing unit of the second computer system.

25. The method of any one of claims 13 to 23, wherein the additional process is performed by a neural processing unit of the second computer system.10026. The method of any one of claims 13 to 25, wherein the first lossy image or video decoding process is performed by a central processing unit of the second computer system.

27. The method of any one of claims 13 to 26, wherein the determination is performed by the second computer system.

28. The method of any one of claims 13 to 26, wherein the determination is performed by a first computer system configured to transmit the latent representation to the second computer system.

29. The method of any one of claims 1 to 28, wherein the determination is based on the available compute at at least one of the first computer system and the second computer system.

30. The method of any one of claims 1 to 29, wherein the determination is based on the runtime of the first lossy image or video encoding, transmission and decoding process at at least one of the first computer system and the second computer system.

31. The method of any one of claims 1 to 28, wherein the determination is predetermined.

32. A method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: performing a first lossy image or video encoding, transmission and decoding process, the process comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first trained neural network to produce a latent101representation; transmitting the latent representation to a second computer system; and decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image; wherein, at a step of the first lossy image or video encoding, transmission and decoding process, a determination is performed at at least one of the first computer system and the second computer system; and based on the result of the determination, an additional process is performed, wherein the additional process is configured to result in an improved approximation of the output image to the input image.

33. A method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first neural network to produce a latent representation; decoding the latent representation using a second neural network to produce an output image, wherein the output image is an approximation of the input image; retrieving the result of a determination; based on the result of the determination, performing an additional process, wherein the additional process is configured to result in an improved approximation of the output image to the input image; evaluating a function based on a difference between the output image and the input102image; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps using a first set of input images to produce a first trained neural network and a second trained neural network.

34. The method of claims 33, wherein the additional process comprises using a third neural network; and the parameters of the third neural network are additionally updated based on the evaluated function to obtain a third trained neural network.

35. The method of claim 33 or 34, wherein the evaluated function is based on a difference between the output image and the input image and a difference between the improved approximation of the output image and the input image.

36. A method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first trained neural network to produce a latent representation; decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image; retrieving the result of a determination;based on the result of the determination, performing an additional process, wherein the additional process is configured to result in an improved approximation of the output image to the input image; wherein the additional process comprises using a third neural network; evaluating a function based on a difference between the output image and the input image; updating the parameters of the third neural network based on the evaluated function; and repeating the above steps using a first set of input images to produce a third trained neural network.

37. A data processing apparatus configured to perform the method of any one of claims 1 to 36.

38. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any one of claims 1 to 36.

39. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of any one of claims 1 to 36.

40. A method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input video at a first computer system; encoding an input frame of the input video using a first trained neural network to produce a latent representation;determining one or more neural network parameters based on one or more frames of the input video; transmitting the latent representation and the one or more neural network parameters to a second computer system; decoding the latent representation using a second trained neural network to produce an output frame, wherein the output frame is an approximation of the input frame; and applying the one or more neural network parameters to the second trained neural network for the decoding of subsequent output frames corresponding to subsequent input frames of the input video.

41. The method of claim 41, wherein the one or more neural network parameters replace one or more of the parameters of the second neural network.

42. The method of claim 41, wherein the one or more neural network parameters are neural network weight parameters.

43. The method of claim 42, wherein the one or more neural network weight parameters are a perturbation of one or more base neural network weights; and the one or more neural network weight parameters are combined with one or more of the weights of the second neural network.

44. The method of any one of claims 40 to 43, wherein the one or more neural network parameters are compressed before being transmitted to the second computer system.

45. The method of claim 44, wherein the one or more neural network parameters are compressed by encoding by a trained neural network.

46. The method of any one of claims 40 to 45, wherein the input frame is an I-frame of the input video.

47. The method of any one of claims 40 to 46, wherein the determination of the one or more neural network parameters is based on all of the frames of the input video.

48. The method of any one of claims 40 to 46, wherein the determination of the one or more neural network parameters is based on a group of frames of the input video comprising an 1-frame and one or more P- frames.

49. The method of any one of claims 40 to 46, wherein the determination of the one or more neural network parameters is based on a sub-group of frames of the input video comprising one or more P- frames.

50. The method of any one of claims 40 to 49, wherein the determination of the one or more neural network parameters based on one or more frames of the input video is based on a tag associated with the one or more frames.

51. The method of any one of claims 40 to 50, wherein the one or more neural network parameters are selected from a plurality of predetermined sets of neural network parameters.

52. The method of any one of claims 40 to 51, wherein the one or more neural network parameters are additionally applied to the first trained neural network for the encoding of subsequent input frames.

53. The method of claim 52 wherein, when the the one or more neural network parameters are selected from a plurality of predetermined sets of neural network parameters, each predetermined set of neural network parameters comprises a set of parameters corresponding to the first trained neural network and a set of parameters corresponding to the second trained neural network.

54. The method of any one of claims 40 to 53, wherein the one or more neural network parameters are obtained from a trained neural network that receives the one or more frames of the input video as an input.

55. The method of any one of claims 40 to 54, further comprising the steps of; after encoding a subsequent input frame, determining a further one or more neural network parameters based on one or more frames of the input video; transmitting the further one or more neural network parameters to the second computer system; and applying the further one or more neural network parameters to the second trained neural network for the decoding of further subsequent output frames corresponding to further subsequent input frames of the input video.10756. A method of training one or more neural network parameter sets, the one or more neural network parameter sets being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input video at a first computer system; encoding an input frame of the input video using a first neural network to produce a latent representation; determining one or more neural network parameters based on one or more frames of the input video; transmitting the latent representation and the one or more neural network parameters to a second computer system; decoding the latent representation using a second neural network to produce an output frame, wherein the output frame is an approximation of the input frame; and applying the one or more neural network parameters to the second neural network for the decoding of subsequent output frames corresponding to subsequent input frames of the input video; evaluating a function based on a difference between the output frame and the input frame; updating the one or more neural network parameters based on the evaluated function; and repeating the above steps using a first set of input videos to produce one or more trained neural network parameter sets.

57. The method of claim 56, wherein the function is additionally based on the bitrate of the one or more neural network parameters.

58. The method of claim 56 or 57, wherein the one or more neural network parameters are selected from a plurality of sets of neural network parameters; and the function is additionally based on the bitrate of each of the plurality of sets of neural network parameters.

59. The method of any one of claims 56 to 58, wherein the parameters of the second neural network are additionally updated based on the evaluated function.

60. A method for lossy image or video encoding and transmission, the method comprising the steps of: receiving an input video at a first computer system; encoding an input frame of the input video using a first trained neural network to produce a latent representation; determining one or more neural network parameters based on one or more frames of the input video; transmitting the latent representation and the one or more neural network parameters.

61. A method for lossy image or video receipt and decoding, the method comprising the steps of: receiving a latent representation and one or more neural network parameters transmitted by a first computer system at a second computer system, the latent representation corresponding to an input frame of an input video; decoding the latent representation using a second trained neural network to produce an output frame, wherein the output frame is an approximation of the input frame; andapplying the one or more neural network parameters to the second trained neural network for the decoding of subsequent output frames corresponding to subsequent input frames of the input video.

62. A data processing system configured to perform the method of any one of claims 40 to 55.

63. A data processing apparatus configured to perform the method of claims 56 to 61.

64. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of claims 56 to 61.

65. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of claims 56 to 61.

66. A method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first neural network to produce a latent representation; decoding the latent representation using a second neural network to produce an output image, wherein the output image is an approximation of the input image; evaluating a first function based on a difference between the output image and the input image; updating the parameters of the first neural network and the second neural network based110on the evaluated function; repeating the above steps using a first set of input images to produce a first trained neural network and a second trained neural network; sending one or more parameters of the first trained neural network or the second trained neural network to a second computer system; and at the second computer system, modifying one or more parameters of a third trained neural network for use in lossy image or video encoding, transmission and decoding with the one or more parameters of the first trained neural network or the second trained neural network.

67. The method of claim 66, wherein the one or more parameters are neural network weights.

68. The method of claim 66 or 67 wherein the one or more parameters are perturbations of neural network weights.

69. The method of any one of claims 66 to 68, wherein the previous one or more parameters of the third neural network are deleted after the modification step.

70. The method of any one of claims 66 to 68, wherein the previous one or more parameters of the third neural network are stored at the second computer system after the modification step.

71. The method of claim 70, wherein the previous one or more parameters are compressed before being stored at the second computer system.I ll72. The method of any one of claims 66 to 71, wherein at least one of the architecture, flow and mapping of a system for lossy image or video encoding, transmission and decoding comprising the third neural network are not changed during the modifying step.

73. The method of any one of claims 66 to 72, wherein the one or more parameters are compressed before the step of sending the one or more parameters to the second computer system.

74. The method of any one of claims 66 to 72, wherein the one or more parameters are compressed after receipt at the second computer system.

75. The method of any one of claims 66 to 74 further comprising the step of, prior to the step of receiving the input image at the first computer system, sending one or more components of a system for lossy image or video encoding, transmission and decoding comprising the third neural network to the second computer system.

76. The method of claim 75, wherein the one or more components of a system for lossy image or video encoding, transmission and decoding comprising the third neural network are sent from a third computer system.

77. The method of any one of claims 66 to 76, wherein the one or more parameters of the third trained neural network have been obtained using a training process based on a second set of input images which is different to the first set of input images.11278. The method of any one of claims 66 to 77, wherein the one or more parameters of the third trained neural network have been obtained using a training process based on a second function which is different to the first function.

79. A method for lossy image or video encoding and transmission, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first trained neural network to produce a latent representation; transmitting the latent representation; sending one or more parameters of a third trained neural network; receiving a second input image at the first computer system; encoding the second input image using the first trained neural network to produce a second latent representation; and transmitting the second latent representation.

80. A method for lossy image or video receipt and decoding, the method comprising the steps of: receiving a latent representation transmitted by a first computer system at a second computer system, the latent representation corresponding to an input image; decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image; receiving one or more parameters of a third trained neural network;modifying one or more parameters of the second trained neural network with the one or more parameters of the third trained neural network; receiving a second latent representation transmitted by a first computer system, the latent representation corresponding to a second input image; and decoding the second latent representation using the second trained neural network to produce a second output image, wherein the second output image is an approximation of the second input image.

81. The method of claim 80, further comprising the step of receiving a flag at the second computer system; and based on the flag, selecting either the unmodified or modified second trained neural network for the decoding of subsequently received latent representations.

82. A data processing apparatus configured to perform the method of any one of claims 66 to 81.

83. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any one of claims 66 to 81.

84. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of any one of claims 66 to 81.

85. A method for lossy image or video encoding, transmission and decoding, the method comprising the steps of:114receiving an input image at a first computer system; selecting a first trained neural network from a first plurality of neural networks; encoding the input image using the first trained neural network to produce a latent representation; transmitting the latent representation to a second computer system; selecting a second trained neural network from a second plurality of neural networks; decoding the latent representation using the second trained neural network to produce an output image, wherein the output image is an approximation of the input image.

86. The method of claim 85, wherein the first plurality of neural networks and the second plurality of neural networks comprise corresponding pairs of neural networks; and a corresponding pair of neural networks are selected during the step of selecting a first trained neural network and the step of selecting a second trained neural network.

87. The method of claim 85 or 86, further comprising receiving, encoding, transmitting and decoding a further plurality of input images; wherein the first and second trained neural networks used for the encoding and decoding of each of the plurality of input images are selected based on a predetermined schedule.

88. The method of claim 87, wherein the predetermined schedule comprises the use of a first pair of first and second neural networks when the input image is an I-frame and a second pair of first and second neural networks when the input image is a not an 1-frame.11589. The method of claim 87, wherein the predetermined schedule comprises the use of different pairs of first and second neural networks for alternating input images.

90. The method of any one of claims 87 to 89, wherein the predetermined schedule is selected such that the average power consumption associated with the operation of at least one of the first trained neural network and the second trained neural network is below a target value.

91. The method of claims 85 or 86, wherein at least one of the first trained neural network and the second trained neural network is selected based on one or more of the following: encoding or decoding complexity, runtime, output image quality, available hardware, quality of service metrics, network conditions and power consumption.

92. The method of claims 85 or 86, wherein the steps of selecting the first trained neural network from the first plurality of neural networks and encoding the input image using the first trained neural network to produce a latent representation comprises encoding the input image using each of the first plurality of neural networks; and selecting one of the produced latent representations for transmission to the second computer system.

93. The method of claim 92, wherein the selection of the produced latent representations is based on one or more of the encoding complexity, runtime, output image quality and power consumption required for operation of the corresponding first neural network.

94. The method of any one of claims 85 to 93, wherein each of the first plurality of neural networks and / or each of the second plurality of neural networks differ in one or more of: the number of layers, the number of filters applied in a particular layer, the hyperparameters and the number and / or location of skip connections.

95. The method of any one of claims 85 to 94, further comprising the step of receiving flow information associated with a previous input image; wherein at least one of the first trained neural network and the second trained neural network is selected based on the received flow information.

96. The method of any one of claims 85 to 95, wherein the first trained neural network additionally receives an input based on a previous input image; the output of the second trained neural network is used to obtain a flow information; and the flow information is used to obtain the output image.

97. The method of claim 96, wherein the first and second plurality of neural networks comprise corresponding pairs of neural networks for flow information compression; and the corresponding pairs compress flow information at different quality levels.

98. The method of any one of claims 85 to 97, further comprising the step of receiving residual information associated with a previous input image; wherein at least one of the first trained neural network and the second trained neural network is selected based on the received residual information.11799. The method of any one of claims 85 to 98, further comprising the step of receiving flow information between the input image and a previous input image; wherein the first trained neural network encodes a residual between the input image and a warped image obtained using the flow information; the second trained neural network decodes an output residual; and the output residual is used to obtain the output image.

100. The method of claim 99, wherein the first and second plurality of neural networks comprise corresponding pairs of neural networks for residual compression; and the corresponding pairs compress residuals at different quality levels.

101. A method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; selecting a first neural network from a first plurality of neural networks; encoding the input image using the first trained neural network to produce a latent representation; selecting a second neural network from a second plurality of neural networks; decoding the latent representation using the second trained neural network to produce an output image, wherein the output image is an approximation of the input image; evaluating a function based on a difference between the output image and the input image;updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps using a first set of input images to produce a first plurality of trained neural network and a second plurality of trained neural networks.

102. A method for lossy image or video encoding and transmission, the method comprising the steps of: receiving an input image at a first computer system; selecting a first trained neural network from a first plurality of neural networks; encoding the input image using the first trained neural network to produce a latent representation; and transmitting the latent representation.

103. A method for lossy image or video receipt and decoding, the method comprising the steps of: receiving a latent representation transmitted by a first computer system at a second computer system, the latent representation corresponding to an input image; selecting a second trained neural network from a second plurality of neural networks; and decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image.

104. A data processing system configured to perform the method of any one of claims 85 to 101.

105. A data processing apparatus configured to perform the method of claims 102 or 103.

106. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of claims 102 or 103.

107. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of claims 102 or 103.

108. A method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first trained neural network to produce a latent representation; transmitting the latent representation to a second computer system; decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image; wherein at least one of the first trained neural network and the second trained neural network comprises a convolution comprising a plurality of values set to zero.

109. The method of claim 108, wherein the pattern of the plurality of values set to zero is predetermined.120110. The method of claim 108, wherein prior to the step of encoding or decoding by the first trained neural network or the second trained neural network, the pattern of the plurality of values set to zero is selected from a group of predetermined patterns.

111. The method of claim 110, wherein the pattern of the plurality of values is selected based on one or more of: a flag indicating the content type of the input image; a quality level; a target runtime and a target power consumption.

112. The method of claim 110 or 111, wherein the second trained neural network comprises the convolution comprising a plurality of values set to zero; and the pattern of the plurality of values is selected based on a flag received from the first computer system.

113. The method of claim 112, wherein the flag is determined based on the latent representation produced by the first trained neural network.

114. The method of any one of claims 110 to 113, wherein the pattern of the plurality of values is selected based on information from a previously encoded and decoded input image.

115. The method of claim 115, wherein the information is flow information.

116. The method of any one of claims 108 to 115, wherein the convolution comprising a plurality of values set to zero comprises at least 50% zero values.121117. The method of claim 116, wherein the convolution comprising a plurality of values set to zero comprises at least 95% zero values.

118. The method of any one of claims 108 to 117, wherein at least one layer of the first or second trained neural network comprises at least 50% convolutions comprising a plurality of values set to zero.

119. The method of claim 118, wherein at least one layer of the first or second trained neural network comprises at least 80% convolutions comprising a plurality of values set to zero.

120. The method of claim 119, wherein at least one layer of the first or second trained neural network comprises only convolutions comprising a plurality of values set to zero.

121. The method of any of claims 108 to 120, wherein at least 50% of the convolutions of at least one of the first trained neural network and the second trained neural network are convolutions comprising a plurality of values set to zero.

122. The method of claim 121, wherein at least 80% of the convolutions of at least one of the first trained neural network and the second trained neural network are convolutions comprising a plurality of values set to zero.

123. The method of claim 122, wherein at least one of the first trained neural network and the second trained neural network comprises only convolutions comprising a plurality of values set to zero.122124. The method of any one of claims 108 to 123, wherein the calculations associated with the convolution comprising a plurality of values set to zero are performed on a first hardware component; the calculations associated with at least one other layer of the neural network comprising the convolution comprising a plurality of values set to zero are performed on a second hardware component; and the first hardware component is optimized for performing convolution operations where the convolution comprises a plurality of values set to zero.

125. A method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first neural network to produce a latent representation; decoding the latent representation using a second neural network to produce an output image, wherein the output image is an approximation of the input image; evaluating a function based on a difference between the output image and the input image; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps using a first set of input images to produce a first trained neural network and a second trained neural network;wherein at least one of the first neural network and the second neural network comprises a convolution comprising a plurality of values set to zero.

126. The method of claim 125, wherein the pattern of the plurality of values set to zero is predetermined.

127. The method of claim 125 or 126, wherein the convolution comprising a plurality of values set to zero is in a predetermined layer of the first neural network or the second neural network.

128. A method for lossy image or video encoding and transmission, the method comprising the steps of: receiving an input image at a first computer system; encoding the input image using a first trained neural network to produce a latent representation; and transmitting the latent representation; wherein the first trained neural network comprises a convolution comprising a plurality of values set to zero.

129. A method for lossy image or video receipt and decoding, the method comprising the steps of: receiving a latent representation transmitted by a first computer system at a second computer system, the latent representation corresponding to an input image; and decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image;124wherein the second trained neural network comprises a convolution comprising a plurality of values set to zero.

130. A data processing system configured to perform the method of any one of claims 108 to 127.

131. A data processing apparatus configured to perform the method of claims 128 or 129.

132. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of claims 128 or 129.

133. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of claims 128 or 129.

134. A method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input frame at a first computer system; encoding the input frame using a first trained neural network to produce a latent representation; determining one or more target values of the latent representation transmitting the latent representation and information relating to the target values to a second computer system; modifying the received latent representation based on the information relating to the target values; and125decoding the latent representation using a second trained neural network to produce an output frame, wherein the output frame is an approximation of the input frame.

135. The method of claim 134, wherein the information relating to the target values comprises the values of the target values.

136. The method of claim 134 or 135, wherein the information relating to the target values comprises the location of the target values.

137. The method of any one of claim 134 to 137, further comprising the step of modifying the latent representation based on the information relating to the target values prior to transmitting the latent representation to the second computer system.

138. The method of any one of claims 134 to 137, wherein the method further comprises the step of: before transmitting the latent representation at the first computer system, performing a transformation on the latent representation.

139. The method of claim 139, wherein the method further comprises the step of: after receipt of the latent representation at the second computer system, performing a corresponding transformation on the received latent representation.

140. The method of claim 138 or 139, wherein the modification of the latent representation or received latent representation is based on the transformation and / or the corresponding transformation.126141. The method of any one of claims 138 to 140, wherein the step of modifying the latent representation or received latent representation comprises modifying the transformation performed on the latent representation or received latent representation.

142. The method of claim 141, wherein the transformation and / or the corresponding transformation comprise a quantization process; and the modification to the transformation and / or the corresponding transformation comprises modifying the quantization process.

143. The method of claim 142, wherein the modification to the quantization process comprises changing the bin size of the quantization process.

144. The method of claim 143, wherein the modification to the quantization process comprises changing the quantization process to a round, floor or ceiling function.

145. The method of claim 143 or 144, wherein the quantization process before the modification comprises a round, floor or ceiling function.

146. A method for lossy frame or video encoding, transmission and decoding, the method comprising the steps of: receiving an input frame at a first computer system; encoding the input frame using a first trained neural network to produce a latent representation; encoding the latent representation using a second trained neural network to obtain a127hyper-latent representation; determining one or more target values of the hyper-latent representation; transmitting the latent representation, the hyper-latent representation and information relating to the target values to a second computer system; decoding the hyper-latent representation using a third trained neural network to obtain a decoded hyper-latent; modifying the decoded hyper- latent based on the information relating to the target values; and decoding the latent representation using a fourth trained neural network and the decoded hyper-latent to produce an output frame, wherein the output frame is an approximation of the input frame.

147. The method of claim 146, wherein the decoded hyper-latent comprises one or more parameters associated with the probability distribution of the latent representation.

148. The method of claim 147, wherein the decoded hyper- latent comprises one or more mean values, and the modification comprises changing one or more of the mean values.

149. The method of claim 147, wherein the decoded hyper-latent comprises one or more scale values, and the modification comprises changing one or more of the scale values.

150. A method for lossy frame or video encoding, transmission and decoding, the method comprising the steps of: receiving an input frame at a first computer system;128encoding the input frame using a first trained neural network to produce a latent representation; encoding the latent representation using a second trained neural network to obtain a hyper-latent representation; encoding the latent representation using a third trained neural network to obtain a hyper-hyper-latent representation; determining one or more target values of the hyper-hyper-latent representation; transmitting the latent representation, the hyper-latent representation, the hyper-hyper- representation and information relating to the target values to a second computer system; decoding the hyper- latent representation using a fourth trained neural network to obtain a decoded hyper-hyper-latent; modifying the decoded hyper-hyper-latent based on the information relating to the target values; decoding the hyper-latent representation using a fifth trained neural network and the decoded hyper-hyper-latent to obtain a decoded hyper-latent; and decoding the latent representation using a sixth trained neural network and the decoded hyper-latent to produce an output frame, wherein the output frame is an approximation of the input frame.

151. A method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input frame at a first computer system; determining a flow between a previous input frame and the input frame;encoding the input frame using a first trained neural network to produce a latent representation; encoding the flow using a second trained neural network to produce a latent flow representation; determining one or more target values of the latent flow representation; transmitting the latent representation, the latent flow representation and information relating to the target values to a second computer system; modifying the received latent flow representation based on the information relating to the target values; decoding the latent flow representation using a third trained neural network to obtain a received flow; and decoding the latent representation using a second trained neural network and the received flow to produce an output frame, wherein the output frame is an approximation of the input frame.

152. A method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input frame at a first computer system; encoding the input frame using a first trained neural network to produce a latent representation; determining one or more target values of the latent representation; decoding the latent representation using a second trained neural network to produce anoutput frame, wherein the output frame is an approximation of the input frame.; evaluating a function based on a difference between the output frame and the input frame; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps using a first set of input frames to produce a first trained neural network and a second trained neural network.

153. The method of claim 152, further comprising the step of, after the step of determining the one or more target values: modifying the latent representation based on the information relating to the target values.

154. The method of claim 152 or 153, wherein the function is additionally based on a property of the one or more target values.

155. The method of claim 154, wherein the function is based on the quantity of the one or more target values.

156. The method of claim 154 or 155, wherein the function is based on the magnitude of one or more of the one or more target values.

157. The method of any one of claims 154 to 156, wherein the function is based on the difference between each of the one or more target values and one or more set values.

158. A method for lossy image or video encoding and transmission, the method comprising the steps of:131receiving an input frame at a first computer system; encoding the input frame using a first trained neural network to produce a latent representation; determining one or more target values of the latent representation transmitting the latent representation and information relating to the target values.

159. A method for lossy image or video receipt and decoding, the method comprising the steps of: receiving a latent representation and information relating to one or more target values transmitted by a first computer system at a second computer system, the latent representation corresponding to an input frame; modifying the received latent representation based on the information relating to the target values; and decoding the latent representation using a second trained neural network to produce an output frame, wherein the output frame is an approximation of the input frame.

160. A data processing system configured to perform the method of any one of claims 134 to 151.

161. A data processing apparatus configured to perform the method of any one of claims 152 to 159.

162. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any one of claims 152 to 159.

163. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method any one of claims 152 to 159.

164. A method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input frame at a first computer system; encoding the input frame using a first trained neural network to produce a latent representation; wherein during the encoding step, a correction is applied to at least one of the values associated with the input frame; transmitting the latent representation to a second computer system; and decoding the latent representation using a second trained neural network to produce an output frame, wherein the output frame is an approximation of the input frame.

165. The method of claim 164, wherein the correction is applied to a feature map associated with the input frame 166. The method of claim 165, wherein a correction is performed for every feature map associated with the input frame.

167. The method of any one of claims 164 to 166, wherein the correction is based on the input to a layer of the first trained neural network and output of that layer.133168. The method of any one of claims 164 to 167 wherein the correction is based on the input frame and / or a previously received input frame.

169. The method of any one of claims 164 to 168 wherein the correction is based on the latent representation and / or a previously produced latent representation.

170. The method of any one of claims 164 to 169 wherein the correction is based on the output frame and / or a previously decoded output frame.

171. The method of any one of claims 164 to 170, wherein the correction is based on one or more probability distribution parameters obtained from the input frame and / or a previous input frame.

172. The method of any one of claims 164 to 171, wherein the correction comprises adding a correction value at least one of the values associated with the input frame.

173. The method of any one of claims 164 to 172, wherein the correction comprises replacing at least one of the values associated with the input frame with a correction value.

174. The method of any one of claim 172 or 173, wherein the correction value is the output of a trained neural network.

175. The method of claim 174, wherein the trained neural network receives one or more of the following as an input: the input frame, a previously received input frame, the latent representation, a previously produced latent representation, the output frame, a previously134decoded output frame, one or more probability distribution parameters obtained from the input frame and / or a previous input frame, and one or more of the feature maps associated with the input frame and / or a previous input frame.

176. The method of any one of claims 164 to 175, wherein the method further comprises the step of: receiving a flag from the second computer system; and the correction is based on the received flag.

177. The method of claim 176, wherein the flag indicates the hardware used to perform the decoding step at the second computer system.

178. A method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input frame at a first computer system; encoding the input frame using a first trained neural network to produce a latent representation; transmitting the latent representation to a second computer system; and decoding the latent representation using a second trained neural network to produce an output frame, wherein the output frame is an approximation of the input frame; wherein during the decoding step, a correction is applied to at least one of the values associated with the latent representation.135179. A method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input frame at a first computer system; encoding the input frame using a first trained neural network to produce a latent representation; transmitting the latent representation to a second computer system; and decoding the latent representation using a second trained neural network to produce an output frame, wherein the output frame is an approximation of the input frame wherein a flag from the second computer system is additionally received at the first computer system prior to the encoding step; and based on the flag, at least one of the transformations performed by the first trained neural network is modified before the encoding step is performed.

180. The method of claim 179, wherein the modified transformation is replaced by a different transformation.

181. The method of claim 179 or 180, wherein the modified transformation is a convolution.

182. The method of claim 179 or 180, wherein the modified transformation is an activation function.

183. The method of claim any one of claims 179 to 182, wherein the flag indicates the hardware used to perform the decoding step at the second computer system.136184. A method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input frame at a first computer system; encoding the input frame using a first trained neural network to produce a latent representation; transmitting the latent representation to a second computer system; and decoding the latent representation using a second trained neural network to produce an output frame, wherein the output frame is an approximation of the input frame wherein a flag from the first computer system is additionally received at the second computer system prior to the decoding step; and based on the flag, at least one of the transformations performed by the second trained neural network is modified before the decoding step is performed.

185. A method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input frame at a first computer system; encoding the input frame using a first trained neural network to produce a latent representation; wherein during the encoding step, a correction is applied to at least one of the values associated with the input frame; transmitting the latent representation to a second computer system; and decoding the latent representation using a second trained neural network to produce anoutput frame, wherein the output frame is an approximation of the input frame; evaluating a function based on a difference between the output frame and the input frame; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps using a first set of input frames to produce a first trained neural network and a second trained neural network.

186. A method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input frame at a first computer system; encoding the input frame using a first trained neural network to produce a latent representation; transmitting the latent representation to a second computer system; and decoding the latent representation using a second trained neural network to produce an output frame, wherein the output frame is an approximation of the input frame; wherein during the decoding step, a correction is applied to at least one of the values associated with the latent representation; evaluating a function based on a difference between the output frame and the input frame; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps using a first set of input frames to produce a first trained neural network and a second trained neural network.

187. A method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input frame at a first computer system; encoding the input frame using a first trained neural network to produce a latent representation; transmitting the latent representation to a second computer system; and decoding the latent representation using a second trained neural network to produce an output frame, wherein the output frame is an approximation of the input frame wherein a flag from the second computer system is additionally received at the first computer system prior to the encoding step; and based on the flag, at least one of the transformations performed by the first trained neural network is modified before the encoding step is performed; evaluating a function based on a difference between the output frame and the input frame; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps using a first set of input frames to produce a first trained neural network and a second trained neural network.

188. A method of training one or more neural networks, the one or more neural networks being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving an input frame at a first computer system;encoding the input frame using a first trained neural network to produce a latent representation; transmitting the latent representation to a second computer system; and decoding the latent representation using a second trained neural network to produce an output frame, wherein the output frame is an approximation of the input frame wherein a flag from the first computer system is additionally received at the second computer system prior to the decoding step; and based on the flag, at least one of the transformations performed by the second trained neural network is modified before the decoding step is performed; evaluating a function based on a difference between the output frame and the input frame; updating the parameters of the first neural network and the second neural network based on the evaluated function; and repeating the above steps using a first set of input frames to produce a first trained neural network and a second trained neural network.

189. A data processing system configured to perform the method of any one of claims 164 to184.140

Images