Method and data processing system for lossy image or video encoding, transmission and decoding

Abstract

A method for lossy image or video encoding, transmission and decoding, the method comprising the steps of: receiving a first image at a first computer system; encoding the first image using a first neural network to produce a latent representation; at least partially masking the latent representation; transmitting the masked latent representation to a second computer system; decoding the masked latent representation using a second neural network to produce an output image, wherein the output image is an approximation of the first image.

Description

[0001] Method and data processing system for lossy image or videoencoding, transmission and decodingBACKGROUNDThis invention relates to a method and system for lossy image or video encoding, transmissionand decoding, a method, apparatus, computer program and computer readable storage mediumfor lossy image or video encoding and transmission, and a method, apparatus, computerprogram and computer readable storage medium for lossy image or video receipt and decoding.There is increasing demand from users of communications networks for images and videocontent. Demand is increasing not just for the number of images viewed, and for the playingtime of video; demand is also increasing for higher resolution content. This places increasingdemand on communications networks and increases their energy use because of the largeramount of data being transmitted.To reduce the impact of these issues, image and video content is compressed for transmissionacross the network. The compression of image and video content can be lossless or lossycompression. In lossless compression, the image or video is compressed such that all of theoriginal information in the content can be recovered on decompression. However, when usinglossless compression there is a limit to the reduction in data quantity that can be achieved. Inlossy compression, some information is lost from the image or video during the compressionprocess. Known compression techniques attempt to minimise the apparent loss of informationby the removal of information that results in changes to the decompressed image or video thatis not particularly noticeable to the human visual system. JPEG, JPEG2000, AVC, HEVC andAVI are examples of compression processes for image and / or video files.In general terms, known lossy image compression techniques use the spatial correlationsbetween pixels in images to remove redundant information during compression. For example,in an image of a blue sky, if a given pixel is blue, there is a high likelihood that the neighbouringpixels, and their neighbouring pixels, and so on, are also blue. There is accordingly no need toretain all the raw pixel data. Instead, we can retain only a subset of the pixels which take upfewer bits and infer the pixel values of the other pixels using information derived from spatialcorrelations.A similar approach is applied in known lossy video compression techniques. That is, spatialcorrelations between pixels allow the removal of redundant information during compression.However, in video compression, there is further information redundancy in the form of temporalcorrelations. For example, in a video of an aircraft flying across a blue-sky background, mostof the pixels of the blue sky do not change at all between frames of the video. The mostof the blue sky pixel data for the frame at position t = 0 in the video is identical to that atposition t = 10. Storing this identical, temporally correlated, information is inefficient. Instead,only the blue sky pixel data for a subset of the frames is stored and the rest are inferred frominformation derived from temporal correlations.In the realm of lossy video compression in particular, the removal of redundant temporallycorrelated information in a video sequence is known inter-frame redundancy.One technique using inter-frame redundancy that is widely used in standard video compressionalgorithms involves the categorization of video frames into three types: I-frames, P-frames, andB-frames. Each frame type carries distinct properties concerning their encoding and decodingprocess, playing different roles in achieving high compression ratios while maintainingacceptable visual quality.I-frames, or intra-coded frames, serve as the foundation of the video sequence. These framesare self-contained, each one encoding a complete image without reference to any other frame.In terms of compression, I-frames are least compressed among all frame types, thus carryingthe most data. However, their independence provides several benefits, including being thestarting point for decompression and enabling random access, crucial for functionalities likefast-forwarding or rewinding the video.P-frames, or predictive frames, utilize temporal redundancy in video sequences to achievegreater compression. Instead of encoding an entire image like an I-frame, a P-frame representsthe difference between itself and the closest preceding I- or P-frame. The process, known asmotion compensation, identifies and encodes only the changes that have occurred, therebysignificantly reducing the amount of data transmitted. Nonetheless, P-frames are dependent onprevious frames for decoding. Consequently, any error during the encoding or transmissionprocess may propagate to subsequent frames, impacting the overall video quality.B-frames, or bidirectionally predictive frames, represent the highest level of compression.Unlike P-frames, B-frames use both the preceding and following frames as references in theirencoding process. By predicting motion both forwards and backwards in time, B-framesencode only the differences that cannot be accurately anticipated from the previous and nextframes, leading to substantial data reduction. Although this bidirectional prediction makesB-frames more complex to generate and decode, it does not propagate decoding errors sincethey are not used as references for other frames. Artificial intelligence (AI) based compressiontechniques achieve compression and decompression of images and videos through the use oftrained neural networks in the compression and decompression process. Typically, duringtraining of the neutral networks, the difference between the original image and video and thecompressed and decompressed image and video is analyzed and the parameters of the neuralnetworks are modified to reduce this difference while minimizing the data required to transmitthe content. However, AI based compression methods may achieve poor compression resultsin terms of the appearance of the compressed image or video or the amount of informationrequired to be transmitted.An example of an AI based image compression process comprising a hyper-network is describedin Ballé, Johannes, et al. “Variational image compression with a scale hyperprior.” arXivpreprint arXiv:1802.01436 (2018), which is hereby incorporated by reference.An example of an AI based video compression approach is shown in Agustsson, E., Minnen, D.,Johnston, N., Balle, J., Hwang, S. J., and Toderici, G. (2020), Scale-space flow for end-to-endoptimized video compression. In Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition (pp. 8503-8512), which is hereby incorporated by reference.A further example of an AI based video compression approach is shown in Mentzer, F.,Agustsson, E., Ballé, J., Minnen, D., Johnston, N., and Toderici, G. (2022, November). Neuralvideo compression using gans for detail synthesis and propagation. In Computer Vision–ECCV2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, PartXXVI (pp. 562-578), which is hereby incorporated by reference.SUMMARYAccording to an aspect of the present disclosure, there is provided a method for lossy imageor video encoding, transmission, and decoding. The method comprises the step of receivinga first image and a second image at a first computer system. In addition, the method furthercomprises the step of producing optical flow information using the first image and the secondimage with a first neural network. The optical flow information is indicative of a differencebetween the first image and the second image. The method also comprises the step of producingresidual information using the first image and the optical flow information with a second neuralnetwork. The residual information is indicative of a difference between the first image andthe second image. Furthermore, the method comprises the step of transmitting the residualinformation to a second computer system. The method further comprises the step of decodingthe residual information to produce an intermediate output image with a third neural network.Lastly, the method comprises the step of combining a portion of a previously decoded imagewith a portion of the intermediate output image to produce a final output image. The finaloutput image is an approximation of the first image.Optionally, the method comprises extracting information from one or more intermediate layersof the first neural network. The portions, as described above, are based on the extractedinformation.Optionally, the method as described above may involve the extraction of information. Theextracted information comprises one or more outputs. These outputs are derived from one ormore convolution layers and / or activation layers of the first neural network.Optionally, the method comprises estimating the portions. The estimation of the portions isachieved by applying a mask. The mask is applied to either the previously decoded image orthe intermediate output image, or to both.Optionally, the method, as described above, further comprises a mask. The mask is based onthe extracted information.Optionally, the method may involve applying the mask. In this context, the mask is applied toone of the previously decoded image or intermediate output image. Additionally, an inverse ofthe mask is applied to the other of the previously decoded image or intermediate output.Optionally, the method, as described above, comprises a mask. The mask in the methodcomprises a non-binary mask.Optionally, the method comprises producing the mask. The production of the mask is facilitatedusing the third neural network, as described above.Optionally, the method further comprises producing the mask using the third neural network.In this context, producing the mask involves applying one or more convolution layers and / oractivation layers to the residual information and the extracted information.Optionally, the method comprises producing the mask. The production of the mask is achievedusing a fourth neural network.Optionally, the method further comprises producing the mask using the fourth neural network.In this context, producing the mask involves applying one or more convolution layers and / oractivation layers to the residual information and the extracted information.Optionally, the method may involve the use of a mask. The mask specifies an extent to whichthe previously decoded image contributes to the final output image. This contribution iscompared to the extent to which the intermediate output image contributes to the final outputimage.Optionally, the method may involve the application of the mask. The mask is applied to thepreviously decoded image and / or the intermediate output image on a pixel-wise basis. Theapplication of the mask serves to perform a pixel-wise, partial frame skip of the first image.Optionally, the method involves decoding the residual information with the third neural network.In this context, the decoding process utilizes the optical flow information that is output by thefirst neural network.Optionally, the method as described above may involve the use of a previously decoded image.In this context, the previously decoded image is an approximation of the second image.Optionally, the method comprises repeating the steps of producing, transmitting, decoding,and combining for a sequence of frames of a video. The sequence of frames is reconstructedat the second computer system. The repetition of these steps serves to mitigate image artefactsin the reconstructed sequence of frames.According to an aspect of the present disclosure, there is provided a data processing system.The data processing system is configured to perform the method as described above.According to an aspect of the present disclosure, there is provided a method for lossy imageor video receipt and decoding. The method comprises the step of decoding optical flowinformation at a first computer system using a first neural network. The optical flow informationis indicative of a difference between a first image and the second image. The method furthercomprises the step of decoding residual information at the first computer system using asecond neural network to produce an intermediate output image. The residual informationis indicative of a difference between the first image and the second image. Additionally, themethod comprises the step of combining a portion of a previously decoded image with aportion of the intermediate output image to produce a final output image. The final outputimage is an approximation of the first image.Optionally, there is provided a data processing apparatus. The data processing apparatus isconfigured to perform the method as described above.Optionally, there is provided a computer program. The computer program comprisesinstructions. When the computer program is executed by a computer, the instructions causethe computer to carry out the method as described above.According to an aspect of the present disclosure, there is provided a computer-readablestorage medium. The computer-readable storage medium comprises instructions. When theseinstructions are executed by a computer, the computer carries out the method as describedabove.According to an aspect of the present disclosure, there is provided a method of training oneor more neural networks, the one or more neural networks being for use in lossy image orvideo encoding, transmission and decoding, the method comprising the steps of: receivinga first image and a second image at a first computer system; with a first neural network,producing optical flow information using the first image and the second image, the optical flowinformation being indicative of a difference between the first image and the second image; witha second neural network, producing residual information using the first image and the opticalflow information, the residual information being indicative of a difference between the firstimage and the second image; with a third neural network, decoding the residual informationto produce an intermediate output image; and combining a portion of a previously decodedimage with a portion of the intermediate output image to produce a final output image, whereinthe final output image is an approximation of the first image; evaluating a function based ona difference between the first image and the output image; updating the parameters of thefirst, second and / or third neural networks based on the evaluated function; and repeating theabove steps using a first set of input images to produce first, second and / or third trained neuralnetworks.According to an aspect of the present disclosure, there is provided a method for lossy imageor video encoding, transmission, and decoding. The method comprises the step of receivinga first image and a second image at a first computer system. In addition, the method furthercomprises the step of producing optical flow information using the first image and the secondimage with a first neural network. The optical flow information is indicative of a differencebetween the first image and the second image. The method also comprises the step of extractinginformation from one or more intermediate layers of the first neural network. Furthermore, themethod comprises the step of producing residual information using the first image and theoptical flow information with a second neural network. The residual information is indicativeof a difference between the first image and the second image. Moreover, the method comprisesthe step of transmitting the residual information to a second computer system. The methodfurther comprises the step of decoding the residual information to produce an intermediateoutput image with a third neural network. Lastly, the method comprises the step of modifyingthe intermediate output image, based on the information extracted from the one or moreintermediate layers, to produce a modified output image. The modified output image is anapproximation of the first image.Optionally, the method comprises using the extracted outputs from the one or more intermediatelayers. These extracted outputs are utilized to produce a mask.Optionally, the method may involve modifying the intermediate output image. As describedabove, this modification process comprises applying the mask to the intermediate output image.Optionally, the method as described above may involve a mask. The mask comprises at leastthree different values. These values are situated between a minimum value and a maximumvalue.Optionally, the method comprises producing the mask. The mask is produced with one ormore layers of the third neural network.Optionally, the method comprises producing the mask with a fourth neural network, asdescribed above.Optionally, the method may involve the extraction of information from one or more intermediatelayers of the first neural network. The information extracted from these intermediate layerscomprises one or more outputs. These outputs are derived from one or more convolutionlayers and / or activation layers of the first neural network.Optionally, the method comprises repeating the steps, as described above, for a sequence offrames of a video. The repetition of these steps serves to reconstruct the sequence of frames ofthe video at the second computer system.Optionally, the method may involve repeating the steps for a sequence of frames of a video.The repetition of these steps serves to mitigate image artefacts in the reconstructed sequenceof frames.Optionally, there is provided a data processing system. The data processing system is configuredto perform the method as described above.According to an aspect of the present disclosure, there is provided a method for lossy imageor video receipt and decoding. The method comprises the step of decoding optical flowinformation at a first computer system using a first neural network. The optical flow informationis indicative of a difference between a first image and the second image. The method furthercomprises the step of extracting information from one or more intermediate layers of the firstneural network. Additionally, the method comprises the step of decoding residual informationat the first computer system using a second neural network to produce an intermediate outputimage. The residual information is indicative of a difference between the first image andthe second image. Moreover, the method comprises the step of modifying the intermediateoutput image. The modification is based on the information extracted from the one or moreintermediate layers of the first neural network. The modification produces a modified outputimage. The modified output image is an approximation of the first image.Optionally, there is provided a data processing apparatus. The data processing apparatus isconfigured to perform the method as described above.Optionally, there is provided a computer program. The computer program comprisesinstructions. When the computer program is executed by a computer, the instructions causethe computer to carry out the method as described above.Optionally, there is provided a computer-readable storage medium. The computer-readablestorage medium comprises instructions. When these instructions are executed by a computer,they cause the computer to carry out the method as described above.According to an aspect, there is provided a method for lossy image or video encoding,transmission and decoding, the method comprising the steps of:receiving a first image at a first computer system;encoding the first image using a first neural network to produce a latent representation;at least partially masking the latent representation;transmitting the masked latent representation to a second computer system;decoding the masked latent representation using a second neural network to produce anoutput image, wherein the output image is an approximation of the first image.Optionally, the at least partially masking the latent representation reduces the entropy of thedistribution of the latent representation.Optionally, said masking comprises producing a mask based on the first image.Optionally, the method comprises producing the mask with a third neural network.Optionally, the method comprises entropy encoding the masked latent representation into abitstream, wherein transmitting the masked latent representation comprises transmitting thebitstream, and wherein the step of at least partially masking the latent representation reduces asize of the bitstream after said entropy encoding compared to a bitstream produced without thestep of at least partially masking.Optionally, the at least partially masking the latent representation comprises at least partiallymasking a normalised representation of the latent representation.Optionally, the method comprises producing the normalised representation of the latentrepresentation using one or more first learned scale and / or zero point parameters.Optionally, the method comprises encoding the latent representation using a fourth neuralnetwork to produce a hyper latent representation, and decoding the hyper latent representationusing a fifth neural network to produce the one or more first learned scale and / or zero pointparameters.Optionally, the method comprises at least partially masking the hyper latent representation toreduce an entropy of a distribution of the hyper latent representation.Optionally, the method comprises entropy encoding and transmitting the masked hyper latentrepresentation to the second computer system.Optionally, the at least partially masking the hyper latent representation comprises at leastpartially masking a normalised representation of the hyper latent representation.Optionally, the method comprises producing the normalised representation of the hyper latentrepresentation using one or more second learned scale and / or zero point parameters.Optionally, the method comprises encoding the hyper latent representation using a sixth neuralnetwork to produce a hyper hyper latent representation, and decoding the hyper hyper latentrepresentation using a seventh neural network to produce the second learned scale and / or zeropoint parameters.Optionally, the method comprises at least partially masking the hyper hyper latent representationto reduce an entropy of a distribution of the hyper hyper latent representation.Optionally, the at least partially masking the hyper hyper latent representation comprises atleast partially masking a normalised representation of the hyper hyper latent representation.Optionally, the method comprises entropy encoding and transmitting the masked hyper hyperlatent representation to the second computer system.According to an aspect, there is provided a method for lossy image or video encoding andtransmission, the method comprising the steps of:receiving a first image at a first computer system;encoding the first image using a first neural network to produce a latent representation;at least partially masking the latent representation;transmitting the masked latent representation to a second computer system.According to an aspect, there is provided a method for lossy image or video receipt anddecoding, the method comprising the steps of:receiving at a second computer system a masked latent representation produced byencoding a first image using a first neural network and by at least partially masking the latentrepresentation; decoding the masked latent representation using a second neural network to produce anoutput image, wherein the output image is an approximation of the first image.According to an aspect, there is provided a data processing apparatus configured to performany of the above methods.According to an aspect, there is provided a computer program comprising instructions which,when the program is executed by a computer, cause the computer to carry out any of the abovemethods.According to an aspect, there is provided a computer-readable storage medium comprisinginstructions which, when executed by a computer, cause the computer to carry out any of theabove methods.According to aspect, there is provided a method for lossy image or video encoding, transmission,and decoding, the method comprising the steps of:receiving a first image and a second image at a first computer system;with a first neural network, producing optical flow information using the first image andthe second image, the optical flow information being indicative of a difference between thefirst image and the second image;with a second neural network, producing residual information using the first image andthe optical flow information, the residual information being indicative of a difference betweenthe first image and the second image;at least partially masking the residual information;transmitting the masked residual information to a second computer system;with a third neural network, decoding the masked residual information to produce outputimage wherein the output image is an approximation of the first image.Optionally, the at least partially masking the residual information reduces the entropy of thedistribution of the the residual information.Optionally, said masking comprises producing a mask based on the first image and / or thesecond image.Optionally, the method comprises producing the mask with a fourth neural network.Optionally, the method comprises entropy encoding the masked residual information into abitstream, wherein transmitting the masked residual information comprises transmitting thebitstream, and wherein the step of at least partially masking the residual information reduces asize of the bitstream after said entropy encoding compared to a bitstream produced without thestep of at least partially masking.Optionally, the at least partially masking the residual information comprises at least partiallymasking a normalised representation of the residual information.Optionally, the method comprises producing the normalised representation of the residualinformation using one or more first learned scale and / or zero point parameters.Optionally, the method comprises encoding the residual information using a fifth neuralnetwork to produce a hyper latent representation of the residual information, and decoding thehyper latent representation of the residual information using a sixth neural network to producethe one or more first learned scale and / or zero point parameters.Optionally, the method comprises at least partially masking the hyper latent representation ofthe residual information to reduce an entropy of a distribution of the hyper latent representationof the residual information.Optionally, the method comprises entropy encoding and transmitting the masked hyper latentrepresentation of the residual information to the second computer system.Optionally, the at least partially masking the hyper latent representation of the residualinformation comprises at least partially masking a normalised representation of the hyperlatent representation of the residual information.Optionally, the method comprises producing the normalised representation of the hyper latentrepresentation of the residual information using one or more second learned scale and / or zeropoint parameters.Optionally, the method comprises encoding the hyper latent representation of the residualinformation using a seventh neural network to produce a hyper hyper latent representation ofthe residual information, and decoding the hyper hyper latent representation of the residualinformation using an eighth neural network to produce the second learned scale and / or zeropoint parameters.Optionally, the method comprises at least partially masking the hyper hyper latent representationof the residual information to reduce an entropy of a distribution of the hyper hyper latentrepresentation of the residual information.Optionally, the at least partially masking the hyper hyper latent representation of the residualinformation comprises at least partially masking a normalised representation of the hyperhyper latent representation of the residual information.Optionally, the method comprises entropy encoding and transmitting the masked hyper hyperlatent representation of the residual information to the second computer system.According to an aspect, there is provided a method for lossy image or video encoding andtransmission, the method comprising the steps of:receiving a first image and a second image at a first computer system;with a first neural network, producing optical flow information using the first image andthe second image, the optical flow information being indicative of a difference between thefirst image and the second image;with a second neural network, producing residual information using the first image andthe optical flow information, the residual information being indicative of a difference betweenthe first image and the second image;at least partially masking the residual information; andtransmitting the masked residual information to a second computer system.According to an aspect, there is provided a method for lossy image or video receit and decoding,the method comprising the steps of:receiving at a second computer system masked residual information produced by, with afirst neural network, producing optical flow information using first and second images and by,with a second neural network, producing residual information using the first image and theoptical flow information, the residual information being indicative of a difference between thefirst image and the second image, and by at least partially masking the residual information;and with a third neural network, decoding the masked residual information to produce outputimage wherein the output image is an approximation of the first image.According to an aspect, there is provided a data processing apparatus configured to performany of the above methods.According to an aspect, there is provided a computer program comprising instructions which,when the program is executed by a computer, cause the computer to carry out any of the abovemethods.According to an aspect, there is provided a computer-readable storage medium comprisinginstructions which, when executed by a computer, cause the computer to carry out any of theabove methods.According to an aspect, there is provided amethod for lossy image or video encoding,transmission, and decoding, the method comprising the steps of:receiving a first image and a second image of an image sequence at a first computersystem; with a first neural network, producing optical flow information using the first image andthe second image, the optical flow information being indicative of a difference between thefirst image and the second image;at least partially masking the optical flow information;transmitting a representation of the masked optical flow information to a second computersystem; with a second neural network decoding the representation of the masked optical flowinformation to produce an approximation of the optical flow information; andwith a third neural network, producing an output image using the optical flow information,wherein the output image is an approximation of the first image.Optionally, the at least partially masking the optical flow information reduces the entropy ofthe distribution of the the optical flow information.Optionally, said masking comprises producing a mask based on the first image and / or thesecond image.Optionally, the method comprises producing the mask with a fourth neural network.Optionally, the method comprises entropy encoding the masked optical flow information into abitstream, wherein transmitting the masked optical flow information comprises transmittingthe bitstream, and wherein the step of at least partially masking the optical flow informationreduces a size of the bitstream after said entropy encoding compared to a bitstream producedwithout the step of at least partially masking.Optionally, the at least partially masking the optical flow information comprises at leastpartially masking a normalised representation of the optical flow information.Optionally, the method comprises producing the normalised representation of the optical flowinformation using one or more first learned scale and / or zero point parameters.Optionally, the method comprises encoding the optical flow information using a fifth neuralnetwork to produce a hyper latent representation of the optical flow information, and decodingthe hyper latent representation of the optical flow information using a sixth neural network toproduce the one or more first learned scale and / or zero point parameters.Optionally, the method comprises at least partially masking the hyper latent representationof the optical flow information to reduce an entropy of a distribution of the hyper latentrepresentation of the optical flow information.Optionally, the method comprises entropy encoding and transmitting the masked hyper latentrepresentation of the optical flow information to the second computer system.Optionally, the at least partially masking the hyper latent representation of the optical flowinformation comprises at least partially masking a normalised representation of the hyperlatent representation of the optical flow information.Optionally, the method comprises producing the normalised representation of the hyper latentrepresentation of the optical flow information using one or more second learned scale and / orzero point parameters.Optionally, the method comprises encoding the hyper latent representation of the optical flowinformation using a seventh neural network to produce a hyper hyper latent representation ofthe optical flow information, and decoding the hyper hyper latent representation of the opticalflow information using an eighth neural network to produce the second learned scale and / orzero point parameters.Optionally, the method comprises at least partially masking the hyper hyper latent representationof the optical flow information to reduce an entropy of a distribution of the hyper hyper latentrepresentation of the optical flow information.Optionally, the at least partially masking the hyper hyper latent representation of the opticalflow information comprises at least partially masking a normalised representation of the hyperhyper latent representation of the optical flow information.Optionally, the method comprises entropy encoding and transmitting the masked hyper hyperlatent representation of the optical flow information to the second computer system.According to an aspect, there is provided a method for lossy image or video encoding andtransmission, the method comprising the steps of:receiving a first image and a second image of an image sequence at a first computersystem; with a first neural network, producing optical flow information using the first image andthe second image, the optical flow information being indicative of a difference between thefirst image and the second image;at least partially masking the optical flow information; andtransmitting a representation of the masked optical flow information to a second computersystem.According to an aspect, there is provided a method for lossy image or video receipt anddecoding, the method comprising the steps of:receiving at a second computer system a representation of masked optical flow informationproduced with a first neural network using a first image and second image, and produced by atleast partially masking the optical flow information;with a second neural network decoding the representation of the masked optical flowinformation to produce an approximation of the optical flow information; andwith a third neural network, producing an output image using the optical flow information,wherein the output image is an approximation of the first image.According to an aspect, there is provided a data processing apparatus configured to performany of the above methods.According to an aspect, there is provided a computer program comprising instructions which,when the program is executed by a computer, cause the computer to carry out any of the abovemethods.According to an aspect, there is provided a computer-readable storage medium comprisinginstructions which, when executed by a computer, cause the computer to carry out any of theabove methods.According to an aspect, there is provided a method for lossy image or video encoding,transmission, and decoding, the method comprising the steps of:receiving a first image and a second image of an image sequence at a first computersystem; with a first neural network, producing optical flow information using the first image andthe second image, the optical flow information being indicative of a difference between thefirst image and the second image;transmitting a representation of the optical flow information to a second computer system;with a second neural network decoding the representation of the optical flow informationto produce an approximation of the optical flow information;warping a previously decoded representation of the second image using the approximationof the optical flow information;combining the warped previously decoded representation of the second image with arepresentation of the first image; andwith a third neural network, producing an output image using the combined, warpedpreviously decoded representation of the second image, wherein the output image is anapproximation of the first image.Optionally, the method comprises producing the representation of the first image with thesecond neural network.Optionally, the method comprises producing a mask with the second neural network, and usingthe mask to combine the representation of the first image with the warped previously decodedrepresentation of the second image.Optionally, the method comprises producing the representation of the first image with a fourthneural network using the first image.Optionally, the method comprises producing a mask with the fourth neural network usingthe first image, and using the mask to combine the representation of the first image with thewarped previously decoded representation of the second image.Optionally, said combining comprises performing a weighted sum of the representation ofthe first image and the warped previously decoded representation of the second image usingweights based on values of the mask.Optionally, said combining reduces a difference between (i) the warped previously decodedrepresentation of the second image and (ii) the first image.Optionally, said combining reduces a difference between (i) the output image produced withthe third neural network using the combined, warped previously decoded representation of thesecond image and (ii) the first image.Optionally, said combining corrects artefacts in the warped previously decoded representationof the second image.According to an aspect, there is provided a method for lossy image or video encoding andtransmission, the method comprising the steps of:receiving a first image and a second image of an image sequence at a first computersystem; with a first neural network, producing optical flow information using the first image andthe second image, the optical flow information being indicative of a difference between thefirst image and the second image; andtransmitting a representation of the optical flow information to a second computer system.According to an aspect, there is provided a method for lossy image or video receipt anddecoding, the method comprising the steps of:receiving a representation of optical flow information at a second computer system, theoptical flow information produced by a first neural network using a first image and a secondimage of an image sequence and being indicative of a difference between the first image andthe second image;with a second neural network decoding the representation of the optical flow informationto produce an approximation of the optical flow information;warping a previously decoded representation of the second image using the approximationof the optical flow information;combining the warped previously decoded representation of the second image with arepresentation of the first image; andwith a third neural network, producing an output image using the combined, warpedpreviously decoded representation of the second image, wherein the output image is anapproximation of the first image.According to an aspect, there is provided a method of training one or more neural networks,the one or more neural networks being for use in lossy image or video encoding, transmissionand decoding, the method comprising the steps of:receiving a first image and a second image of an image sequence at a first computersystem; with a first neural network, producing optical flow information using the first image andthe second image, the optical flow information being indicative of a difference between thefirst image and the second image;transmitting a representation of the optical flow information to a second computer system;with a second neural network decoding the representation of the optical flow informationto produce an approximation of the optical flow information;warping a previously decoded representation of the second image using the approximationof the optical flow information;combining the warped previously decoded representation of the second image with arepresentation of the first image;with a third neural network, producing an output image using the combined, warpedpreviously decoded representation of the second image, wherein the output image is anapproximation of the first image;evaluating a function based on a difference between the first image and the output image;updating the parameters of the first, second and / or third neural networks based on theevaluated function; andrepeating the above steps using a first set of input images to produce first, second and / orthird trained neural networks.Optionally, the method comprises producing the representation of the first image with thesecond neural network.Optionally, the method comprises producing a mask with the second neural network, and usingthe mask to combine the representation of the first image with the warped previously decodedrepresentation of the second image.Optionally, the method comprises producing the representation of the first image with a fourthneural network using the first image.Optionally, the method comprises producing a mask with the fourth neural network usingthe first image, and using the mask to combine the representation of the first image with thewarped previously decoded representation of the second image.Optionally, the function is further based on a difference between (i) the representation of thefirst image produced by the second neural network or the fourth neural network and (ii) arepresentation of the first image produced by a teacher neural network.Optionally, the function is further based on a difference between (i) the mask produced bythe second neural network or the fourth neural network and (ii) a mask produced by a teacherneural network.Optionally, said evaluating the function comprises regularising propagation of gradientsthrough one or more terms of the function.Optionally, said evaluating the function comprises regularising propagation of gradientsthrough a term of the function associated with the difference between (i) the mask producedby the second neural network or the fourth neural network and (ii) the mask produced by theteacher neural network.Optionally, the method comprises controlling said regularising based on a predeterminedregularisation schedule.According to an aspect, there is provided a data processing apparatus configured to performany of the above methods.According to an aspect, there is provided a computer program comprising instructions which,when the program is executed by a computer, cause the computer to carry out any of the abovemethods.According to an aspect, there is provided a computer-readable storage medium comprisinginstructions which, when executed by a computer, cause the computer to carry out any of theabove methods.BRIEF DESCRIPTION OF THE DRAWINGSAspects of the invention will now be described by way of examples, with reference to thefollowing figures in which:Figure 1 illustrates an example of an image or video compression, transmission and decom-pression pipeline.Figure 2 illustrates a further example of an image or video compression, transmission anddecompression pipeline including a hyper-network.Figure 3 illustrates an example of a video compression, transmission and decompressionpipeline.Figure 4 illustrates an example of a video compression, transmission and decompressionsystem.Figure 5a illustrates an example of an output of an image or video compression, transmissionand decompression pipeline.Figure 5b illustrates an example of an output of an image or video compression, transmissionand decompression pipeline.Figure 6 illustrates an example of a video compression, transmission and decompressionpipeline.Figure 7 illustrates an example of a video compression, transmission and decompressionpipeline.Figure 8 illustrates an example of a video compression, transmission and decompressionpipeline.Figure 9 illustrates an example of encode-side components of a video compression, transmissionand decompression pipeline.Figure 10 illustrates an example of decode-side components of a video compression, transmis-sion and decompression pipeline.Figure 11 illustrates Visualisations of outputs of one or more networks of a video compressiontransmission and decompression pipeline.Figure 12 illustrates an example of a video compression, transmission and decompressionpipeline.Figure 13 illustrates an example of a video compression, transmission and decompressionpipeline.DETAILED DESCRIPTION OF THE DRAWINGSCompression processes may be applied to any form of information to reduce the amountof data, or file size, required to store that information. Image and video information is anexample of information that may be compressed. The file size required to store the information,particularly during a compression process when referring to the compressed file, may bereferred to as the rate. In general, compression can be lossless or lossy. In both forms ofcompression, the file size is reduced. However, in lossless compression, no information is lostwhen the information is compressed and subsequently decompressed. This means that theoriginal file storing the information is fully reconstructed during the decompression process.In contrast to this, in lossy compression information may be lost in the compression anddecompression process and the reconstructed file may differ from the original file. Image andvideo files containing image and video data are common targets for compression.In a compression process involving an image, the input image may be represented as ^^. Thedata representing the image may be stored in a tensor of dimensions ^^ × ^^ × ^^, where ^^represents the height of the image, ^^ represents the width of the image and ^^ represents thenumber of channels of the image. Each ^^ × ^^ data point of the image represents a pixel valueof the image at the corresponding location. Each channel ^^ of the image represents a differentcomponent of the image for each pixel which are combined when the image file is displayed bya device. For example, an image file may have 3 channels with the channels representing thered, green and blue component of the image respectively. In this case, the image informationis stored in the RGB colour space, which may also be referred to as a model or a format.Other examples of colour spaces or formats include the CMKY and the YCbCr colour models.However, the channels of an image file are not limited to storing colour information and otherinformation may be represented in the channels. As a video may be considered a series ofimages in sequence, any compression process that may be applied to an image may also beapplied to a video. Each image making up a video may be referred to as a frame of the video.The output image may differ from the input image and may be represented by ^^. The differencebetween the input image and the output image may be referred to as distortion or a differencein image quality. The distortion can be measured using any distortion function which receivesthe input image and the output image and provides an output which represents the differencebetween input image and the output image in a numerical way. An example of such a methodis using the mean square error (MSE) between the pixels of the input image and the outputimage, but there are many other ways of measuring distortion, as will be known to the personskilled in the art. The distortion function may comprise a trained neural network.Typically, the rate and distortion of a lossy compression process are related. An increase inthe rate may result in a decrease in the distortion, and a decrease in the rate may result in anincrease in the distortion. Changes to the distortion may affect the rate in a correspondingmanner. A relation between these quantities for a given compression technique may be definedby a rate-distortion equation.AI based compression processes may involve the use of neural networks. A neural network isan operation that can be performed on an input to produce an output. A neural network maybe made up of a plurality of layers. The first layer of the network receives the input. One ormore operations may be performed on the input by the layer to produce an output of the firstlayer. The output of the first layer is then passed to the next layer of the network which mayperform one or more operations in a similar way. The output of the final layer is the output ofthe neural network.Each layer of the neural network may be divided into nodes. Each node may receive at leastpart of the input from the previous layer and provide an output to one or more nodes in asubsequent layer. Each node of a layer may perform the one or more operations of the layer onat least part of the input to the layer. For example, a node may receive an input from one ormore nodes of the previous layer. The one or more operations may include a convolution, aweight, a bias and an activation function. Convolution operations are used in convolutionalneural networks. When a convolution operation is present, the convolution may be performedacross the entire input to a layer. Alternatively, the convolution may be performed on at leastpart of the input to the layer.Each of the one or more operations is defined by one or more parameters that are associatedwith each operation. For example, the weight operation may be defined by a weight matrixdefining the weight to be applied to each input from each node in the previous layer to eachnode in the present layer. In this example, each of the values in the weight matrix is a parameterof the neural network. The convolution may be defined by a convolution matrix, also knownas a kernel. In this example, one or more of the values in the convolution matrix may be aparameter of the neural network. The activation function may also be defined by values whichmay be parameters of the neural network. The parameters of the network may be varied duringtraining of the network.Other features of the neural network may be predetermined and therefore not varied duringtraining of the network. For example, the number of layers of the network, the number ofnodes of the network, the one or more operations performed in each layer and the connectionsbetween the layers may be predetermined and therefore fixed before the training process takesplace. These features that are predetermined may be referred to as the hyperparameters of thenetwork. These features are sometimes referred to as the architecture of the network.To train the neural network, a training set of inputs may be used for which the expected output,sometimes referred to as the ground truth, is known. The initial parameters of the neuralnetwork are randomized and the first training input is provided to the network. The output ofthe network is compared to the expected output, and based on a difference between the outputand the expected output the parameters of the network are varied such that the differencebetween the output of the network and the expected output is reduced. This process is thenrepeated for a plurality of training inputs to train the network. The difference between theoutput of the network and the expected output may be defined by a loss function. The result ofthe loss function may be calculated using the difference between the output of the networkand the expected output to determine the gradient of the loss function. Back-propagation ofthe gradient descent of the loss function may be used to update the parameters of the neuralnetwork using the gradients ^^^^ / ^^^^ of the loss function. A plurality of neural networks in asystem may be trained simultaneously through back-propagation of the gradient of the lossfunction to each network.In the context of image or video compression, this type of system, where simultaneous trainingwith back-propagation through each element or the whole network architecture may be referredto as end-to-end, learned image or video compression. Unlike in traditional compressionalgorithms that use primarily handcrafted, manually constructed steps, an end-to-end learnedsystem learns itself during training what combination of parameters best achieves the goal ofminimising the loss function. This approach is advantageous compared to systems that are notend-to-end learned because an end-to-end system has a greater flexibility to learn weights andparameters that might be counter-intuitive to someone handcrafting features.It will be appreciated that the term "training" or "learning" as used herein means the processof optimizing an artificial intelligence or machine learning model, based on a given set of data.This involves iteratively adjusting the parameters of the model to minimize the discrepancybetween the model’s predictions and the actual data, represented by the above-describedrate-distortion loss function.The training process may comprise multiple epochs. An epoch refers to one complete passof the entire training dataset through the machine learning algorithm. During an epoch, themodel’s parameters are updated in an effort to minimize the loss function. It is envisaged thatmultiple epochs may be used to train a model, with the exact number depending on variousfactors including the complexity of the model and the diversity of the training data.Within each epoch, the training data may be divided into smaller subsets known as batches.The size of a batch, referred to as the batch size, may influence the training process. A smallerbatch size can lead to more frequent updates to the model’s parameters, potentially leading tofaster convergence to the optimal solution, but at the cost of increased computational resources.Conversely, a larger batch size involves fewer updates, which can be more computationallyefficient but might converge slower or even fail to converge to the optimal solution.The learnable parameters are updated by a specified amount each time, determined by thelearning rate. The learning rate is a hyperparameter that decides how much the parametersare adjusted during the training process. A smaller learning rate implies smaller steps in theparameter space and a potentially more accurate solution, but it may require more epochs toreach that solution. On the other hand, a larger learning rate can expedite the training processbut may risk overshooting the optimal solution or causing the training process to diverge.The training described herein may involve use of a validation set, which is a portion of thedata not used in the initial training, which is used to evaluate the model’s performance and toprevent overfitting. Overfitting occurs when a model learns the training data too well, to thepoint that it fails to generalize to unseen data. Regularization techniques, such as dropout orL1 / L2 regularization, can also be used to mitigate overfitting.It will be appreciated that training a machine learning model is an iterative process thatmay comprise selection and tuning of various parameters and hyperparameters. As will beappreciated, the specific details, such as hyper parameters and so on, of the training processmay vary and it is envisaged that producing a trained model in this way may achieved in anumber of different ways with different epochs, batch sizes, learning rates, regularisations,and so on, the details of which are not essential to enabling the advantages and effects of thepresent disclosure, except where stated otherwise. The point at which an “untrained” neuralnetwork is considered be “trained” is envisaged to be case specific and depend on, for example,on a number of epochs, a plateauing of any further learning, or some other metric and is notconsidered to be essential in achieving the advantages described herein.More details of an end-to-end, learned compression process will now be described. It will beappreciated that in some cases, end-to-end, learned compression processes may be combinedwith one or more components that are handcrafted or trained separately.In the case of AI based image or video compression, the loss function may be defined by therate distortion equation. The rate distortion equation may be represented by ^^^^^^^^ = ^^ + ^^ ∗ ^^,where ^^ is the distortion function, ^^ is a weighting factor, and ^^ is the rate loss. ^^ may bereferred to as a lagrange multiplier. The langrange multiplier provides as weight for a particularterm of the loss function in relation to each other term and can be used to control which termsof the loss function are favoured when training the network.In the case of AI based image or video compression, a training set of input images maybe used. An example training set of input images is the KODAK image set (for exampleat www.cs.albany.edu / xypan / research / snr / Kodak.html). An example training set of inputimages is the IMAX image set. An example training set of input images is the Imagenetdataset (for example at www.image-net.org / download). An example training set of inputimages is the CLIC Training Dataset P (“professional”) and M (“mobile”) (for example athttp: / / challenge.compression.cc / tasks / ).An example of an AI based compression, transmission and decompression process 100 isshown in Figure 1. As a first step in the AI based compression process, an input image 5 isprovided. The input image 5 is provided to a trained neural network 110 characterized by afunction ^^^^ acting as an encoder. The encoder neural network 110 produces an output basedon the input image. This output is referred to as a latent representation of the input image 5. Ina second step, the latent representation is quantised in a quantisation process 140 characterisedby the operation ^^, resulting in a quantized latent. The quantisation process transforms thecontinuous latent representation into a discrete quantized latent. An example of a quantizationprocess is a rounding function.In a third step, the quantized latent is entropy encoded in an entropy encoding process 150 toproduce a bitstream 130. The entropy encoding process may be for example, range or arithmeticencoding. In a fourth step, the bitstream 130 may be transmitted across a communicationnetwork.In a fifth step, the bitstream is entropy decoded in an entropy decoding process 160. Thequantized latent is provided to another trained neural network 120 characterized by a function^^^^ acting as a decoder, which decodes the quantized latent. The trained neural network 120produces an output based on the quantized latent. The output may be the output image of theAI based compression process 100. The encoder-decoder system may be referred to as anautoencoder.Entropy encoding processes such as range or arithmetic encoding are typically able to losslesslycompress given input data up to close to the fundamental entropy limit of that data, as determinedby the total entropy of the distribution of that data. Accordingly, one way in which end-to-end,learned compression can minimise the rate loss term of the rate-distortion loss function andthereby increase compression effectiveness is to learn autoencoder parameter values thatproduce low entropy latent representation distributions. Producing latent representationsdistributed with as low an entropy as possible allows entropy encoding to compress the latentdistributions as close to or to the fundamental entropy limit for that distribution. The lowerthe entropy of the distribution, the more entropy encoding can losslessly compress it and thelower the amount of data in the corresponding bitstream. In some cases where the latentrepresentation is distributed according to a gaussian or Laplacian distribution, this learningmay comprise learning optimal location and scale parameters of the gaussian or Laplaciandistributions, in other cases, it allows the learning of more flexible latent representationdistributions which can further help to achieve the minimising of the rate-distortion lossfunction in ways that are not intuitive or possible to do with handcrafted features. Examples ofthese and other advantages are described in WO2021 / 220008A1, which is incorporated in itsentirety by reference.Something which is closely linked to the entropy encoding of the latent distribution and whichaccordingly also has an effect on the effectiveness of compression of end-to-end learnedapproaches is the quantisation step. During inference, a rounding function may be used toquantise a latent representation distribution into bins of given sizes, a rounding function isnot differentiable everywhere. Rather, a rounding function is effectively one or more stepfunctions whose gradient is either zero (at the top of the steps) or infinity (at the boundarybetween steps). Back propagating a gradient of a loss function through a rounding functionis challenging. Instead, during training, quantisation by rounding function is replaced byone or more other approaches. For example, the functions of a noise quantisation model aredifferentiable everywhere and accordingly do allow backpropagation of the gradient of theloss function through the quantisation parts of the end-to-end, learned system. Alternatively, astraight-through estimator (STE) quantisation model or one other quantisation models may beused. It is also envisaged that different quantisation models may be used for during evaluationof different term of the loss function. For example, noise quantisation may used to evaluate therate or entropy loss term of the rate-distortion loss function while STE quantisation may beused to evaluate the distortion term.In a similar manner to how learning parameters top produce certain distributions of the latentrepresentation facilitates achieving better rate loss term minimisation, end-to-end learning ofthe quantisation process achieves a similar effect. That is, learnable quantisation parametersprovide the architecture with a further degree of freedom to achieve the goal of minimising theloss function. For example, parameters corresponding to quantisation bin sizes may be learnedwhich is likely to result in an improved rate-distortion loss outcome compared to approachesusing hand-crafted quantisation bin sizes.Further, as the rate-distortion loss function constantly has to balance a rate loss term against adistortion loss term, it has been found that the more degrees of freedom the system has duringtraining, the better the architecture is at achieving optimal rate and distortion trade off.The system described above may be distributed across multiple locations and / or devices. Forexample, the encoder 110 may be located on a device such as a laptop computer, desktopcomputer, smart phone or server. The decoder 120 may be located on a separate device whichmay be referred to as a recipient device. The system used to encode, transmit and decode theinput image 5 to obtain the output image 6 may be referred to as a compression pipeline.The AI based compression process may further comprise a hyper-network 105 for thetransmission of meta-information that improves the compression process. The hyper-network105 comprises a trained neural network 115 acting as a hyper-encoder ^^ ℎ^^and a trained neuralnetwork 125 acting as a hyper-decoder ^^ℎ^^. An example of such a is shown in Figure 2.Components of the system not further discussed may be assumed to be the same as discussedabove. The neural network 115 acting as a hyper-decoder receives the latent that is the output ofthe encoder 110. The hyper-encoder 115 produces an output based on the latent representationthat may be referred to as a hyper-latent representation. The hyper-latent is then quantizedin a quantization process 145 characterised by ^^ℎ to produce a quantized hyper-latent. Thequantization process 145 characterised by ^^ℎ may be the same as the quantisation process 140characterised by ^^ discussed above.In a similar manner as discussed above for the quantized latent, the quantized hyper-latent isthen entropy encoded in an entropy encoding process 155 to produce a bitstream 135. Thebitstream 135 may be entropy decoded in an entropy decoding process 165 to retrieve thequantized hyper-latent. The quantized hyper-latent is then used as an input to trained neuralnetwork 125 acting as a hyper-decoder. However, in contrast to the compression pipeline 100,the output of the hyper-decoder may not be an approximation of the input to the hyper-decoder115. Instead, the output of the hyper-decoder is used to provide parameters for use in theentropy encoding process 150 and entropy decoding process 160 in the main compressionprocess 100. For example, the output of the hyper-decoder 125 can include one or more ofthe mean, standard deviation, variance or any other parameter used to describe a probabilitymodel for the entropy encoding process 150 and entropy decoding process 160 of the latentrepresentation. In the example shown in Figure 2, only a single entropy decoding process 165and hyper-decoder 125 is shown for simplicity. However, in practice, as the decompressionprocess usually takes place on a separate device, duplicates of these processes will be presenton the device used for encoding to provide the parameters to be used in the entropy encodingprocess 150.Further transformations may be applied to at least one of the latent and the hyper-latent at anystage in the AI based compression process 100. For example, at least one of the latent and thehyper latent may be converted to a residual value before the entropy encoding process 150,155is performed. The residual value may be determined by subtracting the mean value of thedistribution of latents or hyper-latents from each latent or hyper latent. The residual valuesmay also be normalised.To perform training of the AI based compression process described above, a training set ofinput images may be used as described above. During the training process, the parameters ofboth the encoder 110 and the decoder 120 may be simultaneously updated in each trainingstep. If a hyper-network 105 is also present, the parameters of both the hyper-encoder 115and the hyper-decoder 125 may additionally be simultaneously updated in each training step.The training process may further include a generative adversarial network (GAN). Whenapplied to an AI based compression process, in addition to the compression pipeline describedabove, an additional neutral network acting as a discriminator is included in the system. Thediscriminator receives an input and outputs a score based on the input providing an indicationof whether the discriminator considers the input to be ground truth or fake. For example, theindicator may be a score, with a high score associated with a ground truth input and a lowscore associated with a fake input. For training of a discriminator, a loss function is used thatmaximizes the difference in the output indication between an input ground truth and input fake.When a GAN is incorporated into the training of the compression process, the output image 6may be provided to the discriminator. The output of the discriminator may then be used in theloss function of the compression process as a measure of the distortion of the compressionprocess. Alternatively, the discriminator may receive both the input image 5 and the outputimage 6 and the difference in output indication may then be used in the loss function of thecompression process as a measure of the distortion of the compression process. Training ofthe neural network acting as a discriminator and the other neutral networks in the compressionprocess may be performed simultaneously. During use of the trained compression pipelinefor the compression and transmission of images or video, the discriminator neural network isremoved from the system and the output of the compression pipeline is the output image 6.Incorporation of a GAN into the training process may cause the decoder 120 to performhallucination. Hallucination is the process of adding information in the output image 6 thatwas not present in the input image 5. In an example, hallucination may add fine detail tothe output image 6 that was not present in the input image 5 or received by the decoder 120.The hallucination performed may be based on information in the quantized latent received bydecoder 120.Details of a video compression process will now be described. As discussed above, a video ismade up of a series of images arranged in sequential order. AI based compression process100 described above may be applied multiple times to perform compression, transmissionand decompression of a video. For example, each frame of the video may be compressed,transmitted and decompressed individually. The received frames may then be grouped toobtain the original video.The frames in a video may be labelled based on the information from other frames that is usedto decode the frame in a video compression, transmission and decompression process. Asdescribed above, frames which are decoded using no information from other frames may bereferred to as I-frames. Frames which are decoded using information from past frames may bereferred to as P-frames. Frames which are decoded using information from past frames andfuture frames may be referred to as B-frames. Frames may not be encoded and / or decoded inthe order that they appear in the video. For example, a frame at a later time step in the videomay be decoded before a frame at an earlier time.The images represented by each frame of a video may be related. For example, a number offrames in a video may show the same scene. In this case, a number of different parts of thescene may be shown in more than one of the frames. For example, objects or people in a scenemay be shown in more than one of the frames. The background of the scene may also beshown in more than one of the frames. If an object or the perspective is in motion in the video,the position of the object or background in one frame may change relative to the position ofthe object or background in another frame. The transformation of a part of the image froma first position in a first frame to a second position in a second frame may be referred to asflow, warping or motion compensation. The flow may be represented by a vector. One or moreflows that represent the transformation of at least part of one frame to another frame may bereferred to as a flow map.An example AI based video compression, transmission, and decompression process 200 isshown in Figure 3. The process 200 shown in Figure 3 is divided into an I-frame part 201for decompressing I-frames, and a P-frame part 202 for decompressing P-frames. It will beunderstood that these divisions into different parts are arbitrary and the process 200 may bealso be considered as a single, end-to-end pipeline.As described above, I-frames do not rely on information from other frames so the I-frame part201 corresponds to the compression, transmission, and decompression process illustrated inFigures 1 or 2. The specific details will not be repeated here but, in summary, an input image^^0 is passed into an encoder neural network 203 producing a latent representation which isquantised and entropy encoded into a bitstream 204. The subscript 0 in ^^0 indicates the inputimage corresponds to a frame of a video stream at position t = 0. This may be the first frame ofan entire video stream or the first frame of a chunk of a video stream made up of, for example,an I-frame and a plurality of subsequent P-frames and / or B-frames. The bitstream 204 is thenentropy decoded and passed into a decoder neural network 205 to reproduce a reconstructedimage ^^0 which in this case is an I-frame. The decoding step may be performed both locallyat the same location as where the input image compression occurs as well as at the locationwhere the decompression occurs. This allows the reconstructed image ^^0 to be available forlater use by components of both the encoding and decoding sides of the pipeline.In contrast to I-frames, P-frames (and B-frames) do rely on information from other frames.Accordingly, the P-frame part 202 at the encoding side of the pipeline takes as input not onlythe input image ^^^^ that is to be compressed (corresponding to a frame of a video stream atposition t), but also one or more previously reconstructed images ^^^^−1 from an earlier framet-1. As described above, the previously reconstructed ^^^^−1 is available at both the encodeand decode side of the pipeline and can accordingly be used for various purposes at both theencode and decode sides.At the encode side, previously reconstructed images may be used for generating a flow mapscontaining information indicative of inter-frame movement of pixels between frames. In theexample of Figure 3, both the image being compressed ^^^^ and the previously reconstructedimage from an earlier frame ^^^^−1 are passed into a flow module part 206 of the pipeline. Theflow module part 206 comprises an autoencoder such as that of the autoencoder systems ofFigures 1 and 2 but where the encoder neural network 207 has been trained to produce alatent representation of a flow map from inputs ^^^^−1 and ^^^^ , which is indicative of inter-framemovement of pixels or pixel groups between ^^^^−1 and ^^^^ . The latent representation of the flowmap is quantised and entropy encoded to compress it and then transmitted as a bitstream 208.On the decode side, the bitstream is entropy decoded and passed to a decoder neural network209 to produce a reconstructed flow map ^^ .The reconstructed flow map ^^ is applied to the previously reconstructed image ^^^^−1 to generatea warped image ^^^^−1,^^. It is envisaged that any suitable warping technique may be used, forexample bi-linear or tri-linear warping, as is described in Agustsson, E., Minnen, D., Johnston,N., Balle, J., Hwang, S. J., and Toderici, G. (2020), Scale-space flow for end-to-end optimizedvideo compression. In Proceedings of the IEEE / CVF Conference on Computer Vision andPattern Recognition (pp. 8503-8512), which is hereby incorporated by reference. It is furtherenvisaged that a scale-space flow approach as described in the above paper may also optionallybe used. The warped image ^^^^−1,^^ is a prediction of how the previously reconstructed image^^^^−1 might have changed between frame positions t-1 and t, based on the output flow mapproduced by the flow module part 206 autoencoder system from the inputs of ^^^^ and ^^^^−1.As with the I-frame, the reconstructed flow map ^^ and corresponding warped image ^^^^−1,^^may be produced both on the encode side and the decode side of the pipeline so they areavailable for use by other components of the pipeline on both the encode and decode sides.In the example of Figure 3, both the image being compressed ^^^^ and the ^^^^−1,^^ are passedinto a residual module part 210 of the pipeline. The residual module part 210 comprises anautoencoder system such as that of the autoencoder systems of Figures 1 and 2 but where theencoder neural network 211 has been trained to produce a latent representation of a residualmap indicative of differences between the input mage ^^^^ and the warped image ^^^^−1,^^. Thelatent representation of the residual map is then quantised and entropy encoded into a bitstream212 and transmitted. The bitstream 212 is then entropy decoded and passed into a decoderneural network 213 which reconstructs a residual map ^^ from the decoded latent representation.Alternatively, a residual map may first be pre-calculated between ^^^^ and the ^^^^−1,^^ and thepre-calculated residual map may be passed into an autoencoder for compression only. Thishand-crafted residual map approach is computationally simpler, but reduces the degrees offreedom with which the architecture may learn weights and parameters to achieve its goalduring training of minimising the rate-distortion loss function.Finally, on the decode side, the residual map ^^ is applied (e.g. combined by addition, subtractionor a different operation) to the warped image to produce a reconstructed image ^^^^ which is areconstruction of image ^^^^ and accordingly corresponds to a P-frame at position t in a sequenceof frames of a video stream. It will be appreciated that the reconstructed image ^^^^ can then beused to process the next frame. That is, it can be used to compress, transmit and decompress^^^^+1, and so on until an entire video stream or chunk of a video stream has been processed.Thus, for a block of video frames comprising an I-frame and ^^ subsequent P-frames, thebitstream may contain (i) a quantised, entropy encoded latent representation of the I-frameimage, and (ii) a quantised, entropy encoded latent representation of a flow map and residualmap of each P-frame image. For completeness, whilst not illustrated in Figure 3, any of theautoencoder systems of Figure 3 may comprise hyper and hyper-hyper networks such as thosedescribed in connection with Figure 2. Accordingly, the bitstream may also contain hyper andhyper-hyper parameters, their latent quantised, entropy encoded latent representations and soon, of those networks as applicable.Finally, the above approach may generally also be extended to B-frames, for example as isdescribed in Pourreza, R., and Cohen, T. (2021). Extending neural p-frame codecs for b-framecoding. In Proceedings of the IEEE / CVF International Conference on Computer Vision (pp.6680-6689).The above-described flow and residual based approach is highly effective at reducing theamount of data that needs to be transmitted because, as long as at least one reconstructed frame(e.g. I-frame ^^^^−1) is available, the encode side only needs to compress and transmit a flowmap and a residual map (and any hyper or hyper-hyper parameter information, as applicable)to reconstruct a subsequent frame.Figure 4 shows an example of an AI image or video compression process such as that describedabove in connection with Figures 1-3 implemented in a video streaming system 400. Thesystem 400 comprises a first device 401 and a second device 402. The first and seconddevices 401, 402 may be user devices such as smartphones, tablets, AR / VR headsets or otherportable devices. In contrast to known systems which primarily perform inference on GPUssuch as Nvidia A100, Geforce 3090, Gefore 4090 GPU cards, the system 400 of Figure 4performs inference on a CPU of the first and second devices respectively. That is, computefor performing both encoding and decoding are performed by the respective CPUs of the firstand second devices 401, 402. This places very different power usage, memory and runtimeconstraints on the implementation of the above methods than when implementing AI-basedcompression methods on GPUs. In one example, the CPU of first and second devices 401, 402may comprise a Qualcomm Snapdragon CPU.The first device 401 comprises a media capture device 403, such as a camera, arranged tocapture a plurality of images, referred to hereafter as a video stream 404, of a scene 404. Thevideo stream 404 is passed to a pre-processing module 406 which splits the video stream intoblocks of frames, various frames of which will be designated as I-frames, P-frames, and / orB-frames. The blocks of frames are then compressed by an AI-compression module 407comprising the encode side of the AI-based video compression pipeline of Figure 3. Theoutput of the AI-compression module is accordingly a bitstream 408a which is transmittedfrom the first device 401, for example via a communications channel, for example over oneor more of a WiFi, 3G, 4G or 5G channel, which may comprise internet or cloud-based 409communications.The second device 402 receives the communicated bitstream 408b which is passed to anAI-decompression module 410 comprising the decode side of the AI-based video compressionpipeline of Figure 3. The output of the AI-decompression module 402 is the reconstructedI-frames, P-frames and / or B-frames which are passed to a post-processing module 411 wherethey can prepared, for example passed into a buffer, in preparation for streaming 412 to andrendering on a display device 413 of the second device 402.It is envisaged that the system 400 of Figure 4 may be used for live video streaming at 30fps ofa 1080p video stream, which means a cumulative latency of both the encode and decode sideis below substantially 50ms, for example substantially 30ms or less. Achieving this level ofruntime performance with only CPU compute on user devices presents challenges which arenot addressed by known methods and systems or in the wider AI-compression literature.For example, execution of different parts of the compression pipeline during inferencemay be optimized by adjusting the order in which operations are performed using one ormore known CPU scheduling methods. Efficient scheduling can allow for operations to beperformed in parallel, thereby reducing the total execution time. It is also envisaged thatefficient management of memory resources may be implemented, including optimising cachingmethods such as storing frequently-accessed data in faster memory locations, and memoryreuse, which minimizes memory allocation and deallocation operations.A number of concepts related to the AI compression processes and / or their implementationin a hardware system discussed above will now be described. Although each concept isdescribed separately, one or more of the concepts described below may be applied in an AIbased compression process as described above.Concept 1: Flow-Residual Soft MaskIn AI based flow-residual compression, there exist a number of compression artefact typesthat occur that do not exist in traditional compression pipelines. One such artefact appears inimage sequences where there is little to no movement between frames and manifests itself inthe reconstructed images as halo-like rings around edges and objects in the images. Theseartefacts typically arise where small, barely noticeable changes in pixel values between framessuch as tiny lighting changes, block artefacts from traditional compression (if the input tothe AI compression pipeline came from a traditionally compressed source), noise, and soon become amplified over time by the AI based compression pipeline. This can, in somearchitectures, result in parts the networks of the compression pipeline treating image noise asreal movement and then exaggerating these as the sequence of frames of the video progress.For example, in some architectures, these artefacts may be caused by the operation of the flowmodule(s), in other architectures this may be caused by the operation of the residual module(s),in again other architectures this may be caused by both modules or indeed other parts of thearchitectures, and so on. At the end of the sequence of frames, the artefacts can be quitesevere and detrimental to the image reconstruction quality. This type of artefact is referred tohereinafter as "unrolling".Unrolling artefacts occur across mostly all types of image sequences but are particularlyapparent when AI based flow-residual compression models are used to encode and decodeimage sequences of internet video calls and AR / VR scenes where a foreground user who movesvery little is surrounded by a largely static background which also does not move much), screensharing image sequences, security camera feeds and others. If there is no movement after aperiod of time, objects and edges start to exhibit halo-like ringing around their edges. Theartefacts also result in the compression pipeline assigning a disproportionate number of bits tohandle the hallucinated movement of the unrolling artefacts even though the correspondingparts of the actual images aren’t changing or are hardly changing between frames so ought notto have many bits assigned to them.Concept 1 is directed to solving this unrolling type artefact in static or low-movement imagesequences.Figures 5a and 5b respectively show comparative examples 500a, 500b of an output image ofan AI based compression pipeline with a flow-residual component. In the first example 500a,it can be seen that the image exhibits an unrolling type artefact 501 on a static backgroundobject. That is, the edges of the object exhibit halo-like ringing. In contrast, in the secondexample 500b which is generated using the solutions described below, the unrolling artefactsare mitigated.More particularly, it has been found that outputs from intermediate layers of a flow decoderneural network (e.g. outputs of convolution and / or activation layers) contain high-level featureinformation indicative of motion or non-motion between frames. This extracted informationcontains a strong enough learning signal to facilitate the introduction of a learned, pixel-wise,partial frame skip mechanism in an AI-based compression pipeline in a computationallyefficient way.Extracting these outputs from the intermediate layers of the flow decoder and feeding theseinto the residual encoder network and / or residual decoder network (e.g. as an additional inputto the encoder and / or decoder, and / or any interim layers thereof), and introducing convolutionand / or activation operations to convert this extracted information into an output tensor, allowsthe network of the present disclosure to output a final output image that is based partly onthe the output of the residual decoder and partly on a previously decoded image. The exactamounts of each are determined by the values of the tensor, which are in turn determined bythe operation of the convolution and activation operations on the extracted information.Taking the above approach at a general level, consider the illustrative example shown in Figure6, which shows flow encoder / decoder networks and residual encoder / decoder networks of anAI based compression pipeline. This may be, for example, an AI based compression pipelineof the type shown in Figure 3.The flow encoder neural network takes a current image ^^^^ and a previous image ^^^^−1, encodesthese into a flow latent representation which is optionally quantised and entropy encoded andtransmitted as a bit stream. On the decode side, the bitstream is received, entropy decoded intothe flow latent representation that the flow decoder neural network uses as input to producea representation of flow ^^ , e.g. a flow map. The flow map ^^ is applied to a previouslydecoded image ^^^^−1 to generate a warped version of that previously decoded image ^^^^−1,^^ . Thewarped version of the previously decoded image ^^^^−1,^^ is then fed into the residual encodernetwork, together with the current image ^^^^ to produce a residual latent representation which isoptionally quantised and entropy encoded and transmitted as a bit stream. On the decode side,the bitstream is received and entropy decoded back into the residual latent representation ^^ andused by the residual decoder neural network, in combination with information from the warpedversion of the previously decoded image ^^^^−1,^^ , to produce the reconstructed image ^^^^ . In thecase of Figure 6, the information associated with the warped previously decoded image ^^^^−1,^^is optionally first processed by module ^^ , referred to herein after as a composition adapter, forexample to downsample and / or pad it before it is fed into the residual decoder together withthe entropy decoded residual latent representation to produce the final reconstructed image^^^^ . This process may then be repeated for ^^^^+1 and so on to encode, transmit, and decode asequence of frames.Figure 7 illustrates how the architecture of Figure 6 may be modified according to the presentdisclosure to feed outputs of one or more interim layers of the flow decoder into the residualencoder and or decoder. For completeness, the components that correspond to Figure 6 areshown again. The flow encoder neural network takes a current image ^^^^ and a previous image^^^^−1, encodes these into a flow latent representation which is optionally quantised and entropyencoded and transmitted as a bit stream. On the decode side, the bitstream is received, entropydecoded into the flow latent representation that the flow decoder neural network uses as input toproduce a representation of flow ^^ , e.g. a flow map. The flow map ^^ is applied to a previouslydecoded image ^^^^−1, ^^ ^^^^^^^^ to generate a warped version of that previously decoded image ^^^^−1,^^ .The warped version of the previously decoded image ^^^^−1,^^ is then fed into the residual encodernetwork, together with the current image ^^^^ to produce a residual latent representation ^^ whichis optionally quantised and entropy encoded and transmitted as a bit stream. On the decodeside, the bitstream is received and entropy decoded back into the residual latent representation^^ and used by the residual decoder neural network, in combination with information from thewarped version of the previously decoded image ^^^^−1,^^ , to produce the reconstructed image ^^^^ .The information associated with the warped previously decoded image ^^^^−1,^^ is optionally firstprocessed by the composition adapter ^^ , before it is fed into the residual decoder together withthe entropy decoded residual latent representation to produce the final reconstructed image^^^^ . This process may then be repeated for ^^^^+1 and so on to encode, transmit, and decode asequence of frames.However, unlike in Figure 6, the information (e.g. outputs and / or activations) 700 of one ormore intermediate layers 701 of the flow decoder 209 are extracted and fed into one or both ofthe the residual encoder 211 and / or decoder 212.In the case of the residual encoder 211, this may be, for example, as an input into the residualencoder 211 together with ^^^^−1,^^ and ^^^^ . Additionally or alternatively, this may be as input intoone or more intermediate layers (not shown) of the residual encoder 211.In the case of the residual decoder 212, this may be, for example as an input into the residualdecoder 212 together with ^^ and / or with a representation of ^^^^−1,^^, for example output bycomposition adapter ^^ . Additionally or alternatively, this may be as input into one or moreintermediate layers (not shown) of the residual decoder 212.In both cases, pre-processing may be applied to, for example, upsample or downsample, and / orpad, the outputs 700 to ensure the dimensionality matches that expected by the residual encoder211 and / or decoder 212.Additionally, one or more convolution and / or activation layers of the residual decoder (orindeed of a separate network) are applied to the information extracted from the intermediatelayers of the flow decoder and the residual information which together facilitate the generationof a new tensor ^^ which is referred to herein as a mask.Accordingly, the result of this architecture change in Figure 7 is that the residual decoder 212is now able to output not just a reconstruction of image ^^^^ , but also mask ^^ (e.g. a tensor ofvalues) that, when applied to the pixels of a previously decoded image ^^^^−1, ^^ ^^^^^^^^ and when(inversely) applied to the the pixels of the output of the residual decoder 212, ^^^^ , produce acomposite, final output image made up of a blend of these two. The exact composition ofthis blend is determined by the exact values of the ^^ tensor. At a general level, the mask^^ can be understood as a way to apply a learned, pixel-wise "frame" skip for a given pairof frames. That is, if there is a high degree of motion between frames, then it is likely thatthe final output image blend will contain a greater contribution of pixel information from theoutput of the residual decoder. In contrast, if there is a low degree of motion, it is likely thatthe final output image blend will contain a greater contribution of pixel information from thepreviously decoded frame (indicating that no movement has happened).The exact values of a given output mask tensor ^^ produced by the residual decoder for giveninput image and previously decoded image are dependent the weights and / or biases of theresidual decoder which are learned.This mask ^^ is referred to hereinafter as a "soft mask". The term "soft" is used to denote thatthe mask is not simply a tensor of binary 1s and 0s that would indicate which values to use ornot use in a subsequent step by fully masking those values, rather the mask is "soft" in that itsvalues may be values anywhere between a minimum or maximum value (e.g. between andincluding 0 and 1 but more than only 0 and 1) with a set of values determined by the fixed orfloating point precision regime the residual decoder is operating in (e.g. fp16, int8 and so on).This means that the "soft" or "non-binary" mask only partially masks the values the mask isapplied to, allowing it to emphasise or de-emphasise those values based on whatever trainingobjective the mask was trained for.In the present context, the soft mask emphasises or de-emphasises, to varying degrees,movement between two images in the final output image by partially masking, to differentextents, how much of the final output image is based on the previously decoded image ^^^^−1, ^^ ^^^^^^^^(low motion) whereby the majority of pixels can be "copied" from the previously decodedimage and are accordingly "skipped", and how much is based on the current frame ^^^^ (highmotion) whereby the majority of the pixels are generated by the residual decoder and capturemore of the current frame in which the movement has taken place.This learned, pixel-wise frame skip approach at a general level lets the AI-based compressionnetwork learn how to correctly handle low motion frame sequences in a way that does notresult in unrolling artefacts.For completeness, it is also noted that in extreme cases, the soft mask ^^ may end up lookinglike a binary mask of all 1s (indicating no movement and an identity operation on the previouslydecoded image i.e. ^^^^, ^^ ^^^^^^^^ = ^^^^−1), or all 0s (indicating complete movement whereby the finaloutput image ends up corresponding to the output of the residual decoder i.e. ^^^^, ^^ ^^^^^^^^ = ^^^^) andcorresponds to the output of an architecture such as that in Figure 6.Considering now further details, the final reconstructed image can be produced 702 by addingthe masked, previously decoded image ^^^^^^−1, ^^ ^^^^^^^^ and the inversely masked residual decoderoutput (1 − ^^)^^^^ :^^^^, ^^ ^^^^^^^^ = ^^^^^^−1, ^^ ^^^^^^^^ + (1 − ^^)^^^^ As described above, this new final reconstructed image ^^^^, ^^ ^^^^^^^^ can be understood at a generallevel as being a weighted blend of ^^^^ and ^^^^−1, where the weightings are based on the soft mask^^ and are indicative of how much the flow residual networks are emphasising high motionbetween two images versus low motion. If, for a given input pair of images, the networks havelearned that a high degree of motion contributes to a high reconstruction quality, then thesoft mask ^^ tensor values will end up with values that prioritise the contribution to the finaloutput image ^^^^, ^^ ^^^^^^^^ of the previously decoded image ^^^^−1 . Conversely, for image pairs wherethere is low motion, the soft mask ^^ tensor values will end up with values that prioritise thecontribution to the final output image ^^^^, ^^ ^^^^^^^^ of ^^^^ .At an ever high level of generality, this approach can be understood as the networks learningwhich pixels of an image to "skip" and by how much.A practical consequence of adopting the architecture shown in Figure 7 is the almost completemitigation of unrolling artefacts (i.e. the mitigation of undesirable amplification of noise andtiny pixel variations into hallucinated intra-frame movement).Further, this synergistically improves compression rate and reconstruction quality of thecompression pipeline because the absence of the hallucinated unrolling artefacts allows thenetworks to better assign only minimal bits to the transmission of pixels in images wherethere is little to no movement, and instead the networks now correctly assign more bits topixels where high movement is actually occurring between frames, or areas where there isa lot of spatial detail in an image that need to be retained. As described above, absent asolution to unrolling artefacts, AI based compression networks have a tendency to assign adisproportionate number of bits to the hallucinated intra-frame movement.Further, synergistically, as this approach uses the outputs of intermediate layers of the flowdecoder which are lower-dimensional than the final output flow map or warped image, the overallcompute resource overhead and run time slow down resulting from taking this intermediateflow decoder data and feeding it into the residual encoder / decoder is less than it otherwisewould be had the soft mask been computed from the higher-dimensional final output flowmap or warped image. In other words, it has been found that the lower-resolution, high levelabstraction of flow information found in the outputs of intermediate layers of the flow decodercontains movement and non-movement pixel information that improves the residual decoder’sability to distinguish between movement and non-movement pixel information whereby this isfacilitated in a way that is compute efficient and does not significantly slow run time of thecompression pipeline.Whilst not shown in Figure 7, it is envisaged that to facilitate the residual decoder’s ability tooutput soft mask ^^, the residual decoder 212 may comprise one or more convolution layersand / or activation layers that receive the outputs 700 as input together with ^^^^ but withoutrelying on any other flow information such as ^^^^−1,^^. That is, in some implementations, theresidual decoder 212 may be said to have two parallel threads, one used to calculate the softmask ^^ that takes as input the outputs 700 of the intermediate layers of the flow decoder (alsoreferred to herein as flow side information), and the residual information ^^, and one threadused to calculate ^^^^ that takes as input the warped previously decoded image ^^^^−1,^^ and ^^.The outputs of these two parallel threads are then combined as described above to producethe final reconstructed image i.e. ^^^^, ^^ ^^^^^^^^ = ^^^^^^−1, ^^ ^^^^^^^^ + (1 − ^^)^^^^ . This approach allows thecalculation of the soft mask may take place separately (e.g. in a separate module and separateneural network) and in parallel from the residual decoder decoding the residual informationinto an intermediate output image, allowing the softmask calculation to take advantage ofparallelisation abilities of the hardware it is running on to facilitate speed up.Alternatively, in other implementations the soft mask estimation may be intertwined with theoperation of the residual decoder (e.g. one or more outputs of one or more convolution oractivation layers of the residual decoder may be fed into one or more convolution or activationlayers used to calculate the soft mask or vice versa where the residual decoder may use one ormore intermediate outputs of the the layers estimating the soft mask) in which case the softmask neural network and the residual decoder may be seen to be integrated as a single neuralnetwork with mutual inputs, as well as mutual outputs of ^^^^ and ^^.In a specific, non-limiting implementation example, the residual decoder 212 may accordinglycomprise one or more convolution layers and one or more activation layers that processthe input flow side information 700 and residual information ^^ to produce the soft mask ^^.For example, the residual decoder 212 may comprise a first convolution layer and a secondconvolution each with a predetermined number of input channels, kernel sizes, and strides, anactivation layer that combines these, for example with a float functional add-ReLU operation, afurther convolution layer may then be applied to the output of the add-ReLU operation, whoseoutput is then fed into a further activation function, for example a ReLU layer, before a finalconvolution layer is applied to the output of the ReLU layer. A pixel shuffle operation may thenoptionally be performed on the output of the final convolution layer before a final hardtanhactivation layer is applied to produce the final soft mask output. It will be appreciated thatthe input channel dimensions, kernel sizes and strides will be determined by the input imagetypes and dimensions that form part of the frame sequence being encoded and decoded. Itwill also be appreciated that this is an illustrative implementation only and other exampleimplementations are also envisaged. It will also be appreciated that the weights and biases ofthese layers are learnable and it is envisaged that these may be trained in an end to end mannerwith the rest of the compression pipeline described here.It is also envisaged that the use of the intermediate information from the flow decoder isonly one type of information that contains a strong enough signal to facilitate the learningof a network that outputs a soft mask. Accordingly, whilst the above approach has beendescribed in the context of using flow side information, it is envisaged that other informationmay be additionally or alternatively be used, for example, the latent representation of the flowbefore it is input into the flow decoder may be used to train a network to produce a soft mask.Additionally or alternatively, separately obtained information indicative of differences betweenthe first and second image as identified by a method comprising using bounding box or regionof interest comparisons may be used(i.e. comparing patches or partial patches directly).Finally, it is envisaged that the final output image need not be obtained only from combining amasked previously decoded image and the masked output of the residual decoder. Rather, it isenvisaged that the residual decoder output may be modified by the mask to produce a modifiedoutput image, and then used in some further, other way, to obtain a final output image.Concept 2: Encoder MasksA further problem that exists in AI-based video and image compression is that it is challengingfor the neural networks on the encoder side of AI-based compression pipeline to learn how tobest assign bits to the information being transmitted in a bit stream when it has no knowledgeof how that information will be processed on the decode side. One non-limiting example ofthis challenge can be illustrated by considering the effect of using the soft-mask described inconcept 1 above. That is, if the encoder is assigning a lot of bits to certain pixels of an inputimage that going to be soft-masked out on the decode side anyway, it is a waste of bits that mightbe better spent elsewhere or that might not be needed at all. This challenge accordingly presentsan opportunity to further reduce bitstream size and thus improve compression performance (ine.g. bits per pixel or in any other metric indicative of compression performance).The present inventors have realised that introducing a learned mask on the encode side thatmodifies some or all the data before it is entropy encoded into a bitstream can result insignificantly reduced bitstream sizes without a corresponding reduction in reconstructionaccuracy.Consider the example of a pipeline where a soft-mask such as that described above (or indeedany mask or mechanism where information is discarded on the decode side) has resulted ina reconstructed image ^^^^ where all or most of the information that resulted in the the pixelsof that reconstructed image came from a previously decoded image ^^^^−1 rather than from theoutput of the residual decoder ^^^^ (e.g. because there was no or low motion between ^^^^−1 and ^^^^and the networks have learned that using the soft mask to use more of the previously decodedimage instead of the reconstructed current image results in a better rate and / or distortion). Inthis case, most of the bits that were spent to transmit the information associated with ^^^^ in thebitstream are wasted as they are not used or only partially used on the decode side.The inventors found by examining the reconstruction of the current frame ^^^^ before applicationof the soft-mask that the areas being soft-masked out (i.e. areas where the previously decodedframe ^^^^−1 contribute more to the output frame) were already being made blurrier and lessaccurate by the networks of the pipeline. That is, the introduction of a soft-mask architecturewas unexpectedly causing the networks to begin to implicitly learn that bitstream size could bereduced without affecting reconstruction accuracy by assigning fewer bits to areas that werenot being used after the soft-mask architectural component was applied. In very general terms,the networks were learning to become soft-mask aware implicitly. However, this effect wasonly very subtle and slight. Concept 2 is directed to harnessing this effect by introducing anexplicit architectural mechanism that the networks can use directly to mask out informationbefore producing and sending the bitstream. This works not only with a soft-mask on thedecode side such as that described above, but also with any mechanism where the decode sideis discarding informationn.That is, if the encode side had an architectural mechanism to know when the decode sidewould likely not use some or all of the transmitted information, and to know which informationis likely to not be used, then this information can be omitted from the bitstream. The resultingbitstream is thus smaller and the effect on the decode side in terms of reconstruction accuracywould be minimal because the omitted information was not being used anyway.Presently described concept 2 introduces such a mechanism in the form of an encoder-sidelearned mask that fully or partially masks some or all of the information to be transmittedbefore it is transmitted. In very general terms, it provides an architectural mechanism forthe encoder-side networks to explicitly reduce the entropy of the distribution of the latentrepresentation, hyper latent representation and / or hyper hyper latent representation, allowingthese to be entropy encoded into a smaller bitstream size.For example, the encoder-side learned mask may comprise a tensor applied to a one or all of thelatent representation, the hyper latent representation or the hyper hyper latent representationproduced by the encoder neural networks of the I-frame part 201 and / or of the P-frame part 202(e.g. as shown in Figures 3, 6 and / or 7) before these are entropy encoded into a bitstream. Thevalues of the tensor may be produced by a dedicated encoder mask neural network separateto the neural networks of the respective the I-frame part 201 and / or of the P-frame part 202,or they may be output directly by the existing neural networks of the respective the I-framepart 201 and / or of the P-frame part 202 in the same way that the residual decoder described inconcept 1 outputs not only ^^^^ but also the soft-mask ^^. In both cases, it is envisaged that theneural network(s) that produce the encoder-side learned mask are trained together with theother neural networks of the AI-based compression pipeline to allow the networks to learnhow best to use the architectural mechanism to produce an encoder-side learned mask that bestminimises the loss of whatever loss function is used during training (e.g. a rate distortion lossfunction).Pseudocode illustrating an exemplary training loop is set out below:Algorithm 1 Training with encoder-side maskInputs: Neural network ^^ (^^) including ^^(^^) for outputting encoder mask ^^Config specifying which encoder layer outputs to apply ^^ toVariables: Optimizer ^^, gradient computation flag ^^^^^^^^.Initialise: ^^^^^^^^ = ^^^^^^^^ for ^^ (^^) and ^^(^^) and Training loop counter ^^ = 0.while ^^ < ^^ doForward pass through ^^ (^^) and ^^(^^).Compute rate distortion loss with ^^ applied before entropy encoding step and perform backwardpass with ^^^^^^^^.Apply optimizer ^^ to update ^^ (^^) and ^^(^^)^^ ← ^^ + 1end while

[0002] In general terms, a neural network ^^ (^^) including either one or more layers (or a standaloneneural network defined as ^^(^^) for outputting a mask ^^) and an accompanying configurationspecifying where the encoder mask ^^ is to be applied are initialised. The configuration may,for example specify that the mask ^^ is to be applied to latent representation tensor, hyperlatent representation tensor and / or hyper hyper latent representation tensor output by one ormore of the encoder, hyper encoder, and / or hyper hyper encoders of the I-frame part 201,and / or P-frame part 202 of the pipeline at some point before these are entropy encoded intoa bitstream, for example after a normalising step, as will be described in more detail below.The neural networks ^^ and ^^ are initialised e.g. with random values, and the grad flag isset to true for both the weights and / or biases of the neural network ^^ and ^^ as well as thenoise, indicating that they are trainable. Training for for ^^ steps then commences by repeatedlyperforming a forward pass through ^^ and ^^ with the mask ^^ added to the outputs specified inthe configuration. The loss is computed from, for example, rate distortion loss function (e.g.as described above in the introductory section), and back propagation is performed throughthe elements where ^^^^^^^^ is set to true. An optimiser ^^, e.g. SGD, Adam, or some otheroptimisation algorithm is applied to update the weights and / or biases of ^^ and ^^. Trainingends after some predetermined condition is met, for example after a predetermined number ofepochs, training steps, or after the loss plateaus or in some other way converges. It is envisagedthat this training loop may be performed in a high precision space such as fp32. After training,the implementation of ^^ and / or ^^ may be converted to perform inference in a lower precisionspace such as int8 whereby the weights and biases may be converted into int8 values, and theoperations (such as matmuls and so on) used during inference may be specified to operatefully or in part in a lower precision space as applicable. During inference using the trainedimplementations of ^^ and ^^, it is envisaged that the mask ^^ is applied to the same encoderoutputs as in training.An exemplary implementation of the present concept is illustrated with reference to Figures 8.Figure 8 shows flow encoder / decoder networks and residual encoder / decoder networks of anAI based compression pipeline including an I-frame part 201 and a P-frame part 202. This maybe, for example, similar to the AI based compression pipeline of the type shown in Figure 3.The flow encoder neural network takes a current image ^^^^ and a previous image ^^^^−1, encodesthese into a flow latent representation which is optionally quantised and entropy encoded andtransmitted as a bit stream. On the decode side, the bitstream is received, entropy decoded intothe flow latent representation that the flow decoder neural network uses as input to producea representation of flow ^^ , e.g. a flow map. The flow map ^^ is applied to a previouslydecoded image ^^^^−1 to generate a warped version of that previously decoded image ^^^^−1,^^ . Thewarped version of the previously decoded image ^^^^−1,^^ is then fed into the residual encodernetwork, together with the current image ^^^^ to produce a residual latent representation which isoptionally quantised and entropy encoded and transmitted as a bit stream. On the decode side,the bitstream is received and entropy decoded back into the residual latent representation ^^ andused by the residual decoder neural network, in combination with information from the warpedversion of the previously decoded image ^^^^−1,^^ , to produce the reconstructed image ^^^^ . In thecase of Figure 8, the information associated with the warped previously decoded image ^^^^−1,^^is optionally first processed by module ^^ , referred to herein after as a composition adapter, forexample to downsample and / or pad it before it is fed into the residual decoder together withthe entropy decoded residual latent representation to produce the final reconstructed image^^^^ . This process may then be repeated for ^^^^+1 and so on to encode, transmit, and decode asequence of frames.As is further shown in Figure 8, the entropy encoding that produces the respective bitstreams204, 208, 212 is supported by a corresponding hyper network which in turn is supported bya hyper hyper network. The hyper network for the I-frame module 201 comprises a neuralnetwork 801 that is referred to as a hyper encoder, and a neural network 802 that is referred toas a hyper decoder. As is described in connection with Figure 2, the hyper encoder neuralnetwork 801 encodes the latent representation produced by the encoder neural network 203,and the hyper decoder neural network 802 produces the entropy parameters that define thedistribution of the latent representation. These parameters are used during entropy encoding toproduce the bitstream 204. The encoded latent representation produced by the hyper encoder801 is itself entropy encoded into a bitstream using an entropy model using entropy parametersobtained by encoding the encoded latent representation using the hyper hyper encoder 803,and decoding that encoded latent representation using the hyper hyper decoder 804. Thisnested structure can be extended as desired but it has been found that a hyper and hyper hypernetwork achieve good performance for run time trade off. The entropy parameters that areused to produce the bitstream associated with the hyper hyper network (or whatever the highestlevel of the nested structure is) may optionally be fixed, set parametrically, or produced by anencoder / decoder neural network pair like in the hyper network.As can be seen in Figure 8, this nested hyper network and hyper hyper network approach isalso provided in the optical flow module 206 in the form of hyper encoder neural network 805,hyper decoder neural network 806, hyper hyper encoder neural network 807 and hyper hyperdecoder neural network 808. The same is also provided in the residual module 210 in the formof hyper encoder neural network 809, hyper decoder neural network 810, hyper hyper encoderneural network 811 and hyper hyper decoder neural network 812.Figures 9 and 10 illustrate in more detail an exemplary implementation of the hyper networkand hyper hyper network architectures of, for example, Figures 3, 6, 7 and 8. Figure 9 illustrateshyper encoder 901 and hyper hyper encoder 902 architectures supporting a latent representationencoder 903 of e.g. an I-frame part 201 or P-frame part 202, while Figure 10 illustrates hyperdecoder 901 and hyper hyper decoder 902 architectures supporting a latent representationdecoder 903 of e.g. an I-frame part 201 or a P-frame part 202.Starting with Figure 9, an input image ^^^^ (or in the case of a flow and / or residual module ofa P-frame module, input image(s) ^^^^ , ..., ^^^^−^^, and / or various representations thereof, fromdifferent times of an image sequence) are fed into a first neural network 904 that outputs a latentrepresentation of the input. The latent representation produced by the first neural network 904(the encoder neural network) is then fed into a second neural network 905 (the hyper encoderneural network) that outputs a hyper latent representation (that is a latent representation ofthe latent representation). The hyper latent representation is then decoded by a third neuralnetwork 906 (the hyper decoder neural network), producing entropy parameters 907 indicativeof a distribution of the latent representation. These entropy parameters 907 represent a tensorof location and scale parameters ^^, ^^ and ^^ that define, for a given set of pixel values, whattheir zero-point or mean is, their scale is, and for, example, the variance of the distribution.The entropy parameters 907 allow the distribution (in this case of the latent representation)to be normalised 908 by centering the distribution around a zero point and scaling it aroundthat zero point to produce a tensor of values that have minimal entropy that can accordinglybe very efficiently, losslessly encoded by applying a subsequent entropy encoding operation909 on the normalised distribution, for example using rANS or any other entropy encodingoperation on the normalised distribution to produce a first bitstream part 910 that comprisesthe latent representation information.Whilst not shown, the hyper decoder neural network 906 may also produce one or morequantisation parameters (e.g. bin sizes and bin widths) that allow the latent representation tobe quantised before performing the normalising 908 and / or entropy encoding 909 in a way thatbetter minimises information loss than would be possible without using learned quantisationparameters. That is, using learned quantisation parameters produced in the hyper networkfacilitate better quantisation than using naive quantisation parameters.Returning to Figure 9, the hyper latent representation output by the hyper encoder neuralnetwork 905 is not only fed into the hyper decoder neural network 906 to produce the entropyparameters 907 for entropy encoding the latent representation, but is itself fed into a hyperhyper encoder neural network 911 of the hyper hyper network 902 to produce a hyper hyperlatent representation (that is a latent representation of the latent representation of the latentrepresentation of the input). The hyper hyper latent representation is then fed into the hyperhyper decoder neural network 912 producing a set of entropy parameters 913 that are indicativeof a distribution of the hyper latent representation. That is, the entropy parameters 913represent a tensor of location and scale parameters ^^, ^^ and ^^ that define, for a given set ofpixel values, what their zero-point or mean is, their scale is, and for, example, the varianceof the distribution. In the same way as for the hyper network 901, the entropy parameters913 allow the hyper latent representation to be normalised 914 by centering the distributionaround a zero point and scaling it around that zero point to produce a tensor of values thathave minimal entropy that can accordingly be very efficiently, losslessly encoded by applying asubsequent entropy encoding operation 915 on the normalised distribution, for example usingrANS or any other entropy encoding operation on the normalised distribution to produce asecond bitstream part 916 that comprises the hyper latent representation information that wasused for entropy encoding the latent representation into the first bitstream part 910.As in the hyper network 901, the hyper hyper decoder neural network 912 may also produceoutput quantisation parameters (e.g. bin sizes and bin widths) that all the hyper latentrepresentation to be quantised before performing the normalising operation 914 and / or theentropy encoding operation 915 to facilitate quantisation with reduced information losscompared to using naive quantisation parameters.Finally, the hyper hyper latent representation produced by the hyper hyper encoder neuralnetwork 911 may be normalised 917 using naive (e.g. fixed, not learned) entropy parameters918, or if a hyper hyper hyper network (or some other network for producing entropy parameters)is present, using entropy parameters produced therewith. An entropy encoding operation 919is then performed to produce a third bitstream part 920 that comprises the hyper hyper latentinformation that was used for entropy encoding the hyper latent representation into the secondbitstream part 916.The first, second and third bitstream parts 910. 916, 920 are then ready to be transmitted to thedecode side of the AI-based compression pipeline, an implementation of which is illustrated inFigure 10.Returning now to the concept of a learned encoder mask ^^, Figure 9 further introduces a maskgenerator 921 ^^(^^) for producing a mask ^^. In Figure 9 the mask generator 921 is shown as astandalone neural network but it is also envisaged that it may be integral with one or more ofthe base encoder 904, the hyper encoder 905, the hyper decoder 906, the hyper hyper encoder911, and / or the hyper hyper decoder 912 neural networks. The output of the mask generator921 is a mask ^^ that may be applied to one or more of the normalised distributions resultingfrom the normalisation operations 908, 914, 917. In Figure 9, the mask ^^ is shown to be madeup of different components ^^^^ 922, ^^ ^^ 923, and ^^^^ 924 respectively applied to the normalisedlatent representation, the normalised hyper latent representation and / or the normalised hyperhyper latent representation. Note however that it is envisaged that the mask may alternativelybe applied only to one or some of these and applying a mask to all, as is illustrated in Figure 9,is optional.Thus we can define the operation of the mask generator 921 ^^(^^) with weights ^^ acting on aninput, such as ^^^^ when used in the I-frame part 201 or such as ^^^^ and ^^^^−1 when used in theP-frame part 202 as:^^(^^^^−1, ^^^^ | ^^) = ^^If we define the latent representation as ^^, the hyper latent representation as ^^ and the hyperhyper latent representation as ^^, and the respective normalised versions of these as ^^^^, ^^^^ and^^^^ , then the entropy encoding (EE) operations 909, 915, and 919 may be said to be performedon one or more of ^^^^^^, ^^^^^^ and ^^^^^^, giving:^^^^^^^^^^^^^^^^^^ = ^^^^^^^^^^^^^^^^^^^^ + ^^^^^^^^^^^^^^^^^^^^ + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ = ^^^^ (^^^^^^^^) + ^^^^ (^^ ^^^^^^) + ^^^^ (^^^^^^^^)As described above, it is envisaged that the mask need not be applied to all of ^^, ^^ and ^^, forexample it is also envisaged that that the mask may be applied only to ^^^^, only to ^^^^, only to^^^^, only to ^^^^ and ^^^^, only to ^^^^ and ^^^^, only to ^^^^ and ^^^^, and / or any combination thereof.The mask ^^ may comprise a matrix or tensor of values that modify the latent representation,hyper latent representation and / or hyper hyper latent representation matrix or tensor it isapplied to by for example multiplication or some other operation.The specific form of ^^(^^) is not important, for example it may comprise a simple MLP, aconvolutional neural network, or any other network with trainable parameters. In one example,^^(^^) comprises an alternating sequence of a convolutional layer followed by an activationall or some with learnable weights, and optionally one or more optional reshaping layersto ensure the output mask shape matches that of the latent representation, the hyper latentrepresentation and / or the hyper hyper latent representation it is to be applied to.The inventors have found that introducing this architectural mechanism allows the AI-compression pipeline to learn during better during training which information (e.g. whichpixels in the latent representation, the hyper latent representation and / or the hyper hyper latentrepresentation) can be masked fully or partially away on the encode side without significantlyincreasing distortion scores. That is, it provides an architectural mechanism for the networkto explicitly learn the behaviour which is implicitly learned when, for example, a soft-maskarchitectural mechansim is introduced on the decode side.Accordingly, whilst it is envisaged that concept 2 may be used as a standalone architecturalcomponent, the inventors have found that there is a synergistic effect between the maskproduced by the mask generator ^^(^^) and the soft-described above in concept 1. That is, whena pipeline having both a soft-mask architectural component and an encoder mask architecturalcomponent is trained, the networks of the encoder side learn to use the mask generator ^^(^^)to mask out those parts of the latent representation, hyper latent representation and / or hyperhyper latent representation which would otherwise have been soft masked out on the decodeside. Effectively, the networks learn to use ^^(^^) to closely track the effect of the soft-mask. Asa result, combining both components results in an architecture that is better able to minimisethe sending of bits that would otherwise be wasted when masked out on decode side than whenusing either component alone.Returning now to the decode side, Figure 10 illustrates exemplary decoder, hyper decoder andhyper hyper decoder architectural components that may be coupled with the architecture ofFigure 9. As is shown in Figure 10, the bitstream, made up of first, second and third parts 910,916, 920 is received on the decode side from the encode side of the compression pipeline. Thehyper hyper network part 1001 starts running first, then the hyper network 1002, then the basenetwork 1003. That is, the third bitstream part 920 is entropy decoded 1004 and denormalised1005 using naive entropy parameters 1006 (e.g. fixed, and / or not learned), or if a hyper hyperhyper network or some other network for producing entropy parameters is present, using theentropy parameters produced therewith. These correspond to those that were used on theencode side to normalise and entropy encode the hyper hyper latent representation, as shownin Figure 9. That is, a variance parameter ^^ is used to entropy decode (e.g. rANS decodeor perform some other entropy decode operation) to reconstruct the normalised hyper hyperlatent representation ^^^^ which is then denormalised (i.e. the centering around a zero-point ^^and scaling with ^^ is reversed) to reconstruct the hyper hyper latent representation ^^.The hyper hyper latent representation ^^ is then fed into the hyper hyper decoder 1007, whichmay correspond substantially in terms of weights and / or biases to the hyper hyper decoder912 present on the encode side as shown in Figure 9. The output of the hyper hyper decoder1007 on the decode side accordingly comprises a reconstruction of the entropy parameters1008 that were used to normalise and entropy encode the hyper latent representation on theencode side and can accordingly be used to perform the reverse operations on the secondbitstream part 916 on the decode side. That is, the variance parameter ^^ produced by thehyper hyper decoder neural network 1007 is used to perform a rANS decode (or other entropydecode) operation 1009 on the second bitstream part 916 to produced the normalised hyperlatent representation, and the reconstructed zero point and scale parameters ^^ and ^^ producedby the hyper hyper decoder 1007 are used to denormalise 1010 the normalised hyper latentrepresentation to reconstruct the hyper latent representation ^^ on the decode side.The reconstructed hyper latent representation ^^ is then fed into the hyper decoder neuralnetwork 1011, which may correspond substantially in terms of weights and / or biases to thehyper decoder 906 present on the encode side as shown in Figure 9, to produce a reconstructionof the entropy parameters 1012 that were used on the encode side to normalise and entropyencode the latent representation ^^. These can accordingly be used to perform the reverseoperations on the first bitstream part 910 on the decode side. That is, the variance parameter ^^produced by the hyper decoder neural network 1011 is used to perform a rANS decode (orother entropy decode) operation 1013 on the first bitstream part 910 to produced the normalisedlatent representation, and the reconstructed zero point and scale parameters ^^ and ^^ producedby the hyper decoder 1011 are used to denormalise 1014 the normalised latent representationto reconstruct the latent representation ^^ on the decode side.The latent representation ^^ may then be fed into the base decoder neural network to reconstructthe input as ^^^^ or some representation thereof, or information associated therewith. Forexample, rather than an image, the output may be a reconstruction of information representingflow between two or more input images if used in a flow module of a flow-residual architecture.That is, the architectures of Figures 9 and 10 may be used in any of the parts of an AI-basedcompression pipeline where a base network, a hyper network and / or a hyper hyper networkis used. For example, Figure 10 also shows that a previously decoded frame ^^^^−1 may beintroduced at the base decoder neural network 1015 level, as shown in, for example, Figures 3,6, 7 and 8, either as part of the soft-mask architecture described in concept 1 above, or using asimple residual mechanism as shown in Figure 3, or using a composition adaptor as shown inFigures 6 and 8, and / or through any other mechanism where the final reconstructed output insome way uses a previously decoded image or previous image of an image sequence.More generally, the inventors have found that introducing an encoder mask in the residualmodule of a flow-residual component of an AI-based compression pipeline results a particularsignificant reduction in bit rate arising from the compression of high motion or low motionscenes. It is understood that this is a result of such scenes causing spikes in bits needed tocompress the frames of such scenes. Accordingly, the mask allows the compression pipelineto choose simply to mask the parts of the scenes that cause such spikes, thereby reducing thecompression rate spikes while surprisingly not resulting in significant increases in distortion(i.e. not resulting in a drop in image reconstruction accuracy). Further, in many cases theneural networks of the pipeline learn to use the masking mechanism to mask out certain typesof image content that is typically expensive (in terms of bits) to encode, rather than just contentwhich might be masked out be e.g. a soft-mask architectural component. For example, thenetworks have been observed to mask out parts of latent representations that cause very low"scale" entropy parameters to be produced. In layman’s terms, the networks in this case havelearned that it is better to just let the distortion increase slightly for such images to avoid amuch more significant worsening in compression rate.It will also be appreciated that the effect that the encoder side mask has in terms of saved bitswhen used on the latent representation ^^, the hyper latent representation ^^ and / or the hyperhyper latent representation ^^ may be different. For example, the inventors have observedthat, when the encoder side mask is used together with the soft-mask in the residual moduleof the compression pipeline, the bit savings are proportionally greater for ^^ than for ^^. Anexplanation for these kind of differences is set out as follows. Consider a section of thelatent representation ^^ corresponding to an area that the soft-mask is fully masking out onthe decode side (i.e. where the previously decoded image information is entirely used inthe reconstructed of that section, rather than the received transmitted information). Any bitsspent on this section are entirely wasted. The neural networks of the pipeline accordinglyimprove rate distortion performance by trying to minimise those bits (i.e. not waste bits onsending information associated with those sections). The hyper encoder and hyper decodertogether work to produce the entropy model parameters that define the distribution of the latentrepresentation ^^ that is ending up in the bitstream. For example, if the latent representation ^^is modelled with a discrete Laplace distribution, then the hyper decoder outputs the parametersthat parameterise that distribution. An efficiently entropy encodable Laplace distribution hasthe values of the latent distribution ^^ in the interval (^^ − ^^ / 2, ^^ + ^^ / 2), the bit cost of thisinterval being proportional to the scale parameter ^^. A problem however, is that the residual(base) encoder has no information about what ^^ and ^^ are going to be, because those requiresuper-local information that only arises in the hyper latent representation ^^ after the hyperencoder acts on the latent representation ^^. Accordingly, the onus of minimising wasted bits in^^ actually falls on ^^. It can be identified empirically from observing decomopsition plots of^^, ^^ and ^^ that the latent representation ^^ is the most expensive of ^^, ^^ and ^^ to transmit inthe residual module. From this it can be inferred that the networks, even without the encodermask, are already trying to put as many bits into ^^ (and ^^) as possible to reduce the bits spenton disproportionately larger ^^. The same plots also suggest that the neural networks of theresidual module attempt to do this by getting as close to the ideal Laplace distribution referredto above when it is possible to do so for a given input image.Thus, when the encoder mask is introduced, it forces the base encoder to produce a latentrepresentation ^^ with the ideal location parameter ^^ and scale ^^, regardless of ^^, meaning thereis no extra bit cost in ^^ to get either ^^ or ^^ to be "correct". There is still a bit cost to pay for thescale parameter ^^ to make the Laplace distribution as cheap in terms of bits as it can be. Thus,the bit savings of the encoder mask in this situation are primarily in ^^ rather than in ^^.That being said, there are nevertheless still some bits saved in ^^. In that, if the base encoderproduces a distribution with ^^ and ^^ very far from the ideal Laplace distribution (e.g. whichcan occur with very fine details in some frames), then the mask modifies the distribution to topush it to be closer to ther ideal distribution, thereby resulting in fewer bits needed to transmitthat distribution after entropy encoding.Finally, it will be appreciated that the concept of the encoder mask introduced in Figure 9for masking the normalised latent representation, hyper latent representation and / or hyperhyper latent representation may be applied at different points in the information flow. That is,whilst the inventors have found that introducing the masking operation just before the entropyencoding step allows the masking step to act as a final filter for avoiding the transmission ofunnecessary bits, the mask may also or alternatively be introduced before the normalisationsteps 908, 914, 917. In this case, the entropy parameters that are produced by the encode sidedecoders 906, 912 will be different to when the mask is applied after the normalising step.Specifically, they end up entropy modeling an already masked distribution, which can in somecases be advantageous as masked parts of the distribution contribute very little to the overallentropy of the distribution and can result in very simple distributions after normalisation. Inturn these already have a very low entropy and can be losslessly compressed to small bit streamsizes without a further masking step.Concept 3: Mini frames for warping correctionCertain scene types such as scene cuts, occlusion scenes (where an object moves and revealsare highly complex pixel structure background behind it), large portions of the frame changingdue to high motion, failure to describe high motion resulting in mispositioned objects in theframe, and other non-motion-compensatable changes in scene content can be challenging forAI-based compression pipelines to handle well. Non-motion-compensatable changes meanschanges between a reference frame and a current frame where no amount of optical flowbased motion compensation can model a change because the reference frame does not containenough or any information that is useful for predicting the current frame. Such scenes containframe sequences that may result in the networks of an AI-based compression pipeline spendinga disproportionate number of bits on to compress and result in artefacts and other failuremodes arising in the reconstructed frames. It will be appreciated that a failure mode of aneural network arises when the output of the neural network based on a given input deviatessubstantially from what the network would be expected to produce from the input. For example,an AI-based compression pipeline might for many scene types produce reconstructed framesat some target rate score and / or distortion score but for other scene types suffer from a failuremode which results in a large spike in, for example, the distortion score, indicating the outputframe looks nothing like the input frame.In general terms, this underperformance can be attributed in part to how flow residual basedcompression pipelines operate. That is, they produce a reference frame ^^^^−1,^^ that is usedby the residual part 210 to reconstruct ^^^^ . As described above, this reference frame ^^^^−1,^^ isproduced by optical flow based motion compensation through the warping of a past frame ^^^^−^^with the flow map produced by the flow part 206. However, there are several cases in whicha frame produced by motion compensation is not a useful base for the residual part 210 forexample scene cuts, large portions of the frame changing due to high motion, or failure todescribe high motion resulting in mispositioned objects in the frame, and so on. In effect, thisleads to image content at a given location varying drastically between ^^^^−1,^^ and ^^^^ , whichthe residual part 210 struggles to correct without artefacts. In these cases, the inventors haverealised that it is often bitrate-efficient and perceptually-pleasing to apply a coarse, cheapcorrection to these areas when producing the final reference frame for the residual part 210^^^^−1,^^. Concept 3 is directed to this correction.In particular, concept 3 is directed to correcting a warped reference frame ^^^^−1,^^ before it ispassed to the residual part 210 part of the pipeline, thereby reducing the difficulty of the taskthe residual part 210 as to perform, reducing the number of artefacts in the final output image^^^^ , and reducing the number of bits spent by the residual part 210 to produce the final image ^^^^ .In general terms, this correction may be implemented by introducing a new output tensor onthe decode side of the flow part 206, the new output tensor comprises information about thecurrent frame ^^^^ and information about how to use that information to correct ^^^^−1,^^.For example, the flow hyper decoder may be configured not only to output entropy parameters(e.g. location, scale parameters and so on as shown in e.g. Figure 10), but also a learnedrepresentation of ^^^^ referred to hereinafter as ^^^^,^^^^^^^^, for example a tensor having luma andchroma channels and a predetermined that may be the same resolution as ^^^^ ora lower resolution. The base flow decoder may then be configured to output not only theflow map, but also a learned occlusion mask ^^^^^^^^ that specifies where and by how much thewarped reference frame ^^^^−1,^^ is to be corrected by replacing or modifying its pixels usingthe newly produced ^^^^,^^^^^^^^. In very general terms, the flow part 206 of the decode side ofthe pipeline now comprises an additional output that the networks are able to learn to use tocorrect problematic ^^^^−1,^^ that arise from failure modes in teh flow part so that the residualpart 210 is not burdened with that task.More formally, a corrected warped reference frame ^^^^−1,^^,^^^^^^^^ may be defined as:^^^^−1,^^,^^^^^^^^ = ^^^^^^^^ · ^^^^−1,^^ + (1 − ^^^^^^^^) · ^^^^,^^^^^^^^ where ^^^^^^^^ is a learned mask output by the base flow decoder, ^^^^,^^^^^^^^ is a learned representationof the input frame ^^^^ which was fed into the encode side of the flow part 206, and (1− ^^^^^^^^) isthe inverse of the mask ^^^^^^^^. In one implementation, the mask ^^^^^^^^ may be a binary maskand thus specifies which pixels of ^^^^−1,^^,^^^^^^^^ are taken from ^^^^−1,^^ and which are taken from^^^^,^^^^^^^^. Alternatively the mask need not be binary and may instead specify a pixel-wise blendof ^^^^−1,^^ and ^^^^−1,^^. During training, the networks of the pipeline learn to use ^^^^^^^^ and ^^^^,^^^^^^^^ to construct anyportions of ^^^^−1,^^,^^^^^^^^ primarily from ^^^^,^^^^^^^^ when there is very high motion, and primarilyfrom ^^^^−1,^^,^^^^^^^^ when there is low motion. This means the networks are learning to correctany bad flow artefacts in ^^^^−1,^^ by replacing pixels with pixels taken from ^^^^,^^^^^^^^. At the sametime, the networks learn to include in ^^^^,^^^^^^^^ substantially only those pixels which are likely tobe used for said corrections. As a result, ^^^^−1,^^,^^^^^^^^ more accurately captures actual motionbetween ^^^^ and ^^^^−1 and thus the difficulty of the task that the residual part 210 has to performis reduced and the resulting final reconstructed frame ^^^^ contains fewer artefacts and uses upfewer bits in the bitstream.Configuring a networks of the flow part 206 to output ^^^^^^^^ and ^^^^,^^^^^^^^ allows the flow parts 206to effectively operate simultaneously as an I-frame module whereby it learns during trainingthat it may in some cases have to reconstruct relevant information from ^^^^ directly (in the formof ^^^^,^^^^^^^^) and discard some or all of the pixels of ^^^^−1,^^ whenever non-motion-compensatablechanges between ^^^^ and ^^^^−1 occur, for example scene content differs drastically between ^^^^and ^^^^−1 such that it would result in a spike in bitrate and / or distortion scores. In the extremecase, for example when there is a scene change or scene cut, ^^^^^^^^ and ^^^^,^^^^^^^^ even allows theflow part 206 networks to choose to construct ^^^^−1,^^,^^^^^^^^ entirely from the pixels of ^^^^,^^^^^^^^whereby all of the pixels of ^^^^−1,^^ are discarded. In the toy example where the mask ^^^^^^^^ is abinary mask, this means ^^^^^^^^ comprises all zeroes and ^^^^−1,^^,^^^^^^^^ = ^^^^,^^^^^^^^ and operates as aform of learned scene cut detector. When the residual part 210 receives the corrected warpedframe ^^^^−1,^^,^^^^^^^^ , its task is now simply to tweak ^^^^−1,^^,^^^^^^^^ to get as close to ^^^^ as possible,which is much more straightforward to do with fewer bits than trying to get a badly warped,artefact-filled ^^^^−1,^^ as close to ^^^^ as possible.As described above, the flow part 206 of the pipeline may be modified by configuring one ormore of the flow decoder, hyper decoder and / or hyper hyper decoder to produce additionaloutputs in the form of tensors ^^^^,^^^^^^^^ and ^^^^^^^^ and then using those outputs in the manner asdescribed above. In general terms this may comprise introducing one or more output layers tosaid networks, and the tensors output by those layers (i.e. ^^^^,^^^^^^^^ and ^^^^^^^^) may be passed, asapplicable, to the function that performs the correction of ^^^^−1,^^,^^^^^^^^ .Because the use of these new outputs directly affects the bitrate and / or distortion during aforward pass, the weights of the networks of the flow decoder, hyper decoder and / or hyperhyper decoder during training are updated so that the computed values for the ^^^^,^^^^^^^^ and ^^^^^^^^tensors minimise the bitrate and / or distortion of the rate distortion loss function. When thepixel values of ^^^^,^^^^^^^^ and ^^^^^^^^ are monitored during training, it can be seen that the networkshave a tendency to learn to use these outputs in the manner as described above. That is, as amechanism for correcting poorly warped reference frames arising from the flow network’sfailure modes.For example, visualising ^^^^,^^^^^^^^ and ^^^^^^^^ in a plot shows that ^^^^,^^^^^^^^ is substantially emptyat pixel coordinates (e.g. whereby empty is a grey pixel or some other colour that is thesame for all pixels where there is no or low motion allowing for efficient compression forvery few bits), while at the same time contains a representation of ^^^^ wherever there is arenon-motion-compensatable changes between ^^^^ and ^^^^−1. Further, ^^^^^^^^ ends up as a maskthat covers the empty pixel parts of the frame, but lets through non-motion-compensatablechange parts. Visualisations of ^^^^−1 1100, ^^^^ 1101, ^^^^,^^^^^^^^ 1102 and ^^^^^^^^ 1103 from a toyexample scene is shown in Figure 11. In this case, we will use high motion as an exampleof a non-motion-compensatable change. There is high motion between first frame ^^^^−1 andthe second frame ^^^^ in the form of a character who jumps sideways against a low motionbackground having a sun and a tree. When ^^^^ 1101 and ^^^^−1 1100 are fed through the flow part206 of the pipeline and decoding is performed, in addition to their usual outputs, the flow hyperdecoder now directly outputs a representation of ^^^^ in the form of ^^^^,^^^^^^^^ 1102, and the flowdecoder outputs ^^^^^^^^ 1103. It can be seen in Figure 11 that ^^^^,^^^^^^^^ 1103 has a region of emptypixels 1104 as well as a small patch 1105 where ^^^^ is largely reconstructed, corresponding towhere there was the high motion, non-motion-compensatable change. The same applies to^^^^^^^^ in that there is a region 1106 that is masked out corresponding to the empty pixels in^^^^,^^^^^^^^, and a region 1107 that is not masked corresponding to the region 1105 of ^^^^,^^^^^^^^ where^^^^ was largely reconstructed where there was high motion when combined 1203 using:^^^^−1,^^,^^^^^^^^ = ^^^^^^^^ · ^^^^−1,^^ + (1 − ^^^^^^^^) · ^^^^,^^^^^^^^The result is a frame constructed from the low motion pixels of ^^^^−1,^^ and from the highmotion pixels of ^^^^,^^^^^^^^. This frame better captures high motion than the uncorrected versionof ℎ^^^^^^^^−1,^^. Figure 12 illustratively shows the introduction of the above-described architectural mechanisminto the pipeline of Figure 8. Like-numbered references refer to like-numbered elements andare not repeated here. Unlike in Figure 8 however, both the flow base decoder 209 and flowhyper decoder 806 are configured to output respective tensors (i.e. ^^^^^^^^ 1201 and ^^^^,^^^^^^^^ 1202)that are combined 1203 with ^^^^−1,^^ which is produced by applying a warping operation ^^ to apreviously decoded frame ^^^^−1 using the reconstructed flow ^^ also output by the base flowdecoder as described above. The result of these steps is ^^^^−1,^^,^^^^^^^^ which is then passed to theresidual part 210 in the usual way as described above which uses it to finally reconstruct thecurrent frame ^^^^ .Whilst the above example uses the flow hyper decoder and flow base decoder to produce ^^^^,^^^^^^^^1102 and ^^^^^^^^, any of the networks of the flow part 206 may be chosen for this purpose byintroducing respective new outputs from those networks and performing 1203 the operation^^^^−1,^^,^^^^^^^^ = ^^^^^^^^ · ^^^^−1,^^ + (1 − ^^^^^^^^) · ^^^^,^^^^^^^^ with those outputs as described above. Duringtraining, the learn to use the new outputs as a mechanism to correct poor flowestimation as described above by predicting values for these outputs based on the given inputframes that produce this effect, which can be confirmed by visualising ^^^^^^^^ and ^^^^,^^^^^^^^ in themanner as shown in Figure 11. For example, it is envisaged that any of the flow base decoder209, hyper decoder 806 and hyper hyper decoder 808 may output any of ^^^^^^^^ or ^^^^,^^^^^^^^ in anycombination.Given that a purpose of ^^^^,^^^^^^^^ is to be a source of pixels that can be used to correct "bad"pixels in ^^^^−1,^^, where ^^^^^^^^ specifies where the correction is to be made, the inventors haverealised that it is not necessary for the generation of ^^^^,^^^^^^^^ and / or ^^^^^^^^ to be in the samehigher resolution as ^^^^ and ^^^^−1. This is at least because the residual part 210 is effective atcompensating for poor resolution artefacts as long as the pixels are approximately correct,so the generation of ^^^^,^^^^^^^^ and / or ^^^^^^^^ does not have to be perfectly at full resolution. Thus^^^^,^^^^^^^^ and ^^^^^^^^ may be generated efficiently in a downsampled, lower resolution space relativeto ^^^^ and ^^^^−1 (e.g. h / 8, w / 8 or h / 16, w / 16 or some other resolution that is lower than theh and w resolutions of ^^^^ and ^^^^−1), and then upsampled to the resolution of ^^^^−1,^^ beforeperforming the correction. Alternatively, the inventors have found that there are advantagesto generating ^^^^^^^^ at a higher resolution, for example full resolution. This is because ^^^^^^^^ultimately determines where the corrections are made to ^^^^−1,^^ and, because ^^^^^^^^ can producehigh-entropy shapes very cheaply (few bits) due to such shapes sharing boundaries with what’sbeing transmitted in the optical flow information allowing the mutual information betweenthese to be exploited, which in turns means that ^^^^,^^^^^^^^ can spend fewer bits on making accurateshapes and more bits on accurate colours. Thus, generating ^^^^^^^^ even at full resolution isrelatively cheap and facilitates more accurate and better corrections to ^^^^−1,^^.For completeness, whilst not shown in Figure 12, it is envisaged that the soft-maskingmechanism from Figure 7, and the encoder masks from Figure 9 may be included.More details will now be described of an illustrative training implementation with reference tothe pseudocode below.That is, a training data set X, a learning rate ^^, and a number of training steps or epochs ^^ areselected and the network architecture of ^^^^ that comprises a flow part ^^^^ and a residual part ℎ^^Algorithm 2 Training with mini frames for warping correctionInputs: Training dataset X, learning rate ^^, network architecture ^^^^ comprising flow part ^^^^ andresidual part ℎ^^ , number of epochs ^^Initialize network parameters ^^for epoch = 1 to ^^ dofor each batch in X doPerform forward pass to compute ^^^^ = ^^^^ (^^^^−1, ^^^^ ) including: predicting ^^^^^^^^ and ^^^^ ,^^^^^^^^ and flowwith ^^^^ (^^^^−1, ^^^^ ), warping ^^^^−1 with the predicted flow to produce ^^^^−1,^^ , using ^^^^^^^^ and ^^^^ ,^^^^^^^^ to produce^^^^−1,^^,^^^^^^^^ , and finally computing ^^^^ = ℎ^^ (^^^^−1,^^,^^^^^^^^ , ^^^^ ) Compute Loss = ^^ + ^^^^Backward pass to compute gradients ∇(^^,^^,^^)LossUpdate parameters with optimizer O: (^^, ^^, ^^) ← O((^^, ^^, ^^), ∇(^^,^^,^^)Loss, ^^)end for end forOptionally evaluate on validation setis defined, for example as shown in Figure 3. The network parameters ^^, ^^, ^^ are initialised(e.g. randomly) and then the training loop is started. For each batch in the training data X, aforward pass is performed through ^^^^ . As described above, this comprises predicting opticalflow information with the flow part ^^^^ of the architecture, using the optical flow information towarp a previously decoded frame to produce ^^^^−1,^^, as well as using the flow part ^^^^ topredict a mask ^^^^^^^^ and associated representation of the the current frame ^^^^,^^^^^^^^. The mask^^^^^^^^ is then used to combine ^^^^,^^^^^^^^ and ^^^^−1,^^ to produce ^^^^−1,^^,^^^^^^^^ . This combined, warpedpreviously decoded frame is then fed into the residual part ℎ^^ of the architecturetogether with the current frame ^^^^ to compute ^^^^ = ℎ^^ (^^^^−1,^^,^^^^^^^^ , ^^^^). total ^^^^^^^^ is thencalculated by combining a distortion term ^^ and any other loss terms (notshown). The backwards pass is then performed to compute gradients based on the loss, andthe parameters ^^, ^^, ^^ are optimised using the optimiser, such as stochastic gradient descentSGD, or some other known optimiser.Optionally, a validation loss can be calculated. The learning rate, batch size, and / or numberof epochs may be optimised during training, for example using a learning rate scheduler orsome other hyperparameter optimisation method. More generally, the hyperparameters maybe optimised experimentally.One issue that can arise in training is that the networks of ^^^^ comprising flow part ^^^^ andresidual part ℎ^^ have a tendency to "cheat" by finding a local rather than global loss minimumby learning produce occlusion masks ^^^^^^^^ that have values that completely mask away^^^^,^^^^^^^^ for all input frames (i.e. resulting in ^^^^−1,^^,^^^^^^^^ = ^^^^−1,^^) meaning that the benefits ofthis architectural mechanism acting as a correction that compensates for the effects of badflows and bad warps is not realised. To prevent this, an annealing or regularisation schedulemay be introduced that gradually increases the effect that ^^^^^^^^ can have on the overall loss bychanging which loss terms contribute to the overall loss, and / or by using supervised learningusing a teacher network that has been trained to produce high quality ^^^^,^^^^^^^^ and ^^^^^^^^.For example, assume for the sake of argument that the loss function comprises, in addition tothe usual rate and distortion terms, the following further loss term ^^:^^ = ^^^^^^(^^^^−1,^^,^^^^^^^^ , ^^^^) + ^^^^^^(^^^^^^^^, ^^^^^^^^^^ℎ^^^^)where ^^^^^^ is a function that estimates a mean square error value between its input arguments,where ^^^^^^^^^^ℎ^^^^ is computed by a teacher network, and where ^^^^^^^^ and ^^^^−1,^^,^^^^^^^^ (the referenceframe corrected using ^^^^,^^^^^^^^) are computed by the networks being trained i.e. by ^^^^ with flowpart ^^^^ and residual part ℎ^^ . The teacher network may comprise a network having the same orsimilar architecture as ^^^^ that comprises a flow part ^^^^ and a residual part ℎ^^ but which hasbeen trained using a training scheme that allows it produce high-quality occlusion masksand miniframes, but not necessarily in a way that makes it feasible for end-device deployment.For example, the teacher’s architecture may be similar in structure to the student’s but muchlarger than what could be run on-device, and the unsupervised training schedule may takemuch longer than feasible for iterative development. By way of non-limiting example, if thestudent training schedule has 200,000 training steps, then the teacher training schedule mayhave 1,000,000 or more steps.During the forward pass, the new overall loss may thus be computed as:Loss = ^^ + ^^^^ + ^^Loss = ^^ + ^^^^ + ^^^^^^(^^^^−1,^^,^^^^^^^^ , ^^^^) + ^^^^^^(^^^^^^^^, ^^^^^^^^^^ℎ^^^^)However, at the start of training there is a conflict between the gradients from ^^^^^^(^^^^−1,^^,^^^^^^^^ , ^^^^)and ^^^^^^(^^^^^^^^, ^^^^^^^^^^ℎ^^^^). The miniframe ^^^^,^^^^^^^^ used to produce ^^^^−1,^^,^^^^^^^^ at this point is ofpoor quality, so ^^^^^^(^^^^−1,^^,^^^^^^^^ , ^^^^) penalises the occlusion mask ^^^^^^^^ for being anything thanall 1s (i.e. applying a full mask and thus not the poor quality miniframe ^^^^,^^^^^^^^ at all). Onthe other hand, the teacher’s occlusion mask ^^^^^^^^^^ℎ^^^^ generally has more 0 areas (i.e. usingthe miniframe ^^^^,^^^^^^^^ more) than the student because it assumes the teacher’s high-qualityminiframe ^^^^,^^^^^^^^ is to be used, and so they encourage those areas of the student’s mask tobe 0. To ensure that the mask is learned efficiently, gradients can be prevented from flowingfrom ^^^^^^(^^^^−1,^^,^^^^^^^^ , ^^^^) to the occlusion mask ^^^^^^^^ at the start of training and only graduallyintroduced with an annealing or regularisation schedule. Two ways to accomplish this areto detach the mask in the function that produces the warped reference frame for this part oftraining or to use the teacher’s occlusion mask ^^^^^^^^^^ℎ^^^^ . The latter option is preferred as itperforms slightly better but both options are possible.One reason the latter option is preferred is that the student occlusion mask quickly ^^^^^^^^becomes almost entirely all 1s (i.e. operating to fully mask out the miniframe) once thesupervised training is introduced. However, the only gradients that the miniframe ^^^^,^^^^^^^^receives in the initial training is through ^^^^^^(^^^^−1,^^,^^^^^^^^ , ^^^^), so if the all-1 student mask ^^^^^^^^is used then the miniframe ^^^^,^^^^^^^^ until the student mask improves.This leads to undertrained miniframes ^^^^,^^^^^^^^ in which is detrimental to performance. Usingthe teacher’s mask ^^^^^^^^^^ℎ^^^^ in the warping operation means that the miniframe ^^^^,^^^^^^^^ is used alot more (i.e. through the more frequent 0 areas of the teacher mask ^^^^^^^^^^ℎ^^^^ , and so thereforereceives useful gradients throughout training.Returning to the annealing or regularisation schedule, we define the overall loss as:Loss = ^^ + ^^^^ + ^^^^^^^^^^^^^^where:^^^^^^^^^^^^^^ = ^^·^^^^^^(^^^^−1,^^,^^^^^^^^ , ^^^^)+(1−^^) ·^^^^^^(^^^^−1,^^,^^^^^^^^^^^^^^.^^^^^^^^^^^^ℎ , ^^^^)+^^·^^^^^^(^^^^^^^^, ^^^^^^^^^^ℎ^^^^) where ^^^^−1,^^,^^^^^^^^^^^^^^.^^^^^^^^^^^^ℎ is the warped reference frame corrected using the miniframe ^^^^,^^^^^^^^ andmask ^^^^^^^^ detached so gradients are prevented from flowing through ^^^^^^(^^^^−1,^^,^^^^^^^^ , ^^^^)to the occlusion mask ^^^^^^^^, and where ^^^^−1,^^,^^^^^^^^ is the warped reference frame correctedusing the miniframe ^^^^,^^^^^^^^ and mask ^^^^^^^^ without the detach operation so that gradients canflow through to the mask ^^^^^^^^.It will be appreciated that gradients on the occlusion mask ^^^^^^^^ are thus of the form:^^^^ ^^^^^^^^ ^^^^^^^^(^^^^−1,^^,^^^^^^^^ , ^^^^) ^^^^^^^^(^^^^^^^^, ^^^^^^^^^^ℎ^^ )=+ ^^ · + ^^ · + ^^ · ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ which facilitates the gradual toggling in of the gradients from ^^^^^^(^^^^−1,^^,^^^^^^^^ , ^^^^) using theregularisation or annealing parameters ^^ and ^^.At the start of the training schedule, ^^ = 0, indicating that gradients may not propagate throughto the mask from ^^^^^^(^^^^−1,^^,^^^^^^^^ , ^^^^), thereby allowing the networks to focus on learning ^^^^,^^^^^^^^and ^^^^^^^^ as separate tasks. At the end of the training schedule, ^^ = 1, indicating that gradientsmay propagate fully through to the mask from ^^^^^^(^^^^−1,^^,^^^^^^^^ , ^^^^), better aligning the tasks oflearning the two tensors (i.e. to minimise ^^^^^^(^^^^−1,^^,^^^^^^^^ , ^^^^)). The value of ^^ may be set to 0at this point, so that the occlusion mask ^^^^^^^^ fully adapts to the miniframes ^^^^,^^^^^^^^ that thenetworks produce, rather than receiving encouragement from ^^^^^^(^^^^^^^^, ^^^^^^^^^^ℎ^^^^) to matchmasks that assume the teacher’s miniframes. The values of ^^ may gradually move from 0 to1 following some predefined schedule based on e.g. number of training steps or some othercriteria, for example how close to convergence of the overall loss to some predetermined lossvalue, or some other requirement.For completeness, it is noted that an annealing or regularisation schedule such as that illustratedabove is particularly effective at facilitating the use of ^^^^^^^^ and ^^^^,^^^^^^^^ in combination withthe encoder masks of concept 2 above. This is because the inventors have found that using^^^^^^^^ and ^^^^,^^^^^^^^ together with an encoder mask makes it easier for the networks to "cheat"during training as both the encoder mask and the occlusion mask ^^^^^^^^ are synergistically usedby the networks to control what information is sent by the encode side and / or masked outon the decode side. When used together, a local minimum can be reached in training wherethe networks mask away ^^^^,^^^^^^^^ for all input frames on the decode side and the encoder maskthen learns to mask on the encode side any information that would be used for ^^^^,^^^^^^^^. Theabove-described annealing schedule is highly effective at overcoming this and accordinglyhelps to facilitate combining encoder masks of concept 2 with the ^^^^^^^^ and ^^^^,^^^^^^^^ architecturalmechanisms of the present concept.As will be appreciated, whilst the examples described above use the flow hyper decoder 806and flow base decoder 209 to predict ^^^^^^^^ 1201 and ^^^^,^^^^^^^^ 1202 on the decode side, this isillustrative only. At the most general level, the decode side is correcting the warped previouslydecoded frame ^^^^−1,^^ using some reference that contains useful information to perform thatcorrection. In the examples above, this additional functionality is bootstrapped onto thenetworks that primarily serve other purposes (i.e. the networks of the flow part 206 of thepipeline). However it is additionally or alternatively envisaged that dedicated networks may beintroduced into the pipeline whose primary purpose is to produce the reference information tobe used for correcting ^^^^−1,^^. One such illustrative example is shown in Figure 13.Figure 13 corresponds to Figure 12 except now a dedicated encoder 1301 and decoder 1302pair 1302 that produce a bitstream portion 1303 are used to compute ^^^^,^^^^^^^^ 1202 from ^^^^(and / or ^^^^−1 not shown). This dedicated encoder 1301 and decoder pair 1302 may thus be saidin some ways to be similar to the I-frame module 201 of the compression pipeline in that itdoes not rely on temporal information and produces a representation of ^^^^ . Alternatively, itmay also use temporal information by including ^^^^−1 as an input. Additionally, the dedicatedencoder 1301 and decoder 1302 pair may operate at a much lower resolution than the I-framemodule. As previously described, the generation of ^^^^,^^^^^^^^ 1202 need not be performed atfull resolution as the purpose is primarily to serve as a source of information to correct forartefacts introduced by the flow network’s predicting flow poorly when failure modes arise.This purpose is served even when ^^^^,^^^^^^^^ is generated with low resolution information andthus is coarse when upsampled to the resolution of the reference frame when performingthe correction. That is, even a very coarse representation of information that approximatelyrepresents ^^^^ is better than the artefact-ridden creation of a flow network suffering from acatastrophic failure mode.Accordingly, one or more downsampling operations may be introduced into encoder 1301 tooperate on its input ^^^^ , allowing the encoder 1301 and decoder 1302 pair to operate in e.g. 1 / 2,1 / 4, 1 / 8, 1 / 16, 1 / 32 or some other lower resolution space compared to the I-frame module. Byoperating in lower-resolution space, the compute overhead of the dedicated encoder 1301 anddecoder 1302 pair may be kept small, helping to facilitate operation in real time or near realtime.Whilst Figure 13 still uses the base flow decoder 209 to predict the occlusion mask ^^^^^^^^1201 in addition to the optical flow information, this functionality may also be offloaded to adedicated network or networks (not shown). For example, this functionality may be offloadedonto the dedicated encoder 1301 decoder 1302 pair, or some other dedicated network ornetworks where the inputs include both ^^^^ and ^^^^−1.Further, whilst not shown in Figure 13, the dedicated encoder 1301 and decoder 1302 mayalso be supported by an associated hyper network and / or hyper hyper network in the same wayas the other networks of Figure 13.An advantage of offloading the computation of ^^^^^^^^ and ^^^^,^^^^^^^^ to one or more dedicatednetworks is that, in many cases, training stability is improved as the networks are not trying tolearn a set of weights that are able to simultaneously serve whatever their primary purpose is(e.g. like the base flow decoder and hyper flow decoder) as well as serve a secondary purposeof also producing ^^^^^^^^ and ^^^^,^^^^^^^^.The subject matter and the functional operations described in this specification can beimplemented in digital electronic circuitry, in tangibly-embodied computer software orfirmware, in computer hardware, including the structures disclosed in this specification andtheir structural equivalents, or in combinations of one or more of them. The subject matterdescribed in this specification can be implemented as one or more computer programs, i.e.,one or more modules of computer program instructions encoded on a tangible non transitoryprogram carrier for execution by, or to control the operation of, data processing apparatus.Alternatively or in addition, the program instructions can be encoded on an artificially generatedpropagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, thatis generated to encode information for transmission to suitable receiver apparatus for executionby a data processing apparatus. The computer storage medium can be a machine-readablestorage device, a machine-readable storage substrate, a random or serial access memory device,or a combination of one or more of them. The computer storage medium is not, however, apropagated signal.The term “data processing apparatus” encompasses all kinds of apparatus, devices, andmachines for processing data, including by way of example a programmable processor, acomputer, or multiple processors or computers. The apparatus can include special purposelogic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specificintegrated circuit). The apparatus can also include, in addition to hardware, code that createsan execution environment for the computer program in question, e.g., code that constitutesprocessor firmware, a protocol stack, a database management system, an operating system, ora combination of one or more of them.A computer program (which may also be referred to or described as a program, software, asoftware application, a module, a software module, a script, or code) can be written in anyform of programming language, including compiled or interpreted languages, or declarative orprocedural languages, and it can be deployed in any form, including as a stand alone program oras a module, component, subroutine, or other unit suitable for use in a computing environment.A computer program may, but need not, correspond to a file in a file system. A program can bestored in a portion of a file that holds other programs or data, e.g., one or more scripts storedin a markup language document, in a single file dedicated to the program in question, or inmultiple coordinated files, e.g., files that store one or more modules, sub programs, or portionsof code. A computer program can be deployed to be executed on one computer or on multiplecomputers that are located at one site or distributed across multiple sites and interconnected bya communication network.The processes and logic flows described in this specification can be performed by one or moreprogrammable computers executing one or more computer programs to perform functionsby operating on input data and generating output. The processes and logic flows can also beperformed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g.,an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).Computers suitable for the execution of a computer program include, by way of example,can be based on general or special purpose microprocessors or both, or any other kind ofcentral processing unit. Generally, a central processing unit will receive instructions and datafrom a read only memory or a random access memory or both. The essential elements ofa computer are a central processing unit for performing or executing instructions and oneor more memory devices for storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer data to, or both, one or moremass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks.However, a computer need not have such devices. Moreover, a computer can be embedded inanother device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio orvideo player, a VR headset, a game console, a Global Positioning System (GPS) receiver, aserver, a mobile phones, a tablet computer, a notebook computer, a music player, an e-bookreader, a laptop or desktop computer, a PDAs, a smart phone, or other stationary or portabledevices, that includes one or more processors and computer readable media, or a portablestorage device, e.g., a universal serial bus (USB) flash drive, to name just a few.Computer readable media suitable for storing computer program instructions and data includeall forms of non-volatile memory, media and memory devices, including by way of examplesemiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magneticdisks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM andDVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in,special purpose logic circuitry.The subject matter described in this specification can be implemented in a computing systemthat includes a back end component, e.g., as a data server, or that includes a middlewarecomponent, e.g., an application server, or that includes a front end component, e.g., a clientcomputer having a graphical user interface or a Web browser through which a user can interactwith an implementation of the subject matter described in this specification, or any combinationof one or more such back end, middleware, or front end components. The components of thesystem can be interconnected by any form or medium of digital data communication, e.g., acommunication network. Examples of communication networks include a local area network(“LAN”) and a wide area network (“WAN”), e.g., the Internet.The computing system can include clients and servers. A client and server are generally remotefrom each other and typically interact through a communication network. The relationship ofclient and server arises by virtue of computer programs running on the respective computersand having a client-server relationship to each other.While this specification contains many specific implementation details, these should beconstrued as descriptions of features that may be specific to particular examples of particularinventions. Certain features that are described in this specification in the context of separateexamples can also be implemented in combination in a single example. Conversely, variousfeatures that are described in the context of a single example can also be implemented inmultiple examples separately or in any suitable subcombination.Similarly, while operations are depicted in the drawings in a particular order, this should notbe understood as requiring that such operations be performed in the particular order shownor in sequential order, or that all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processing may be advantageous.Moreover, the separation of various system modules and components in the examples describedabove should not be understood as requiring such separation in all examples, and it should beunderstood that the described program components and systems can generally be integratedtogether in a single software product or packaged into multiple software products.

Claims

CLAIMS1. A method for lossy image or video encoding, transmission, and decoding, the methodcomprising the steps of:receiving a first image and a second image at a first computer system;with a first neural network, producing optical flow information using the first image andthe second image, the optical flow information being indicative of a difference between thefirst image and the second image;with a second neural network, producing residual information using the first image andthe optical flow information, the residual information being indicative of a difference betweenthe first image and the second image;transmitting the residual information to a second computer system;with a third neural network, decoding the residual information to produce an intermediateoutput image; andcombining a portion of a previously decoded image with a portion of the intermediateoutput image to produce a final output image, wherein the final output image is an approximationof the first image.

2. The method of claim 1, comprising extracting information from one or more intermediatelayers of the first neural network, and wherein the portions are based on the extractedinformation.

3. The method of claim 2, wherein the extracted information comprises one or more outputsfrom one or more convolution layers and / or activation layers of the first neural network.

4. The method of any of claims 1 to 3, comprising estimating the portions by applying a maskto the previously decoded image and / or the intermediate output image.

5. The method of claim 4, when dependent on claim 2, wherein the mask is based on theextracted information.

6. The method of claim 4 or 5, wherein applying the mask comprises applying the mask toone of the previously decoded image or intermediate output image, and applying an inverse ofthe mask to the other of the previously decoded image or intermediate output.

7. The method of any of claims 4 to 6, wherein the mask comprises a non-binary mask.

8. The method of any of claims 4 to 7, comprising producing the mask using the third neuralnetwork.

9. The method of claim 8, wherein producing the mask using the third neural networkcomprises applying one or more convolution layers and / or activation layers to the residualinformation and the extracted information.

10. The method of any of claims 4 to 7, comprising producing the mask using a fourth neuralnetwork.

11. The method of claim 10, wherein producing the mask using the fourth neural networkcomprises applying one or more convolution layers and / or activation layers to the residualinformation and the extracted information.

12. The method of any of claims 4 to 11, wherein the mask specifies an extent to which thepreviously decoded image contributes to the final output image compared to the extent towhich the intermediate output image contributes to the final output image.

13. The method of any of claims 4 to 12, wherein the mask is applied to the previously decodedimage and / or the intermediate output image on a pixel-wise basis to perform a pixel-wise,partial frame skip of the first image.

14. The method of any of claims 1 to 13, wherein decoding the residual information with thethird neural network uses the optical flow information output by the first neural network,15. The method of any of claims 1 to 14, wherein the previously decoded image is anapproximation of the second image.

16. The method of any of claims 1 to 13, comprising repeating said producing, transmitting,decoding and combining for a sequence of frames of a video to reconstruct the sequence offrames at the second computer system to mitigate image artefacts in the reconstructed sequenceof frames.

17. A data processing system configured to perform the method of any one of claims 1 to 16.

18. A method for lossy image or video receipt and decoding, the method comprising the stepsof: decoding optical flow information at a first computer system using a first neuralnetwork, the optical flow information being indicative of a difference between a first imageand the second image;decoding residual information at the first computer system using a second neural networkto produce an intermediate output image, the residual information being indicative of adifference between the first image and the second image;combining a portion of a previously decoded image with a portion of the intermediateoutput image to produce a final output image, wherein the final output image is an approximationof the first image.

19. A data processing apparatus configured to perform the method of claim 18.

20. A computer program comprising instructions which, when the program is executed by acomputer, cause the computer to carry out the method of any of claims 1 to 18.

21. A computer-readable storage medium comprising instructions which, when executed by acomputer, cause the computer to carry out the method of any of claims 1 to 18.

22. A method of training one or more neural networks, the one or more neural networks beingfor use in lossy image or video encoding, transmission and decoding, the method comprisingthe steps of:receiving a first image and a second image at a first computer system;with a first neural network, producing optical flow information using the first image andthe second image, the optical flow information being indicative of a difference between thefirst image and the second image;with a second neural network, producing residual information using the first image andthe optical flow information, the residual information being indicative of a difference betweenthe first image and the second image;with a third neural network, decoding the residual information to produce an intermediateoutput image; andcombining a portion of a previously decoded image with a portion of the intermediateoutput image to produce a final output image, wherein the final output image is an approximationof the first image;evaluating a function based on a difference between the first image and the output image;updating the parameters of the first, second and / or third neural networks based on theevaluated function; andrepeating the above steps using a first set of input images to produce first, second and / orthird trained neural networks.

23. A method for lossy image or video encoding, transmission, and decoding, the methodcomprising the steps of:receiving a first image and a second image at a first computer system;with a first neural network, producing optical flow information using the first image andthe second image, the optical flow information being indicative of a difference between thefirst image and the second image;extracting information from one or more intermediate layers of the first neural network;with a second neural network, producing residual information using the first image andthe optical flow information, the residual information being indicative of a difference betweenthe first image and the second image;transmitting the residual information to a second computer system;with a third neural network, decoding the residual information to produce an intermediateoutput image; andmodifying the intermediate output image, based on the information extracted from theone or more intermediate layers, to produce a modified output image, wherein the modifiedoutput image is an approximation of the first image.

24. The method of claim 23, comprising using the extracted outputs from the one or moreintermediate layers to produce a mask.

25. The method of claim 24, wherein modifying the intermediate output image comprisesapplying the mask to the intermediate output image.

26. The method of any of claims 23 to 25, wherein the mask comprises at least three differentvalues between a minimum value and a maximum value.

27. The method of any of claims 23 to 26, comprising producing the mask with one or morelayers of the third neural network.

28. The method of any of claims 23 to 27, comprising producing the mask with a fourth neuralnetwork.

29. The method of any of claims 23 to 28, wherein the information extracted from one ormore intermediate layers of the first neural network comprises one or more outputs from oneor more convolution layers and / or activation layers of the first neural network.

30. The method of any of claims 23 to 29, comprising repeating said steps for a sequence offrames of a video to reconstruct the sequence of frames of the video at the second computersystem.

31. The method of claim 30, wherein repeating said steps for a sequence of frames of a videois performed to mitigate image artefacts in the reconstructed sequence of frames.

32. A data processing system configured to perform the method of any one of claims 23 to 31.

33. A method for lossy image or video receipt and decoding, the method comprising the stepsof: decoding optical flow information at a first computer system using a first neuralnetwork, the optical flow information being indicative of a difference between a first imageand the second image;extracting information from one or more intermediate layers of the first neural network;decoding residual information at the first computer system using a second neural networkto produce an intermediate output image, the residual information being indicative of adifference between the first image and the second image;modifying the intermediate output image, based on the information extracted from theone or more intermediate layers of the first neural network, to produce a modified output image,wherein the modified output image is an approximation of the first image.

34. A data processing apparatus configured to perform the method of claim 33.

35. A computer program comprising instructions which, when the program is executed by acomputer, cause the computer to carry out the method of any of claims 23 to 33.

36. A computer-readable storage medium comprising instructions which, when executed by acomputer, cause the computer to carry out the method of any of claims 23 to 33.

37. A method for lossy image or video encoding, transmission and decoding, the methodcomprising the steps of:receiving a first image at a first computer system;encoding the first image using a first neural network to produce a latent representation;at least partially masking the latent representation;transmitting the masked latent representation to a second computer system;decoding the masked latent representation using a second neural network to produce anoutput image, wherein the output image is an approximation of the first image.

38. The method of claim 37, wherein the at least partially masking the latent representationreduces the entropy of the distribution of the latent representation.

39. The method of claim 37 or 38, wherein said masking comprises producing a mask basedon the first image.

40. The method of claim 39, comprising producing the mask with a third neural network.

41. The method of any of claims 38 to 40, comprising entropy encoding the maskedlatent representation into a bitstream, wherein transmitting the masked latent representationcomprises transmitting the bitstream, and wherein the step of at least partially masking thelatent representation reduces a size of the bitstream after said entropy encoding compared to abitstream produced without the step of at least partially masking.

42. The method of any of claims 38 to 41, wherein the at least partially masking the latentrepresentation comprises at least partially masking a normalised representation of the latentrepresentation.

43. The method of claim 42, comprising producing the normalised representation of the latentrepresentation using one or more first learned scale and / or zero point parameters.

44. The method of claim 43, comprising encoding the latent representation using a fourth neuralnetwork to produce a hyper latent representation, and decoding the hyper latent representationusing a fifth neural network to produce the one or more first learned scale and / or zero pointparameters.

45. The method of claim 44, comprising at least partially masking the hyper latent representationto reduce an entropy of a distribution of the hyper latent representation.

46. The method of any of claims 44 to 45, comprising entropy encoding and transmitting themasked hyper latent representation to the second computer system.

47. The method of claim 46, wherein the at least partially masking the hyper latent repre-sentation comprises at least partially masking a normalised representation of the hyper latentrepresentation.

48. The method of claim 47, comprising producing the normalised representation of the hyperlatent representation using one or more second learned scale and / or zero point parameters.

49. The method of claim 48, comprising encoding the hyper latent representation using a sixthneural network to produce a hyper hyper latent representation, and decoding the hyper hyperlatent representation using a seventh neural network to produce the second learned scale and / orzero point parameters.

50. The method of claim 49, comprising at least partially masking the hyper hyper latentrepresentation to reduce an entropy of a distribution of the hyper hyper latent representation.

51. The method of claim 50, wherein the at least partially masking the hyper hyper latentrepresentation comprises at least partially masking a normalised representation of the hyperhyper latent representation.

52. The method of claim 50 or 51, comprising entropy encoding and transmitting the maskedhyper hyper latent representation to the second computer system.

53. A method for lossy image or video encoding and transmission, the method comprising thesteps of:receiving a first image at a first computer system;encoding the first image using a first neural network to produce a latent representation;at least partially masking the latent representation;transmitting the masked latent representation to a second computer system.

54. A method for lossy image or video receipt and decoding, the method comprising the stepsof: receiving at a second computer system a masked latent representation produced byencoding a first image using a first neural network and by at least partially masking the latentrepresentation; decoding the masked latent representation using a second neural network to produce anoutput image, wherein the output image is an approximation of the first image.

55. A data processing apparatus configured to perform the method of any of claims 37 to 54.

56. A computer program comprising instructions which, when the program is executed by acomputer, cause the computer to carry out the method of any of claims 37 to 54.

57. A computer-readable storage medium comprising instructions which, when executed by acomputer, cause the computer to carry out the method of any of claims 37 to 54.

58. A method for lossy image or video encoding, transmission, and decoding, the methodcomprising the steps of:receiving a first image and a second image at a first computer system;with a first neural network, producing optical flow information using the first image andthe second image, the optical flow information being indicative of a difference between thefirst image and the second image;with a second neural network, producing residual information using the first image andthe optical flow information, the residual information being indicative of a difference betweenthe first image and the second image;at least partially masking the residual information;transmitting the masked residual information to a second computer system;with a third neural network, decoding the masked residual information to produce outputimage wherein the output image is an approximation of the first image.

59. The method of claim 58, wherein the at least partially masking the residual informationreduces the entropy of the distribution of the the residual information.

60. The method of claim 58 or 59, wherein said masking comprises producing a mask basedon the first image and / or the second image.

61. The method of claim 60, comprising producing the mask with a fourth neural network.

62. The method of any of claims 58 to 61, comprising entropy encoding the masked residualinformation into a bitstream, wherein transmitting the masked residual information comprisestransmitting the bitstream, and wherein the step of at least partially masking the residualinformation reduces a size of the bitstream after said entropy encoding compared to a bitstreamproduced without the step of at least partially masking.

63. The method of any of claims 58 to 62, wherein the at least partially masking the residualinformation comprises at least partially masking a normalised representation of the residualinformation.

64. The method of claim 63, comprising producing the normalised representation of theresidual information using one or more first learned scale and / or zero point parameters.

65. The method of claim 64, comprising encoding the residual information using a fifth neuralnetwork to produce a hyper latent representation of the residual information, and decoding thehyper latent representation of the residual information using a sixth neural network to producethe one or more first learned scale and / or zero point parameters.

66. The method of claim 65, comprising at least partially masking the hyper latent representationof the residual information to reduce an entropy of a distribution of the hyper latent representationof the residual information.

67. The method of claim 66, comprising entropy encoding and transmitting the masked hyperlatent representation of the residual information to the second computer system.

68. The method of any of claims 66 to 67, wherein the at least partially masking the hyper latentrepresentation of the residual information comprises at least partially masking a normalisedrepresentation of the hyper latent representation of the residual information.

69. The method of claim 68, comprising producing the normalised representation of the hyperlatent representation of the residual information using one or more second learned scale and / orzero point parameters.

70. The method of claim 69, comprising encoding the hyper latent representation of the residualinformation using a seventh neural network to produce a hyper hyper latent representation ofthe residual information, and decoding the hyper hyper latent representation of the residualinformation using an eighth neural network to produce the second learned scale and / or zeropoint parameters.

71. The method of claim 70, comprising at least partially masking the hyper hyper latentrepresentation of the residual information to reduce an entropy of a distribution of the hyperhyper latent representation of the residual information.

72. The method of claim 71, wherein the at least partially masking the hyper hyper latentrepresentation of the residual information comprises at least partially masking a normalisedrepresentation of the hyper hyper latent representation of the residual information.

73. The method of claim 71 or 72, comprising entropy encoding and transmitting the maskedhyper hyper latent representation of the residual information to the second computer system.

74. A method for lossy image or video encoding and transmission, the method comprising thesteps of:receiving a first image and a second image at a first computer system;with a first neural network, producing optical flow information using the first image andthe second image, the optical flow information being indicative of a difference between thefirst image and the second image;with a second neural network, producing residual information using the first image andthe optical flow information, the residual information being indicative of a difference betweenthe first image and the second image;at least partially masking the residual information; andtransmitting the masked residual information to a second computer system.

75. A method for lossy image or video receit and decoding, the method comprising the stepsof: receiving at a second computer system masked residual information produced by, with afirst neural network, producing optical flow information using first and second images and by,with a second neural network, producing residual information using the first image and theoptical flow information, the residual information being indicative of a difference between thefirst image and the second image, and by at least partially masking the residual information;and with a third neural network, decoding the masked residual information to produce outputimage wherein the output image is an approximation of the first image.

76. A data processing apparatus configured to perform the method of any of claims 58 to 75.

77. A computer program comprising instructions which, when the program is executed by acomputer, cause the computer to carry out the method of any of claims 58 to 75.

78. A computer-readable storage medium comprising instructions which, when executed by acomputer, cause the computer to carry out the method of any of claims 58 to 75.

79. A method for lossy image or video encoding, transmission, and decoding, the methodcomprising the steps of:receiving a first image and a second image of an image sequence at a first computersystem; with a first neural network, producing optical flow information using the first image andthe second image, the optical flow information being indicative of a difference between thefirst image and the second image;at least partially masking the optical flow information;transmitting a representation of the masked optical flow information to a second computersystem; with a second neural network decoding the representation of the masked optical flowinformation to produce an approximation of the optical flow information; andwith a third neural network, producing an output image using the optical flow information,wherein the output image is an approximation of the first image.

80. The method of claim 79, wherein the at least partially masking the optical flow informationreduces the entropy of the distribution of the the optical flow information.

81. The method of claim 80, wherein said masking comprises producing a mask based on thefirst image and / or the second image.

82. The method of claim 80 or 81, comprising producing the mask with a fourth neuralnetwork.

83. The method of any of claims 79 to 82, comprising entropy encoding the masked opticalflow information into a bitstream, wherein transmitting the masked optical flow informationcomprises transmitting the bitstream, and wherein the step of at least partially masking theoptical flow information reduces a size of the bitstream after said entropy encoding comparedto a bitstream produced without the step of at least partially masking.

84. The method of any of claims 79 to 83, wherein the at least partially masking the opticalflow information comprises at least partially masking a normalised representation of the opticalflow information.

85. The method of claim 84, comprising producing the normalised representation of theoptical flow information using one or more first learned scale and / or zero point parameters.

86. The method of claim 85, comprising encoding the optical flow information using a fifthneural network to produce a hyper latent representation of the optical flow information, anddecoding the hyper latent representation of the optical flow information using a sixth neuralnetwork to produce the one or more first learned scale and / or zero point parameters.

87. The method of claim 86, comprising at least partially masking the hyper latent representationof the optical flow information to reduce an entropy of a distribution of the hyper latentrepresentation of the optical flow information.

88. The method of claim 87, comprising entropy encoding and transmitting the masked hyperlatent representation of the optical flow information to the second computer system.

89. The method of claim 87 or 88, wherein the at least partially masking the hyper latentrepresentation of the optical flow information comprises at least partially masking a normalisedrepresentation of the hyper latent representation of the optical flow information.

90. The method of claim 89, comprising producing the normalised representation of the hyperlatent representation of the optical flow information using one or more second learned scaleand / or zero point parameters.

91. The method of claim 90, comprising encoding the hyper latent representation of the opticalflow information using a seventh neural network to produce a hyper hyper latent representationof the optical flow information, and decoding the hyper hyper latent representation of theoptical flow information using an eighth neural network to produce the second learned scaleand / or zero point parameters.

92. The method of claim 91, comprising at least partially masking the hyper hyper latentrepresentation of the optical flow information to reduce an entropy of a distribution of thehyper hyper latent representation of the optical flow information.

93. The method of claim 92, wherein the at least partially masking the hyper hyper latentrepresentation of the optical flow information comprises at least partially masking a normalisedrepresentation of the hyper hyper latent representation of the optical flow information.

94. The method of claim 92 or 93, comprising entropy encoding and transmitting the maskedhyper hyper latent representation of the optical flow information to the second computersystem.

95. A method for lossy image or video encoding and transmission, the method comprising thesteps of:receiving a first image and a second image of an image sequence at a first computersystem; with a first neural network, producing optical flow information using the first image andthe second image, the optical flow information being indicative of a difference between thefirst image and the second image;at least partially masking the optical flow information; andtransmitting a representation of the masked optical flow information to a second computersystem.

96. A method for lossy image or video receipt and decoding, the method comprising the stepsof: receiving at a second computer system a representation of masked optical flow informationproduced with a first neural network using a first image and second image, and produced by atleast partially masking the optical flow information;with a second neural network decoding the representation of the masked optical flowinformation to produce an approximation of the optical flow information; andwith a third neural network, producing an output image using the optical flow information,wherein the output image is an approximation of the first image.

97. A data processing apparatus configured to perform the method of any of claims 79 to 96.

98. A computer program comprising instructions which, when the program is executed by acomputer, cause the computer to carry out the method of any of claims 79 to 96.

99. A computer-readable storage medium comprising instructions which, when executed by acomputer, cause the computer to carry out the method of any of claims 79 to 96.

100. A method for lossy image or video encoding, transmission, and decoding, the methodcomprising the steps of:receiving a first image and a second image of an image sequence at a first computersystem; with a first neural network, producing optical flow information using the first image andthe second image, the optical flow information being indicative of a difference between thefirst image and the second image;transmitting a representation of the optical flow information to a second computer system;with a second neural network decoding the representation of the optical flow informationto produce an approximation of the optical flow information;warping a previously decoded representation of the second image using the approximationof the optical flow information;combining the warped previously decoded representation of the second image with arepresentation of the first image; andwith a third neural network, producing an output image using the combined, warpedpreviously decoded representation of the second image, wherein the output image is anapproximation of the first image.

101. The method of claim 100, comprising producing the representation of the first imagewith the second neural network.

102. The method of claim 101, comprising producing a mask with the second neural network,and using the mask to combine the representation of the first image with the warped previouslydecoded representation of the second image.

103. The method of claim 100, comprising producing the representation of the first imagewith a fourth neural network using the first image.

104. The method of claim 103, comprising producing a mask with the fourth neural networkusing the first image, and using the mask to combine the representation of the first image withthe warped previously decoded representation of the second image.

105. The method of claim 102 or 104, wherein said combining comprises performing aweighted sum of the representation of the first image and the warped previously decodedrepresentation of the second image using weights based on values of the mask.

106. The method of any of claims 100 to 105, wherein said combining reduces a differencebetween (i) the warped previously decoded representation of the second image and (ii) the firstimage.

107. The method of any of claims 100 to 106, wherein said combining reduces a differencebetween (i) the output image produced with the third neural network using the combined,warped previously decoded representation of the second image and (ii) the first image.

108. The method of any of claims 100 to 107, wherein said combining corrects artefacts in thewarped previously decoded representation of the second image.

109. A method for lossy image or video encoding and transmission, the method comprisingthe steps of:receiving a first image and a second image of an image sequence at a first computersystem; with a first neural network, producing optical flow information using the first image andthe second image, the optical flow information being indicative of a difference between thefirst image and the second image; andtransmitting a representation of the optical flow information to a second computer system.

110. A method for lossy image or video receipt and decoding, the method comprising the stepsof: receiving a representation of optical flow information at a second computer system, theoptical flow information produced by a first neural network using a first image and a secondimage of an image sequence and being indicative of a difference between the first image andthe second image;with a second neural network decoding the representation of the optical flow informationto produce an approximation of the optical flow information;warping a previously decoded representation of the second image using the approximationof the optical flow information;combining the warped previously decoded representation of the second image with arepresentation of the first image; andwith a third neural network, producing an output image using the combined, warpedpreviously decoded representation of the second image, wherein the output image is anapproximation of the first image.

111. A method of training one or more neural networks, the one or more neural networks beingfor use in lossy image or video encoding, transmission and decoding, the method comprisingthe steps of:receiving a first image and a second image of an image sequence at a first computersystem; with a first neural network, producing optical flow information using the first image andthe second image, the optical flow information being indicative of a difference between thefirst image and the second image;transmitting a representation of the optical flow information to a second computer system;with a second neural network decoding the representation of the optical flow informationto produce an approximation of the optical flow information;warping a previously decoded representation of the second image using the approximationof the optical flow information;combining the warped previously decoded representation of the second image with arepresentation of the first image;with a third neural network, producing an output image using the combined, warpedpreviously decoded representation of the second image, wherein the output image is anapproximation of the first image;evaluating a function based on a difference between the first image and the output image;updating the parameters of the first, second and / or third neural networks based on theevaluated function; andrepeating the above steps using a first set of input images to produce first, second and / orthird trained neural networks.

112. The method of claim 111, comprising producing the representation of the first imagewith the second neural network.

113. The method of claim 112, comprising producing a mask with the second neural network,and using the mask to combine the representation of the first image with the warped previouslydecoded representation of the second image.

114. The method of claim 111, comprising producing the representation of the first imagewith a fourth neural network using the first image.

115. The method of claim 114, comprising producing a mask with the fourth neural networkusing the first image, and using the mask to combine the representation of the first image withthe warped previously decoded representation of the second image.

116. The method of claim 112 or 114, wherein the function is further based on a differencebetween (i) the representation of the first image produced by the second neural network or thefourth neural network and (ii) a representation of the first image produced by a teacher neuralnetwork.

117. The method of claim 113 or 115, wherein the function is further based on a differencebetween (i) the mask produced by the second neural network or the fourth neural network and(ii) a mask produced by a teacher neural network.

118. The method of any of claims 111 to 117, wherein said evaluating the function comprisesregularising propagation of gradients through one or more terms of the function.

119. The method of claim 118 when dependent on claim 117, wherein said evaluating thefunction comprises regularising propagation of gradients through a term of the functionassociated with the difference between (i) the mask produced by the second neural network orthe fourth neural network and (ii) the mask produced by the teacher neural network.

120. The method of claim 118 or 119, comprising controlling said regularising based on apredetermined regularisation schedule.

121. A data processing apparatus configured to perform the method of any of claims 100 to120.

122. A computer program comprising instructions which, when the program is executed by acomputer, cause the computer to carry out the method of any of claims 100 to 120.

123. A computer-readable storage medium comprising instructions which, when executed by acomputer, cause the computer to carry out the method of any of claims 100 to 120.

Images