Method and data processing system for lossy image or video encoding, transmission and decoding

Abstract

A method for lossy image or video encoding, transmission, and decoding, the method comprising the steps of: receiving a first image at a first computer system; encoding the first image using a first neural network to produce a latent representation; at least partially masking the latent representation; transmitting the masked latent representation to a second computer system; decoding the masked latent representation using a second neural network to produce an output image, wherein the output image is an approximation of the first image.

Description

Background Technology

[0001] The present invention relates to methods and systems for encoding, transmitting and decoding lossy images or videos, methods, apparatuses, computer programs and computer-readable storage media for encoding and transmitting lossy images or videos, and methods, apparatuses, computer programs and computer-readable storage media for receiving and decoding lossy images or videos.

[0002] Users of communication networks are increasingly demanding more image and video content. This increasing demand extends not only to the quantity of images viewed but also to the duration of video playback; there is also a growing need for higher resolution content. This places increasing demands on communication networks and increases their energy consumption due to the need to transmit larger volumes of data.

[0003] To mitigate the impact of these issues, image and video content is compressed for transmission across networks. Image and video content compression can be lossless or lossy. In lossless compression, images or videos are compressed so that all original information can be recovered during decompression. However, the reduction in data volume achievable with lossless compression is limited. In lossy compression, some information from the image or video is lost during the compression process. Known compression techniques attempt to minimize significant information loss by removing information that causes changes to the decompressed image or video, changes that are not particularly noticeable to the human visual system. JPEG, JPEG2000, AVC, HEVC, and AVI are examples of image and / or video file compression processes.

[0004] Generally, known lossy image compression techniques use the spatial correlation between pixels in an image to remove redundant information during compression. For example, in an image of a blue sky, if a given pixel is blue, then its neighboring pixels and their neighbors are also likely to be blue. Therefore, it is not necessary to retain all the original pixel data. Instead, we can retain only a subset of pixels that occupy fewer bits and use the information derived from spatial correlation to infer the pixel values ​​of other pixels.

[0005] Similar methods are applied in known lossy video compression techniques. That is, spatial correlation between pixels allows for the removal of redundant information during compression. However, in video compression, there is further information redundancy in the form of temporal correlation. For example, in a video of an aircraft flying over a blue sky background, most of the blue sky pixels remain completely unchanged between frames. Most of the blue sky pixel data in the frame at position t = 0 is the same as at position t = 10. Storing this identical, temporally correlated information is inefficient. Instead, only a subset of the blue sky pixel data from the frames is stored, while the remainder is inferred from information derived from temporal correlation.

[0006] Especially in the field of lossy video compression, removing redundant time-related information from a video sequence is called inter-frame redundancy.

[0007] One technique widely used in standard video compression algorithms that employs inter-frame redundancy involves dividing video frames into three types: I-frames, P-frames, and B-frames. Each frame type carries different properties regarding its encoding and decoding processes, playing a different role in achieving high compression ratios while maintaining acceptable visual quality.

[0008] I-frames, or intra-coded frames, serve as the basis for video sequences. These frames are independent, each encoding a complete image without referencing any other frames. In terms of compression, I-frames are the least compressed of all frame types, thus carrying the most data. However, their independence offers several benefits, including serving as a starting point for decompression and enabling random access, which is crucial for features such as fast-forwarding or rewinding video.

[0009] P-frames, or predicted frames, utilize temporal redundancy in a video sequence to achieve greater compression. Unlike I-frames, which encode the entire image, P-frames represent the difference between themselves and the nearest preceding I-frame or P-frame. This process, known as motion compensation, identifies and encodes only the changes that have occurred, significantly reducing the amount of data transmitted. However, P-frames rely on previous frames for decoding. Therefore, any errors during encoding or transmission can propagate to subsequent frames, affecting overall video quality.

[0010] B-frames, or bidirectional prediction frames, represent the highest level of compression. Unlike P-frames, B-frames use both previous and subsequent frames as references during their encoding process. By predicting both forward and backward motion in time, B-frames only encode differences that cannot be accurately predicted based on previous and subsequent frames, resulting in significant data reduction. Although this bidirectional prediction makes the generation and decoding of B-frames more complex, it does not propagate decoding errors because they are not used as references for other frames. Artificial intelligence (AI)-based compression techniques achieve the compression and decompression of images and videos by using trained neural networks during the compression and decompression processes. Typically, during the training of the neural network, the differences between the original image and video and the compressed and decompressed images and videos are analyzed, and the parameters of the neural network are modified to reduce these differences while minimizing the data required to transmit the content. However, AI-based compression methods may achieve poor compression results in terms of the appearance of the compressed image or video or the amount of information that needs to be transmitted.

[0011] An example of an AI-based image compression process involving a hypernetwork is described in Ballé, Johannes, et al., “Variational image compression with a scale hyperprior,” arXiv preprint arXiv:1802.01436 (2018), which is incorporated herein by reference.

[0012] An example of an AI-based video compression method is illustrated in “Scale-space flow for end-to-end optimized video compression” by Agustsson, E., Minnen, D., Johnston, N., Balle, J., Hwang, SJ. and Toderici, G. (2020) in the proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 8503–8512), which is incorporated herein by reference.

[0013] Further examples of AI-based video compression methods are illustrated in “Neural video compression using gans for detail synthesis and propagation” by Mentzer, F., Agustsson, E., Ballé, J., Minnen, D., Johnston, N., and Toderici, G. (November 2022), Proceedings of Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Part XXVI (pp. 562–578), which is incorporated herein by reference. Summary of the Invention

[0014] According to one aspect of this disclosure, a method for encoding, transmitting, and decoding lossy images or videos is provided. The method includes the steps of receiving a first image and a second image at a first computer system. Furthermore, the method further includes the step of generating optical flow information using the first and second images via a first neural network. The optical flow information indicates a difference between the first and second images. The method also includes the step of generating residual information using the first image and the optical flow information via a second neural network. The residual information indicates a difference between the first and second images. Furthermore, the method includes the step of transmitting the residual information to a second computer system. The method further includes the step of decoding the residual information using a third neural network to generate an intermediate output image. Finally, the method includes the step of combining a portion of the previously decoded image with a portion of the intermediate output image to generate a final output image. The final output image is an approximation of the first image.

[0015] Optionally, the method includes extracting information from one or more intermediate layers of the first neural network. As described above, this part is based on the extracted information.

[0016] Optionally, as described above, the method may involve information extraction. The extracted information includes one or more outputs. These outputs are derived from one or more convolutional layers and / or activation layers of a first neural network.

[0017] Optionally, the method includes estimating the portion. The estimation of the portion is achieved by applying a mask. The mask is applied to a previously decoded image or an intermediate output image, or both.

[0018] Optionally, as described above, the method further includes a mask. The mask is based on the extracted information.

[0019] Optionally, the method may involve applying the mask. In this context, the mask is applied to one of the previously decoded images or intermediate output images. Alternatively, the inverse of the mask is applied to the other of the previously decoded images or intermediate outputs.

[0020] Optionally, as described above, the method includes a mask. The mask in the method includes a non-binary mask.

[0021] Optionally, the method includes generating a mask. As described above, a third neural network is used to facilitate mask generation.

[0022] Optionally, the method further includes using a third neural network to generate a mask. In this context, generating a mask involves applying one or more convolutional layers and / or activation layers to the residual information and the extracted information.

[0023] Optionally, the method includes generating a mask. A fourth neural network is used to generate the mask.

[0024] Optionally, the method further includes using a fourth neural network to generate a mask. In this context, generating a mask involves applying one or more convolutional layers and / or activation layers to the residual information and the extracted information.

[0025] Optionally, the method may involve using a mask. The mask specifies the degree to which the previously decoded image contributes to the final output image. This contribution is compared with the degree to which intermediate output images contribute to the final output image.

[0026] Optionally, the method may involve applying a mask. The mask is applied on a pixel-by-pixel basis to the previously decoded image and / or intermediate output image. Applying the mask is used to perform pixel-by-pixel partial frame skipping of the first image.

[0027] Optionally, the method involves decoding the residual information using a third neural network. In this context, the decoding process utilizes optical flow information output by the first neural network.

[0028] Alternatively, as described above, the method may involve using a previously decoded image. In this context, the previously decoded image is an approximation of the second image.

[0029] Optionally, the method includes the steps of repeatedly generating, transmitting, decoding, and combining a sequence of video frames. The sequence of frames is reconstructed at a second computer system. These steps are repeated to mitigate image artifacts in the reconstructed sequence of frames.

[0030] According to one aspect of this disclosure, a data processing system is provided. The data processing system is configured to perform the methods described above.

[0031] According to one aspect of this disclosure, a method for receiving and decoding lossy images or videos is provided. The method includes the step of decoding optical flow information using a first neural network at a first computer system. The optical flow information indicates a difference between a first image and a second image. The method further includes the step of decoding residual information using a second neural network at the first computer system to generate an intermediate output image. The residual information indicates a difference between the first image and the second image. Furthermore, the method includes the step of combining a portion of the previously decoded image with a portion of the intermediate output image to generate a final output image. The final output image is an approximation of the first image.

[0032] Optionally, a data processing apparatus is provided. The data processing apparatus is configured to perform the method described above.

[0033] Optionally, a computer program is provided. The computer program includes instructions. When executed by a computer, the instructions cause the computer to perform the method described above.

[0034] According to one aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium includes instructions. When these instructions are executed by a computer, the computer performs the method as described above.

[0035] According to one aspect of this disclosure, a method is provided for training one or more neural networks for use in lossy image or video encoding, transmission, and decoding, the method comprising the steps of: receiving a first image and a second image at a first computer system; generating optical flow information using the first image and the second image using a first neural network, the optical flow information indicating a difference between the first image and the second image; generating residual information using the first image and the optical flow information using a second neural network, the residual information indicating a difference between the first image and the second image; decoding the residual information using a third neural network to generate an intermediate output image; and combining a portion of the previously decoded image with a portion of the intermediate output image to generate a final output image, wherein the final output image is an approximation of the first image; The function is evaluated based on the difference between the first image and the output image; the parameters of the first, second, and / or third neural networks are updated based on the evaluated function; and the above steps are repeated using the first set of input images to generate the first, second, and / or third trained neural networks.

[0036] According to one aspect of this disclosure, a method for encoding, transmitting, and decoding lossy images or videos is provided. The method includes the steps of receiving a first image and a second image at a first computer system. Furthermore, the method further includes the step of generating optical flow information using the first and second images with a first neural network. The optical flow information indicates a difference between the first and second images. The method also includes the step of extracting information from one or more intermediate layers of the first neural network. Additionally, the method includes the step of generating residual information using the first image and the optical flow information with a second neural network. The residual information indicates a difference between the first and second images. Furthermore, the method includes the step of transmitting the residual information to a second computer system. The method further includes the step of decoding the residual information using a third neural network to generate an intermediate output image. Finally, the method includes the step of modifying the intermediate output image based on information extracted from one or more intermediate layers to generate a modified output image. The modified output image is an approximation of the first image.

[0037] Optionally, the method includes using output extracted from one or more intermediate layers. This extracted output is used to generate a mask.

[0038] Optionally, the method may involve modifying the intermediate output image. As described above, this modification process includes applying a mask to the intermediate output image.

[0039] Alternatively, as described above, the method may involve a mask. The mask comprises at least three distinct values. These values ​​lie between a minimum and a maximum value.

[0040] Optionally, the method includes generating a mask. The mask is generated using one or more layers of a third neural network.

[0041] Optionally, the method includes using a fourth neural network to generate a mask, as described above.

[0042] Optionally, the method may involve extracting information from one or more intermediate layers of a first neural network. The information extracted from these intermediate layers includes one or more outputs. These outputs are derived from one or more convolutional layers and / or activation layers of the first neural network.

[0043] Optionally, the method includes repeating the steps, as described above, for a sequence of video frames. Repeating these steps is used to reconstruct the sequence of video frames at a second computer system.

[0044] Optionally, the method may involve repeating the steps for a sequence of frames in a video. Repeating these steps is used to mitigate image artifacts in the reconstructed sequence of frames.

[0045] Alternatively, a data processing system is provided. The data processing system is configured to perform the methods described above.

[0046] According to one aspect of this disclosure, a method for receiving and decoding lossy images or videos is provided. The method includes the step of decoding optical flow information using a first neural network at a first computer system. The optical flow information indicates a difference between a first image and a second image. The method further includes the step of extracting information from one or more intermediate layers of the first neural network. Furthermore, the method includes the step of decoding residual information using a second neural network at the first computer system to generate an intermediate output image. The residual information indicates a difference between the first image and the second image. Additionally, the method includes the step of modifying the intermediate output image. The modification is based on information extracted from one or more intermediate layers of the first neural network. The modification produces a modified output image. The modified output image is an approximation of the first image.

[0047] Optionally, a data processing apparatus is provided. The data processing apparatus is configured to perform the method described above.

[0048] Optionally, a computer program is provided. The computer program includes instructions. When executed by a computer, the instructions cause the computer to perform the method described above.

[0049] Optionally, a computer-readable storage medium is provided. The computer-readable storage medium includes instructions. When executed by a computer, these instructions cause the computer to perform the method described above.

[0050] According to one aspect, a method for encoding, transmitting, and decoding lossy images or videos is provided, the method comprising the following steps: Receive the first image at the first computer system; The first image is encoded using a first neural network to generate a latent representation; At least partially mask the underlying representation; Transmit the masked potential representation to a second computer system; A second neural network is used to decode the latent representation of the mask to produce an output image, which is an approximation of the first image.

[0051] Optionally, at least partially masking the latent representation reduces the entropy of the distribution of the latent representation.

[0052] Optionally, the masking includes generating a mask based on the first image.

[0053] Optionally, the method includes using a third neural network to generate a mask.

[0054] Optionally, the method includes entropy encoding a masked latent representation into a bit stream, wherein the transmission masked latent representation includes the transmission bit stream, and wherein the step of at least partially masking the latent representation reduces the size of the bit stream after entropy encoding compared to a bit stream generated without the step of at least partially masking.

[0055] Optionally, at least partially masking the latent representation includes at least partially masking the normalized representation of the latent representation.

[0056] Optionally, the method includes using one or more first-learned scaling and / or zero-point parameters to generate a normalized representation of the latent representation.

[0057] Optionally, the method includes encoding the latent representation using a fourth neural network to generate a hyperlatent representation, and decoding the hyperlatent representation using a fifth neural network to generate one or more first-learned scaling and / or zero-point parameters.

[0058] Optionally, the method includes at least partially masking the hyperpotential representation to reduce the entropy of the distribution of the hyperpotential representation.

[0059] Optionally, the method includes entropy encoding the masked hyperlatent representation and transmitting it to a second computer system.

[0060] Optionally, at least partially masking the hyperpotential representation includes at least partially masking the normalized representation of the hyperpotential representation.

[0061] Optionally, the method includes using one or more second-learned scaling and / or zero-point parameters to generate a normalized representation of the hyperlatent representation.

[0062] Optionally, the method includes encoding the hyperlatent representation using a sixth neural network to generate a hyperhyperlatent representation, and decoding the hyperhyperlatent representation using a seventh neural network to generate second-learned scaling and / or zero-point parameters.

[0063] Optionally, the method includes at least partially masking the hyper-hyper-latent representation to reduce the entropy of the distribution of the hyper-hyper-latent representation.

[0064] Optionally, at least partially masking the hyper-hyper-latent representation includes at least partially masking the normalized representation of the hyper-hyper-latent representation.

[0065] Optionally, the method includes entropy encoding the masked hyper-hyper-latent representation and transmitting it to a second computer system.

[0066] According to one aspect, a method for encoding and transmitting lossy images or videos is provided, the method comprising the following steps: Receive the first image at the first computer system; The first image is encoded using a first neural network to generate a latent representation; At least partially mask the underlying representation; The masked potential representation is transmitted to a second computer system.

[0067] According to one aspect, a method for receiving and decoding lossy images or videos is provided, the method comprising the following steps: A masked latent representation is received at a second computer system, the masked latent representation being generated by encoding the first image using a first neural network and by at least partially masking the latent representation; A second neural network is used to decode the latent representation of the mask to produce an output image, which is an approximation of the first image.

[0068] According to one aspect, a data processing apparatus is provided, which is configured to perform any of the methods described above.

[0069] According to one aspect, a computer program is provided that includes instructions that, when executed by a computer, cause the computer to perform any of the methods described above.

[0070] According to one aspect, a computer-readable storage medium is provided, comprising instructions that, when executed by a computer, cause the computer to perform any of the methods described above.

[0071] According to one aspect, a method for encoding, transmitting, and decoding lossy images or videos is provided, the method comprising the following steps: Receive the first image and the second image at the first computer system; Optical flow information is generated using a first neural network with a first image and a second image, the optical flow information indicating the difference between the first image and the second image; A second neural network is used to generate residual information using the first image and optical flow information, the residual information indicating the difference between the first image and the second image; At least partially masking residual information; The masked residual information is transmitted to the second computer system; The masked residual information is decoded using a third neural network to generate an output image, wherein the output image is an approximation of the first image.

[0072] Optionally, at least partially masking the residual information reduces the entropy of the residual information distribution.

[0073] Optionally, the masking includes generating a mask based on a first image and / or a second image.

[0074] Optionally, the method includes using a fourth neural network to generate a mask.

[0075] Optionally, the method includes entropy encoding the masked residual information into a bitstream, wherein transmitting the masked residual information includes transmitting the bitstream, and wherein the step of at least partially masking the residual information reduces the size of the bitstream after entropy encoding compared to the bitstream generated without the step of at least partially masking.

[0076] Optionally, at least partially masking the residual information includes at least partially masking the normalized representation of the residual information.

[0077] Optionally, the method includes using one or more first-learned scaling and / or zero-point parameters to generate a normalized representation of the residual information.

[0078] Optionally, the method includes encoding the residual information using a fifth neural network to generate a hyperlatent representation of the residual information, and decoding the hyperlatent representation of the residual information using a sixth neural network to generate one or more first-learned scaling and / or zero-point parameters.

[0079] Optionally, the method includes at least partially masking the hyperlatent representation of the residual information to reduce the entropy of the distribution of the hyperlatent representation of the residual information.

[0080] Optionally, the method includes entropy encoding of the masked hyperlatent representation of the residual information and transmitting it to a second computer system.

[0081] Optionally, the hyperlatent representation that at least partially masks the residual information includes a normalized representation of the hyperlatent representation that at least partially masks the residual information.

[0082] Optionally, the method includes using one or more second-learned scaling and / or zero-point parameters to generate a normalized representation of the hyperlatent representation of the residual information.

[0083] Optionally, the method includes encoding the hyper-latent representation of the residual information using a seventh neural network to generate a hyper-hyper-latent representation of the residual information, and decoding the hyper-hyper-latent representation of the residual information using an eighth neural network to generate the scaling and / or zero-point parameters of the second learning.

[0084] Optionally, the method includes at least partially masking the hyper-hyper-latent representation of the residual information to reduce the entropy of the distribution of the hyper-hyper-latent representation of the residual information.

[0085] Optionally, the hyper-hyper-latent representation that at least partially masks the residual information includes a normalized representation of the hyper-hyper-latent representation that at least partially masks the residual information.

[0086] Optionally, the method includes entropy encoding of the masked hyper-hyper-latent representation of the residual information and transmitting it to a second computer system.

[0087] According to one aspect, a method for encoding and transmitting lossy images or videos is provided, the method comprising the following steps: Receive the first image and the second image at the first computer system; Optical flow information is generated using a first neural network with a first image and a second image, the optical flow information indicating the difference between the first image and the second image; A second neural network is used to generate residual information using the first image and optical flow information, the residual information indicating the difference between the first image and the second image; At least partially masking residual information; and The masked residual information is transmitted to the second computer system.

[0088] According to one aspect, a method for receiving and decoding lossy images or videos is provided, the method comprising the following steps: Masked residual information is received at a second computer system. This masked residual information is generated by using a first neural network to generate optical flow information from a first image and a second image, and by using a second neural network to generate residual information from the first image and the optical flow information, and by at least partially masking the residual information. The residual information indicates the difference between the first image and the second image. The masked residual information is decoded using a third neural network to generate an output image, wherein the output image is an approximation of the first image.

[0089] According to one aspect, a data processing apparatus is provided, which is configured to perform any of the methods described above.

[0090] According to one aspect, a computer program is provided, including instructions that, when executed by a computer, cause the computer to perform any of the methods described above.

[0091] According to one aspect, a computer-readable storage medium is provided, comprising instructions that, when executed by a computer, cause the computer to perform any of the methods described above.

[0092] According to one aspect, a method for encoding, transmitting, and decoding lossy images or videos is provided, the method comprising the following steps: Receive the first and second images of the image sequence at the first computer system; Optical flow information is generated using a first neural network with a first image and a second image, the optical flow information indicating the difference between the first image and the second image; At least partially masking optical flow information; The representation of the masked optical flow information is transmitted to the second computer system; The second neural network is used to decode the representation of the masked optical flow information to generate an approximation of the optical flow information; and An output image is generated using optical flow information through a third neural network, wherein the output image is an approximation of the first image.

[0093] Optionally, at least partially masking the optical flow information reduces the entropy of the distribution of the optical flow information.

[0094] Optionally, the masking includes generating a mask based on a first image and / or a second image.

[0095] Optionally, the method includes using a fourth neural network to generate a mask.

[0096] Optionally, the method includes entropy encoding the masked optical flow information into a bit stream, wherein transmitting the masked optical flow information includes transmitting the bit stream, and wherein the step of at least partially masking the optical flow information reduces the size of the bit stream after entropy encoding compared to the bit stream generated without the step of at least partially masking.

[0097] Optionally, at least partially masking the optical flow information includes at least partially masking a normalized representation of the optical flow information.

[0098] Optionally, the method includes using one or more first-learned scaling and / or zero-point parameters to generate a normalized representation of the optical flow information.

[0099] Optionally, the method includes encoding the optical flow information using a fifth neural network to generate a hyperlatent representation of the optical flow information, and decoding the hyperlatent representation of the optical flow information using a sixth neural network to generate one or more first-learned scaling and / or zero-point parameters.

[0100] Optionally, the method includes at least partially masking the hyperpotential representation of optical flow information to reduce the entropy of the distribution of the hyperpotential representation of optical flow information.

[0101] Optionally, the method includes entropy encoding of the masked hyperlatent representation of optical flow information and transmitting it to a second computer system.

[0102] Optionally, the hyperlatent representation that at least partially masks the optical flow information includes a normalized representation of the hyperlatent representation that at least partially masks the optical flow information.

[0103] Optionally, the method includes using one or more second-learned scaling and / or zero-point parameters to generate a normalized representation of the hyperlatent representation of optical flow information.

[0104] Optionally, the method includes encoding the hyper-latent representation of optical flow information using a seventh neural network to generate a hyper-hyper-latent representation of optical flow information, and decoding the hyper-hyper-latent representation of optical flow information using an eighth neural network to generate second-learned scaling and / or zero-point parameters.

[0105] Optionally, the method includes at least partially masking the hyper-hyper-latent representation of optical flow information to reduce the entropy of the distribution of the hyper-hyper-latent representation of optical flow information.

[0106] Optionally, the hyper-hyper-latent representation that at least partially masks the optical flow information includes a normalized representation of the hyper-hyper-latent representation that at least partially masks the optical flow information.

[0107] Optionally, the method includes entropy encoding of the masked hyper-hyper-latent representation of optical flow information and transmitting it to a second computer system.

[0108] According to one aspect, a method for encoding and transmitting lossy images or videos is provided, the method comprising the following steps: Receive the first and second images of the image sequence at the first computer system; Optical flow information is generated using a first neural network with a first image and a second image, the optical flow information indicating the difference between the first image and the second image; At least partially masking optical flow information; and The representation of the masked optical flow information is transmitted to the second computer system.

[0109] According to one aspect, a method for receiving and decoding lossy images or videos is provided, the method comprising the following steps: The representation of masked optical flow information received at the second computer system, the masked optical flow information being generated using a first neural network with a first image and a second image, and being generated by at least partially masking the optical flow information; The second neural network is used to decode the representation of the masked optical flow information to generate an approximation of the optical flow information; and An output image is generated using optical flow information through a third neural network, wherein the output image is an approximation of the first image.

[0110] According to one aspect, a data processing apparatus is provided, which is configured to perform any of the methods described above.

[0111] According to one aspect, a computer program is provided that includes instructions that, when executed by a computer, cause the computer to perform any of the methods described above.

[0112] According to one aspect, a computer-readable storage medium is provided, comprising instructions that, when executed by a computer, cause the computer to perform any of the methods described above.

[0113] According to one aspect, a method for encoding, transmitting, and decoding lossy images or videos is provided, the method comprising the following steps: Receive the first and second images of the image sequence at the first computer system; Optical flow information is generated using a first neural network with a first image and a second image, the optical flow information indicating the difference between the first image and the second image; The optical flow information is transmitted to the second computer system; A second neural network is used to decode the representation of optical flow information to generate an approximation of optical flow information. The representation of the previously decoded second image is distorted using an approximation of optical flow information; Combine the distorted representation of the previously decoded second image with the representation of the first image; and An output image is generated using a third neural network with a combined, distorted representation of the previously decoded second image, wherein the output image is an approximation of the first image.

[0114] Optionally, the method includes using a second neural network to generate a representation of the first image.

[0115] Optionally, the method includes using a second neural network to generate a mask and using the mask to combine a representation of the first image with a distorted representation of the previously decoded second image.

[0116] Optionally, the method includes using a fourth neural network to generate a representation of the first image from the first image.

[0117] Optionally, the method includes using a fourth neural network to generate a mask from the first image, and using the mask to combine a representation of the first image with a distorted representation of a previously decoded second image.

[0118] Optionally, the combination includes performing a weighted sum of the representation of the first image and the distorted representation of the previously decoded second image, using weights based on the values ​​of the mask.

[0119] Optionally, the combination reduces the difference between (i) the distorted representation of the previously decoded second image and (ii) the first image.

[0120] Optionally, the combination reduces the difference between (i) the output image generated by using the combined distorted representation of the previously decoded second image with a third neural network and (ii) the first image.

[0121] Optionally, the combination corrects artifacts in the representation of the previously decoded second image that are distorted.

[0122] According to one aspect, a method for encoding and transmitting lossy images or videos is provided, the method comprising the following steps: Receive the first and second images of the image sequence at the first computer system; Optical flow information is generated using a first neural network with a first image and a second image, the optical flow information indicating the difference between the first image and the second image; and The optical flow information is transmitted to the second computer system.

[0123] According to one aspect, a method for receiving and decoding lossy images or videos is provided, the method comprising the following steps: The optical flow information received at the second computer system is generated by the first neural network using a first image and a second image of an image sequence and indicates the difference between the first image and the second image. A second neural network is used to decode the representation of optical flow information to generate an approximation of optical flow information. The representation of the previously decoded second image is distorted using an approximation of optical flow information; Combine the distorted representation of the previously decoded second image with the representation of the first image; and An output image is generated using a third neural network with a combined, distorted representation of the previously decoded second image, wherein the output image is an approximation of the first image.

[0124] According to one aspect, a method is provided for training one or more neural networks for use in lossy image or video encoding, transmission, and decoding, the method comprising the following steps: Receive the first and second images of the image sequence at the first computer system; Optical flow information is generated using a first neural network with a first image and a second image, the optical flow information indicating the difference between the first image and the second image; The optical flow information is transmitted to the second computer system; A second neural network is used to decode the representation of optical flow information to generate an approximation of optical flow information. The representation of the previously decoded second image is distorted using an approximation of optical flow information; Combine the distorted representation of the previously decoded second image with the representation of the first image; An output image is generated using a third neural network with a combined, distorted representation of the previously decoded second image, wherein the output image is an approximation of the first image; The function is evaluated based on the difference between the first image and the output image; The parameters of the first, second, and / or third neural networks are updated based on the evaluated function; and Repeat the above steps using the first set of input images to generate the first, second, and / or third trained neural networks.

[0125] Optionally, the method includes using a second neural network to generate a representation of the first image.

[0126] Optionally, the method includes using a second neural network to generate a mask and using the mask to combine a representation of the first image with a distorted representation of the previously decoded second image.

[0127] Optionally, the method includes using a fourth neural network to generate a representation of the first image from the first image.

[0128] Optionally, the method includes using a fourth neural network to generate a mask from the first image, and using the mask to combine a representation of the first image with a distorted representation of a previously decoded second image.

[0129] Optionally, the function is further based on the difference between (i) the representation of the first image generated by the second neural network or the fourth neural network and (ii) the representation of the first image generated by the teacher neural network.

[0130] Optionally, the function is further based on the difference between (i) the mask generated by the second or fourth neural network and (ii) the mask generated by the teacher neural network.

[0131] Optionally, evaluating the function includes regularizing the propagation of gradients through one or more terms of the function.

[0132] Optionally, the evaluation of the function includes regularizing the propagation of the gradient of the term associated with the difference between (i) the mask generated by the second neural network or the fourth neural network and (ii) the mask generated by the teacher neural network.

[0133] Optionally, the method includes controlling the regularization based on a predetermined regularization schedule.

[0134] According to one aspect, a data processing apparatus is provided, which is configured to perform any of the methods described above.

[0135] According to one aspect, a computer program is provided, including instructions that, when executed by a computer, cause the computer to perform any of the methods described above.

[0136] According to one aspect, a computer-readable storage medium is provided, including instructions that, when executed by a computer, cause the computer to perform any of the methods described above. Attached Figure Description

[0137] Various aspects of the invention will now be described by way of example with reference to the following figures, in which: Figure 1 The illustration shows an example of an image or video compression, transmission, and decompression pipeline.

[0138] Figure 2 The illustration shows a further example of an image or video compression, transmission, and decompression pipeline that includes a hypernet.

[0139] Figure 3The diagram illustrates an example of a video compression, transmission, and decompression pipeline.

[0140] Figure 4 The illustration shows an example of a video compression, transmission, and decompression system.

[0141] Figure 5a The illustration shows an example of the output of an image or video compression, transmission, and decompression pipeline.

[0142] Figure 5b The illustration shows an example of the output of an image or video compression, transmission, and decompression pipeline.

[0143] Figure 6 The diagram illustrates an example of a video compression, transmission, and decompression pipeline.

[0144] Figure 7 The diagram illustrates an example of a video compression, transmission, and decompression pipeline.

[0145] Figure 8 The diagram illustrates an example of a video compression, transmission, and decompression pipeline.

[0146] Figure 9 The illustration shows an example of the encoding-side components of a video compression, transmission, and decompression pipeline.

[0147] Figure 10 The illustration shows an example of the decoding-side components of a video compression, transmission, and decompression pipeline.

[0148] Figure 11 The illustration shows a visualization of the output of one or more networks in a video compression, transmission, and decompression pipeline.

[0149] Figure 12 The diagram illustrates an example of a video compression, transmission, and decompression pipeline.

[0150] Figure 13 The diagram illustrates an example of a video compression, transmission, and decompression pipeline. Detailed Implementation

[0151] Compression can be applied to any form of information to reduce the amount of data or file size required to store that information. Image and video information are examples of information that can be compressed. The file size required to store this information, especially when referring to the compressed file during the compression process, can be called the bitrate. Generally, compression can be lossless or lossy. In both forms of compression, the file size is reduced. However, in lossless compression, no information is lost when the information is compressed and subsequently decompressed. This means that the original file storing the information is completely reconstructed during the decompression process. Conversely, in lossy compression, information may be lost during compression and decompression, and the reconstructed file may differ from the original file. Image and video files containing image and video data are common targets for compression.

[0152] In image compression, the input image can be represented as x. The data representing the image can be stored in an H×W×C dimensional tensor, where H represents the height of the image, W represents the width of the image, and C represents the number of channels of the image. Each H×W data point of the image represents the pixel value at the corresponding location in the image. Each channel C of the image represents the different components of the image that combine each pixel when the image file is displayed by the device. For example, an image file can have 3 channels, where the channels represent the red, green, and blue components of the image, respectively. In this case, the image information is stored in the RGB color space (which can also be called a model or format). Other examples of color spaces or formats include the CMKY and YCbCr color models. However, the channels of an image file are not limited to storing color information, and other information can also be represented in the channels. Since video can be considered as a series of images in a sequence, any compression process that can be applied to images can also be applied to video. Each image that makes up a video can be called a frame of the video.

[0153] The output image can be different from the input image, and can be generated by... The difference between the input and output images can be referred to as distortion or a difference in image quality. Distortion can be measured using any distortion function that receives the input and output images and provides an output that digitally represents the difference between the input and output images. An example of such a method is to use the mean squared error (MSE) between the pixels of the input and output images, but as those skilled in the art will know, there are many other ways to measure distortion. The distortion function can include a trained neural network.

[0154] Typically, bitrate and distortion are related in lossy compression. An increase in bitrate may lead to a decrease in distortion, while a decrease in bitrate may lead to an increase in distortion. Changes in distortion may affect bitrate in a corresponding way. For a given compression technique, the relationship between these quantities can be defined by a rate-distortion equation.

[0155] AI-based compression processes may involve the use of neural networks. A neural network is an operation that can be performed on input to produce an output. A neural network can consist of multiple layers. The first layer of the network receives the input. This layer can perform one or more operations on the input to produce its output. The output of the first layer is then passed to the next layer of the network, which can perform one or more operations in a similar manner. The output of the final layer is the output of the neural network.

[0156] Each layer of a neural network can be divided into nodes. Each node can receive at least a portion of the input from previous layers and provide output to one or more nodes in subsequent layers. Each node in a layer can perform one or more operations of that layer on at least a portion of the input to that layer. For example, a node can receive input from one or more nodes in previous layers. One or more operations can include convolution, weights, biases, and activation functions. Convolution operations are used in convolutional neural networks. When a convolution operation is present, convolution can be performed across the entire input of the layer. Alternatively, convolution can be performed on at least a portion of the input to that layer.

[0157] Each of one or more operations is defined by one or more parameters associated with that operation. For example, a weight operation can be defined by a weight matrix, which defines the weights to be applied to each input from each node in the previous layer to each node in the current layer. In this example, each value in the weight matrix is ​​a parameter of the neural network. A convolution can be defined by a convolution matrix (also called a kernel). In this example, one or more values ​​in the convolution matrix can be parameters of the neural network. An activation function can also be defined by values ​​that can be parameters of the neural network. The parameters of the network can change during the training of the network.

[0158] Other features of a neural network can be predetermined and therefore remain unchanged during training. For example, the number of layers, the number of nodes, one or more operations performed in each layer, and the connections between layers can be predetermined and are therefore fixed before training occurs. These predetermined features can be called the network's hyperparameters. These features are sometimes referred to as the network's architecture.

[0159] To train a neural network, a training set of inputs is used, for which the expected output (sometimes called the ground truth) is known. The initial parameters of the neural network are randomized, and the first training input is provided to the network. The network's output is compared with the expected output, and based on the difference between the output and the expected output, the network's parameters are changed to reduce the difference between the network's output and the expected output. This process is then repeated for multiple training inputs to train the network. The difference between the network's output and the expected output can be defined by a loss function. The result of the loss function can be calculated using the difference between the network's output and the expected output to determine the gradient of the loss function. Backpropagation of the gradient descent of the loss function can be used to update the parameters of the neural network using the gradient dL / dy of the loss function. By backpropagating the gradient of the loss function to each network, multiple neural networks in the system can be trained simultaneously.

[0160] In the context of image or video compression, this type of system, where training is performed simultaneously with backpropagation of each element or the entire network architecture, can be referred to as end-to-end learning image or video compression. Unlike traditional compression algorithms that primarily use hand-designed, manually constructed steps, end-to-end learning systems learn during training which combinations of parameters best achieve the objective of minimizing the loss function. This approach is advantageous compared to non-end-to-end learning systems because end-to-end systems have greater flexibility in learning weights and parameters that might be counterintuitive to someone who designs features by hand.

[0161] It will be understood that, as used in this paper, the terms "training" or "learning" refer to the process of optimizing an artificial intelligence or machine learning model based on a given dataset. This involves iteratively tuning the model's parameters to minimize the difference between the model's predictions, represented by the rate-distortion loss function described above, and the actual data.

[0162] The training process can include multiple epochs. An epoch refers to a complete pass through the entire training dataset by a machine learning algorithm. During an epoch, the model's parameters are updated in an effort to minimize the loss function. It is conceivable that the model can be trained using multiple epochs, the exact number of which depends on various factors, including the model's complexity and the diversity of the training data.

[0163] Within each epoch, the training data can be divided into smaller subsets called batches. The size of these batches, known as the batch size, can affect the training process. Smaller batch sizes can lead to more frequent updates to the model's parameters, potentially resulting in faster convergence to the optimal solution, but at the cost of increased computational resources. Conversely, larger batch sizes involve fewer updates, which may be more computationally efficient but may converge more slowly or even fail to converge to the optimal solution.

[0164] The learnable parameters are updated by a specified amount determined by the learning rate each time. The learning rate is a hyperparameter that determines the extent to which the parameters are tuned during training. A smaller learning rate means smaller steps in the parameter space and potentially more accurate solutions, but it may require more epochs to reach that solution. On the other hand, a larger learning rate can speed up the training process, but there is a risk of overshooting the optimal solution or causing the training process to diverge.

[0165] The training described in this article may involve the use of a validation set, which is a portion of data not used in the initial training. This validation set is used to evaluate the model's performance and prevent overfitting. Overfitting occurs when a model learns too well from the training data, to the point that it cannot generalize to unseen data. Regularization techniques, such as dropout or L1 / L2 regularization, can also be used to mitigate overfitting.

[0166] It will be understood that training a machine learning model is an iterative process that may include the selection and tuning of various parameters and hyperparameters. As will be understood, the specific details of the training process (such as hyperparameters, etc.) can vary, and it is conceivable that models trained in this manner can be implemented in many different ways with different epochs, batch sizes, learning rates, regularization, etc., the details of which are not essential to enabling the advantages and effects of this disclosure, unless otherwise stated. The moment when an “untrained” neural network is considered “trained” is conceivable as situation-specific and depends on, for example, multiple epochs, the plateau period of any further learning, or some other metric, and is not considered essential to achieving the advantages described herein.

[0167] More details about the end-to-end learning compression process will now be described. It will be understood that, in some cases, the end-to-end learning compression process can be combined with one or more components that are separately designed or trained by hand.

[0168] In the case of AI-based image or video compression, the loss function can be defined by the rate-distortion equation. The rate-distortion equation can be expressed as Loss = D + λ * R, where D is the distortion function, λ is a weighting factor, and R is the bitrate loss. λ can be called the Lagrange multiplier. The Lagrange multiplier provides a weight relative to each other term of the loss function for a specific term and can be used to control which terms of the loss function are favored during network training.

[0169] In the case of AI-based image or video compression, a training set of input images can be used. Example training sets for input images are the KODAK image set (e.g., at www.cs.albany.edu / xypan / research / snr / Kodak.html). Example training sets for input images are the IMAX image set. Example training sets for input images are the ImageNet dataset (e.g., at www.image-net.org / download). Example training sets for input images are the CLIC training datasets P (“Professional”) and M (“Mobile”) (e.g., at http: / / challenge.compression.cc / tasks / ).

[0170] Figure 1 An example of an AI-based compression, transmission, and decompression process 100 is shown. As the first step in the AI-based compression process, an input image 5 is provided. The input image 5 is fed to a trained neural network 110, which is powered by a function f acting as an encoder. θ Representation. The encoder neural network 110 generates an output based on the input image. This output is referred to as the latent representation of the input image 5. In the second step, the latent representation is quantized in a quantization process 140, characterized by the operation Q, to produce a quantized latent representation. The quantization process transforms the continuous latent representation into a discrete quantized latent representation. An example of a quantization process is a rounding function.

[0171] In the third step, the quantized latent representation is entropy encoded in the entropy encoding process 150 to produce a bitstream 130. The entropy encoding process can be, for example, range or arithmetic encoding. In the fourth step, the bitstream 130 can be transmitted across a communication network.

[0172] In the fifth step, the bitstream undergoes entropy decoding in entropy decoding process 160. The quantized latent representation is provided to another trained neural network 120, which is trained by a function g acting as a decoder. θ The decoder represents the quantized potential. A trained neural network 120 produces an output based on the quantized potential. This output can be the output image of an AI-based compression process 100. This encoder-decoder system can be referred to as an autoencoder.

[0173] Entropy coding processes, such as range or arithmetic coding, are typically able to losslessly compress given input data until approaching the fundamental entropy limit of that data, as determined by the total entropy of the data's distribution. Therefore, one way in which end-to-end learning compression can minimize the rate-loss term of the rate-distortion loss function and thus improve compression efficiency is to learn the autoencoder parameter values ​​that produce a low-entropy latent representation distribution. Producing a latent representation with the lowest possible entropy distribution allows entropy coding to compress the latent distribution to or near the fundamental entropy limit of that distribution. The lower the entropy of the distribution, the more losslessly the entropy coding can compress it, and the less data is in the corresponding bitstream. In some cases where the latent representation is distributed according to a Gaussian or Laplace distribution, this learning may include learning the optimal position and scaling parameters of the Gaussian or Laplace distribution; in others, it allows learning a more flexible latent representation distribution, which can further contribute to minimizing the rate-distortion loss function in ways that are not intuitive or impossible to achieve with hand-designed features. Examples of these and other advantages are described in WO2021 / 220008A1, which is incorporated herein by reference in its entirety.

[0174] Closely related to the entropy encoding of the latent distribution and therefore also affecting the efficiency of compression in end-to-end learning methods is the quantization step. During inference, a rounding function can be used to quantize the latent representation distribution into bins of a given size; this rounding function is not differentiable everywhere. Instead, the rounding function is actually one or more step functions with gradients of zero (at the top of the step) or infinity (at the boundaries between steps). Backpropagating the gradient of the loss function through the rounding function is challenging. Instead, during training, quantization via the rounding function is replaced by one or more other methods. For example, the function of a noisy quantization model is differentiable everywhere, and therefore does allow the gradient of the loss function to be backpropagated through the quantized portion of the end-to-end learning system. Alternatively, a pass-through estimator (STE) quantization model or another quantization model can be used. It is also envisioned that different quantization models can be used during the evaluation of different terms of the loss function. For example, noisy quantization can be used to evaluate the rate or entropy loss term of the rate-distortion loss function, while STE quantization can be used to evaluate the distortion term.

[0175] End-to-end learning of the quantization process achieves a similar effect, much like how learning parameters to produce latent representations facilitates better minimization of the rate loss term. In other words, learnable quantization parameters provide further degrees of freedom to the architecture to achieve the goal of minimizing the loss function. For example, parameters corresponding to the quantization bin size can be learned, which may lead to improved rate-distortion loss results compared to methods using hand-designed quantization bin sizes.

[0176] Furthermore, since the rate-distortion loss function must constantly balance the rate loss term and the distortion loss term, it has been found that the more degrees of freedom the system has during training, the better the architecture can achieve the optimal bitrate and distortion trade-off.

[0177] The aforementioned system can be distributed across multiple locations and / or devices. For example, encoder 110 can be located on a device such as a laptop computer, desktop computer, smartphone, or server. Decoder 120 can be located on a separate device, which may be referred to as a receiving device. The system for encoding, transmitting, and decoding input image 5 to obtain output image 6 may be referred to as a compression pipeline.

[0178] The AI-based compression process may further include a hypernetwork 105 for transmitting metadata that improves the compression process. The hypernetwork 105 includes components that act as a hyperencoder. The trained neural network 115 and acting as a super decoder The trained neural network 125. Examples of such systems are in Figure 2 The components of the system, which are not discussed further, can be assumed to be the same as those discussed above. The neural network 115, acting as a super encoder, receives a latent representation as the output of the encoder 110. The super encoder 115 produces an output based on the latent representation, which may be referred to as the hyperlatent representation. Then, in the Q... h The quantization process 145 of the representation quantizes the hyperlatent representation to produce a quantized hyperlatent representation. This is achieved through Q... h The quantization process 145 of the representation can be the same as the quantization process 140 of the Q representation discussed above.

[0179] In a similar manner to that discussed above for quantizing the latent representation, the quantized hyperlatent representation is then entropy encoded in entropy encoding process 155 to produce bitstream 135. Bitstream 135 can be entropy decoded in entropy decoding process 165 to retrieve the quantized hyperlatent representation. The quantized hyperlatent representation is then used as input to neural network 125, which serves as training for the superdecoder. However, unlike the compression pipeline 100, the output of the superdecoder may not be an approximation of the input to the superencoder 115. Instead, the output of the superdecoder is used to provide parameters for use in the entropy encoding process 150 and entropy decoding process 160 within the main compression process 100. For example, the output of superdecoder 125 may include one or more of the mean, standard deviation, variance, or any other parameters used to describe the probabilistic model of the entropy encoding process 150 and entropy decoding process 160 of the latent representation. Figure 2In the example shown, for simplicity, only a single entropy decoding process 165 and super decoder 125 are illustrated. However, in practice, since the decompression process typically occurs on a separate device, copies of these processes will exist on the device used for encoding to provide the parameters to be used in the entropy encoding process 150.

[0180] At any stage of the AI-based compression process 100, further transformations can be applied to at least one of the latent and hyperlatent representations. For example, before performing entropy encoding processes 150, 155, at least one of the latent and hyperlatent representations can be converted into residual values. The residual values ​​can be determined by subtracting the mean of the distribution of the latent or hyperlatent representation from each latent or hyperlatent representation. The residual values ​​can also be normalized.

[0181] To train the AI-based compression process described above, a training set of input images can be used as described. During the training process, the parameters of both encoder 110 and decoder 120 can be updated simultaneously in each training step. If a supernetwork 105 is also present, the parameters of both superencoder 115 and superdecoder 125 can also be updated simultaneously in each training step. The training process may further include a generative adversarial network (GAN). When applied to the AI-based compression process, in addition to the compression pipeline described above, the system includes an additional neural network acting as a discriminator. The discriminator receives the input and outputs a score based on that input, which provides an indication of whether the discriminator considers the input to be true or false. For example, the indicator can be a score, where a high score is associated with a true input and a low score is associated with a false input. For training the discriminator, a loss function is used that maximizes the difference in the output indication between true and false inputs.

[0182] When a GAN is incorporated into the training of the compression process, the output image 6 can be provided to the discriminator. The discriminator's output can then be used in the loss function of the compression process as a measure of compression distortion. Alternatively, the discriminator can receive both the input image 5 and the output image 6, and the difference in the output can then be used as a measure of compression distortion in the loss function. The training of the neural network acting as the discriminator and other neural networks during compression can be performed simultaneously. During the compression and transmission of images or videos using the trained compression pipeline, the discriminator neural network is removed from the system, and the output of the compression pipeline is the output image 6.

[0183] Incorporating GANs into the training process may result in the decoder 120 performing illusions. An illusion is the process of adding information to the output image 6 that is not present in the input image 5. In the example, the illusion could add fine details to the output image 6 that are not present in the input image 5 or were not received by the decoder 120. The illusion performed could be based on information in the quantized latent representation received by the decoder 120.

[0184] The details of the video compression process will now be described. As discussed above, a video consists of a series of images arranged in sequence. The AI-based compression process 100 described above can be applied multiple times to perform video compression, transmission, and decompression. For example, each frame of the video can be compressed, transmitted, and decompressed individually. The received frames can then be grouped to obtain the original video.

[0185] Frames in a video can be tagged based on information from other frames, which is used to decode the frame during video compression, transmission, and decompression. As mentioned above, a frame that is not decoded using information from other frames can be called an I-frame. A frame that is decoded using information from past frames can be called a P-frame. A frame that is decoded using information from both past and future frames can be called a B-frame. Frames may not be encoded and / or decoded in the order they appear in the video. For example, a frame at a later time step in the video may be decoded before a frame at an earlier time step.

[0186] The images represented by each frame of a video may be related. For example, multiple frames in a video may show the same scene. In this case, multiple different parts of the scene may be shown in more than one frame. For example, objects or people in the scene may be shown in more than one frame. The background of the scene may also be shown in more than one frame. If an object or viewpoint is in motion in the video, the position of an object or background in one frame may change relative to the position of an object or background in another frame. The transformation of a portion of an image from a first position in a first frame to a second position in a second frame can be referred to as flow, warp, or motion compensation. Flow can be represented by vectors. One or more flows representing the transformation of at least a portion of one frame to another can be called a flow graph.

[0187] Figure 3 Example AI-based video compression, transmission, and decompression process 200 is shown. Figure 3 The process 200 shown is divided into an I-frame portion 201 for decompressing I-frames and a P-frame portion 202 for decompressing P-frames. It will be understood that the division of these different portions is arbitrary, and the process 200 can also be considered as a single end-to-end pipeline.

[0188] As mentioned above, I-frames do not depend on information from other frames, therefore I-frame portion 201 corresponds to Figure 1 The compression, transmission, and decompression process is illustrated in Figure 2. Specific details will not be repeated here, but in general, the input image x0 is passed to the encoder neural network 203 to produce a latent representation, which is quantized and entropy-encoded into a bitstream 204. The subscript 0 in x0 indicates that the input image corresponds to a frame in the video stream at position t = 0. This could be the first frame of the entire video stream, or the first frame of a block of video stream consisting of, for example, an I-frame and multiple subsequent P-frames and / or B-frames. The bitstream 204 is then entropy-decoded and passed to the decoder neural network 205 to reconstruct the image. In this case, reconstruct the image It is an I-frame. The decoding step can be performed locally at the same location where the input image compression occurred, or it can be performed at the location where decompression occurred. This allows for the reconstruction of the image. It can be used subsequently by components on both the encoding and decoding sides of the pipeline.

[0189] Compared to I-frames, P-frames (and B-frames) do indeed rely on information from other frames. Therefore, the P-frame portion 202 at the encoding side of the pipeline not only compresses the input image x... t (The frame corresponding to the video stream at position t) is taken as input, and one or more previously reconstructed images from an earlier frame t-1 are also taken as input. As input. As mentioned above, the previously reconstructed It is available on both the encoding and decoding sides of the pipeline, and therefore can be used for various purposes on both the encoding and decoding sides.

[0190] On the encoding side, the previously reconstructed image can be used to generate a flow graph containing information indicating inter-frame movement of pixels between frames. Figure 3 In the example, the compressed image x t and previously reconstructed images from earlier frames Both are passed to the flow module section 206 of the pipeline. The flow module section 206 includes, for example,... Figure 1 and 2 An autoencoder system, but in which the encoder neural network 207 has been trained to learn from the input and x t The potential representation that generates the flow graph, which indicates and x t Inter-frame movement of pixels or groups of pixels. The latent representation of the flow graph is quantized and entropy-encoded to compress it, and then transmitted as a bitstream 208. On the decoding side, the bitstream is entropy-decoded and passed to the decoder neural network 209 to produce the reconstructed flow graph f.

[0191] Apply the reconstructed flow graph f to the previously reconstructed image. To generate distorted images It is conceivable that any suitable warping technique could be used, such as bilinear or trilinear warping as described in “Scale-space flow for end-to-end optimized video compression” by Agustsson, E., Minnen, D., Johnston, N., Balle, J., Hwang, SJ and Toderici, G. (2020) in the proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 8503-8512), which is incorporated herein by reference. Further, it is conceivable that scaling spatial flow methods, as described in the aforementioned papers, could also be optionally used. Warped image Based on the 206 automatic encoder system from the stream module section, from x t and The output flow graph generated from the input is used to reconstruct the image from the previous image. Predictions of how the frame position might change between frame position t-1 and t.

[0192] Similar to I-frames, a reconstructed flow graph f and a corresponding warped image can be generated on both the encoding and decoding sides of the pipeline. Therefore, they can be used by other components of the pipeline on both the encoding and decoding sides.

[0193] exist Figure 3 In the example, the compressed image x t and Both are passed to the residual module section 210 of the pipeline. The residual module section 210 includes an automatic encoder system, such as... Figure 1 and 2 An autoencoder system, but in which the encoder neural network 211 has been trained to generate a latent representation of the residual map, which indicates the input image x. t and distorted images The difference between them. The latent representation of the residual graph is then quantized and entropy-encoded into a bitstream 212 and transmitted. The bitstream 212 is then entropy-decoded and passed to a decoder neural network 213, which reconstructs the residual graph r from the decoded latent representation.

[0194] Alternatively, it can be done first at x t and The residual map is pre-computed, and the pre-computed residual map can be passed to the autoencoder for compression only. This hand-designed residual map approach is computationally simpler, but reduces the architecture's degrees of freedom to learn weights and parameters to achieve its objective during training to minimize the rate-distortion loss function.

[0195] Finally, on the decoding side, the residual map r is applied (e.g., by combining addition, subtraction, or different operations) to the warped image to produce a reconstructed image. The reconstructed image It is image x t The reconstructed image corresponds to frame P at position t in the sequence of frames of the video stream. It will be understood that the reconstructed image... Then it can be used to process the next frame. That is, it can be used for compression, transmission, and decompression of x. t+1 Wait until the entire video stream or a block of the video stream has been processed.

[0196] Therefore, for a block of video frames comprising an I-frame and n subsequent P-frames, the bitstream may contain (i) a quantized, entropy-coded latent representation of the I-frame image, and (ii) a quantized, entropy-coded latent representation of the flow graph and residual graph of each P-frame image. For completeness, although in Figure 3 There is no illustration in the image, but Figure 3 Any autoencoder system in the autoencoder system can include hypernetworks and ultra-hypernetworks, such as those combining... Figure 2 The networks described. Therefore, the bitstream may also contain the hyperparameters and hyperhyperparameters of those networks, their potential quantizations, their potential entropy-coded representations, and so on (if applicable).

[0197] Finally, the above methods can generally be extended to B-frames, as described, for example, in “Extending neural p-frame codecs for b-frame coding” by Pourreza, R. and Cohen, T. (2021) in the proceedings of the IEEE / CVF International Conference on Computer Vision (pp. 6680-6689).

[0198] The aforementioned stream- and residual-based methods are highly effective in reducing the amount of data that needs to be transmitted, because only one reconstructed frame (e.g., an I-frame) is required. If available, the encoding side only needs to compress and transmit the flow graph and residual graph (and any hyperparameter information or hyper-hyperparameter information (if applicable)) to reconstruct subsequent frames.

[0199] Figure 4This illustrates an AI image or video compression process implemented in a video streaming system 400 (such as the one described above). Figure 1-3 Example of a described process. System 400 includes a first device 401 and a second device 402. The first and second devices 401 and 402 can be user devices, such as smartphones, tablet computers, AR / VR headsets, or other portable devices. Compared to known systems that primarily perform inference on GPUs such as Nvidia A100, GeForce 3090, and GeForce 4090 GPU cards, Figure 4 System 400 performs inference on the CPUs of the first and second devices, respectively. That is, the computations for both encoding and decoding are performed by the respective CPUs of the first and second devices 401 and 402. This introduces very different power usage, memory, and runtime constraints compared to implementing AI-based compression methods on GPUs. In one example, the CPUs of the first and second devices 401 and 402 may include Qualcomm Snapdragon CPUs.

[0200] The first device 401 includes a media capture device 403, such as a camera, arranged to capture multiple images of a scene 404, hereinafter referred to as video stream 404. Video stream 404 is passed to a preprocessing module 406, which splits the video stream into blocks of frames, where each frame is designated as an I-frame, P-frame, and / or B-frame. The blocks of frames are then compressed by an AI compression module 407, which includes... Figure 3 The encoding side of the AI-based video compression pipeline. Therefore, the output of the AI ​​compression module is a bitstream 408a transmitted from the first device 401, for example via a communication channel, such as one or more of WiFi, 3G, 4G or 5G channels, which may include the Internet or cloud-based communication 409.

[0201] The second device 402 receives the transmitted bitstream 408b, which is then passed to the AI ​​decompression module 410, which includes... Figure 3 The decoding side of the AI-based video compression pipeline. The output of the AI ​​decompression module 402 is reconstructed I-frames, P-frames, and / or B-frames, which are passed to the post-processing module 411, where they can be prepared, for example, by being passed to a buffer, in preparation for streaming 412 to the display device 413 of the second device 402 and rendered on the display device 413.

[0202] Imagine Figure 4The system 400 can be used for 30fps live video streaming of 1080p video streams, meaning that the cumulative latency on both the encoding and decoding sides is significantly less than 50ms, for example, significantly less than 30ms or less. Achieving this level of runtime performance on a user device using only CPU computation presents a challenge that has not been addressed by known methods and systems or in the broader AI compression literature.

[0203] For example, the execution of different parts of a compressed pipeline during inference can be optimized by adjusting the order in which operations are performed using one or more known CPU scheduling methods. Efficient scheduling allows operations to be executed in parallel, thereby reducing overall execution time. It is also envisioned that efficient management of memory resources can be achieved, including optimizing caching methods such as storing frequently accessed data in faster memory locations, and memory reuse, which minimizes memory allocation and deallocation operations.

[0204] The following describes several concepts related to the AI ​​compression processes discussed above and / or their implementation in hardware systems. Although each concept is described separately, one or more of the concepts described below can be applied to the AI-based compression processes described above.

[0205] Concept 1: Stream-Residual Soft Mask In AI-based streaming-residual compression, several types of compression artifacts occur that are not present in traditional compression pipelines. One such artifact appears in image sequences with almost no motion between frames and manifests itself in the reconstructed image as halos around edges and objects in the image. These artifacts typically arise when tiny, almost imperceptible changes in pixel values ​​between frames (such as minute lighting variations, block artifacts from traditional compression if the AI ​​compression pipeline's input comes from a traditional compression source), noise, etc., are amplified over time by the AI-based compression pipeline. In some architectures, this can cause parts of the compression pipeline network to treat image noise as real movement and then exaggerate these as the sequence of video frames progresses. For example, in some architectures, these artifacts might be caused by the operation of one or more streaming modules; in others, by the operation of one or more residual modules; again, in still others, by two modules or actually other parts of the architecture, and so on. At the end of the frame sequence, the artifacts can be quite severe and detrimental to the quality of image reconstruction. This type of artifact is referred to below as "unrolling".

[0206] Unfolding artifacts occur across almost all types of image sequences, but are particularly noticeable in AI-based streaming-residual compression models used for encoding and decoding image sequences in internet video calls and AR / VR scenes (where a nearly stationary foreground user is surrounded by a largely static background that also moves very little), screen-sharing image sequences, security camera feeds, and others. If there is no movement for a period of time, objects and edges begin to exhibit halos around their edges. These artifacts also cause compression pipelines to allocate a disproportionate number of bits to handle the illusory movement caused by unfolding artifacts, even though the corresponding portions of the actual image do not change or change very little between frames, and therefore should not be allocated many bits.

[0207] Concept 1 aims to address this type of unfolding artifact in static or low-motion image sequences.

[0208] Figure 5a and 5b Comparative examples 500a and 500b are shown of the output images of an AI-based compression pipeline with a flow-residual component. In the first example 500a, an unfolding artifact 501 can be seen on the image over a static background object. That is, the edges of the object appear as halo-like rings. In contrast, in the second example 500b, generated using the solution described below, the unfolding artifact is mitigated.

[0209] More specifically, it has been found that the outputs of intermediate layers from the stream decoder neural network (e.g., the outputs of convolutional and / or activation layers) contain high-level feature information indicating motion or non-motion between frames. This extracted information contains a learning signal strong enough to facilitate the introduction of a learned, pixel-by-pixel partial frame skipping mechanism in an AI-based compression pipeline in a computationally efficient manner.

[0210] These outputs are extracted from intermediate layers of the stream decoder and fed into a residual encoder network and / or residual decoder network (e.g., as additional inputs to the encoder and / or decoder, and / or any intermediate layers thereof), and convolution and / or activation operations are introduced to transform the extracted information into an output tensor, allowing the network of this disclosure to output a final output image that is partly based on the output of the residual decoder and partly based on the previously decoded image. The exact quantity of each is determined by the value of the tensor, which in turn is determined by the operations performed on the extracted information by the convolution and activation operations.

[0211] When applying the above method at the general level, consider... Figure 6 The illustrative example shown illustrates an AI-based compression pipeline streaming encoder / decoder network and a residual encoder / decoder network. For example, this could be... Figure 3 The type of AI-based compression pipeline shown is illustrated.

[0212] The stream encoder neural network acquires the current image x t And previous image x t-1 These are encoded into a stream latent representation, which is optionally quantized and entropy-encoded and transmitted as a bitstream. On the decoding side, the bitstream is received, its entropy is decoded into a stream latent representation, and the stream decoder neural network uses this as input to produce a representation of the stream. For example, a flow graph. (The flow graph is then used in this context.) Applied to previously decoded images To generate a distorted version of the previously decoded image Then, the distorted version of the previously decoded image... Together with the current image x t The data is fed into the residual encoder network to produce a residual latent representation, which is optionally quantized and entropy-encoded and transmitted as a bitstream. On the decoding side, the bitstream is received and its entropy is decoded back into the residual latent representation r, which is then combined by the residual decoder neural network with a distorted version from the previously decoded image. The information used to generate reconstructed images .exist Figure 6 In this case, before feeding it together with the residual latent representation of the entropy decoding into the residual decoder, it is compared with the distorted previously decoded image. The associated information may optionally be first processed by module T (hereinafter referred to as the synthesis adapter) to, for example, downsample and / or pad it to produce the final reconstructed image. Then we can work on x. t+1 This process is repeated to encode, transmit, and decode the sequence of frames.

[0213] Figure 7 The diagram illustrates how modifications can be made according to this disclosure. Figure 6 The architecture is designed to feed the output of one or more intermediate layers of the streaming decoder into the residual encoder and / or decoder. For completeness, the corresponding architecture is shown again. Figure 6 The components. The stream encoder neural network acquires the current image x. t And previous image x t-1 These are encoded into a stream latent representation, which is optionally quantized and entropy-encoded and transmitted as a bitstream. On the decoding side, the bitstream is received, its entropy is decoded into a stream latent representation, and the stream decoder neural network uses this as input to produce a representation of the stream. For example, a flow graph. (The flow graph is then used in this context.) Applied to previously decoded images To generate a distorted version of the previously decoded image Then, the distorted version of the previously decoded image... Together with the current image x tThe data is fed into the residual encoder network to produce a residual latent representation r, which is optionally quantized and entropy-encoded and transmitted as a bitstream. On the decoding side, the bitstream is received and its entropy is decoded back to the residual latent representation r, which is then combined by the residual decoder neural network with a distorted version from the previously decoded image. The information used to generate reconstructed images Before feeding it, along with the residual latent representation from entropy decoding, into the residual decoder, it is compared with the distorted previously decoded image. The associated information may optionally be processed first by the synthesis adapter T to produce the final reconstructed image. Then we can work on x. t+1 This process is repeated to encode, transmit, and decode the sequence of frames.

[0214] However, with Figure 6 Unlike in the above, information (e.g., output and / or activation) 700 of one or more intermediate layers 701 of the stream decoder 209 is extracted and fed to one or both of the residual encoder 211 and / or decoder 212.

[0215] In the case of residual encoder 211, for example, this can be combined with and x t As an input to the residual encoder 211. Alternatively, this can be used as an input to one or more intermediate layers (not shown) of the residual encoder 211.

[0216] In the case of residual decoder 212, for example, this can be together with r and / or with, for example, the output of synthesis adapter T. The representation is used as input to the residual decoder 212. Alternatively, this can be used as input to one or more intermediate layers (not shown) of the residual decoder 212.

[0217] In both cases, preprocessing can be applied, for example, upsampling or downsampling, and / or padding, to output 700 to ensure that the dimension matches what is expected by the residual encoder 211 and / or decoder 212.

[0218] Additionally, one or more convolutional and / or activation layers of the residual decoder (or in fact, a separate network) are applied to information extracted from the intermediate layers of the stream decoder, along with residual information, which together facilitate the generation of a new tensor m, referred to in this paper as a mask.

[0219] therefore, Figure 7 The result of this architectural change is that the residual decoder 212 is now able to output not only images The reconstruction can also output a mask m (e.g., a tensor of values) that is applied to the previously decoded image. The pixels and (conversely) the output applied to the residual decoder 212. When the number of pixels is equal to the number of pixels, a composite final output image is produced, consisting of a mixture of the two. The exact composition of this mixture is determined by the exact value of the tensor m. At a general level, the mask m can be understood as the way a learned pixel-by-pixel "frame" skipping method is applied to a given pair of frames. That is, if there is a high degree of motion between the frames, the final output image mixture will likely contain a larger contribution of pixel information from the output of the residual decoder. In contrast, if there is a low degree of motion, the final output image mixture will likely contain a larger contribution of pixel information from the previously decoded frames (indicating that no movement has occurred).

[0220] For a given input image and a previously decoded image, the exact value of a given output mask tensor m produced by the residual decoder depends on the learned weights and / or biases of the residual decoder.

[0221] The mask m is referred to below as a "soft mask". The term "soft" is used to indicate that the mask is not a simple tensor of binary 1s and 0s (which would indicate which values ​​to use or not use in subsequent steps by completely masking those values), but rather that the mask is "soft" because its values ​​can be any value between a minimum and a maximum (e.g., between and including 0 and 1, but not only 0 and 1), and it has a set of values ​​determined by the fixed-point or floating-point precision mechanism (e.g., fp16, int8, etc.) that the residual decoder is operating on. This means that a "soft" or "non-binary" mask only partially masks the values ​​to which the mask is applied, thus allowing it to emphasize or weaken those values ​​based on whatever training objective the mask was trained on.

[0222] In this context, the final output image is determined by partially masking, to varying degrees, how much of it is based on the previously decoded image. (Low motion) allows most pixels to be "copied" from previously decoded images and "skipped" accordingly, and how much is based on the current frame. (High motion) Thus, most pixels are generated by the residual decoder, and the soft mask emphasizes or weakens the movement between two images in the final output image to varying degrees, and captures more of the current frame in which the movement has already occurred.

[0223] This pixel-by-pixel frame skipping method, learned at a general level, allows AI-based compression networks to learn how to correctly process low-motion frame sequences without causing unrolling artifacts.

[0224] For completeness, it should also be noted that in extreme cases, the soft mask m may end up looking like a binary mask of all 1s (indicating no movement and identity operation on the previously decoded image, i.e.) ) or all 0 (indicating a complete shift, thus the final output image corresponds to the output of the residual decoder, i.e. ), and corresponding to such as Figure 6 The output of the architecture in the architecture.

[0225] Now considering further details, the final reconstructed image can be achieved by combining the masked, previously decoded image... and the output of the inverse masked residual decoder Add them together to produce 702: As described above, this new final reconstructed image At the general level, this can be understood as... and The weighted mixture is calculated, where the weights are based on a soft mask *m* and indicate the extent to which the streaming residual network emphasizes high motion over low motion between two images. If, for a given pair of input images, the network has learned that a high degree of motion contributes to high reconstruction quality, the soft mask *m* tensor values ​​will prioritize the previously decoded image. For the final output image The contribution value ends there. Conversely, for image pairs containing low motion, the soft mask m-tensor value will be given priority. For the final output image The contribution value ends here.

[0226] At an even higher level of generalization, this method can be understood as the network learning which pixels of an image to “skip” and how many to “skip”.

[0227] use Figure 7 The practical consequence of the architecture shown is that it almost completely reduces unfolding artifacts (i.e., reduces intra-frame movement that unintentionally amplifies noise and minute pixel variations into an illusion).

[0228] Furthermore, this synergistically improves the compression ratio and reconstruction quality of the compression pipeline, because the absence of illusory unrolling artifacts allows the network to better allocate only the minimum bits to the transmission of pixels in images where there is little or no movement. Instead, the network now correctly allocates more bits to pixels where there is actually high movement between frames, or to areas in the image where a large amount of spatial detail needs to be preserved. As mentioned above, in the absence of solutions to unrolling artifacts, AI-based compression networks tend to allocate a disproportionate number of bits to illusory intra-frame movement.

[0229] Furthermore, collaboratively, since this method uses the output of an intermediate layer of the stream decoder that is lower in dimension than the final output stream graph or warped image, the total computational resource overhead and runtime slowdown caused by acquiring this intermediate stream decoder data and feeding it into the residual encoder / decoder are less than those that would otherwise result from computing the soft mask from the higher-dimensional final output stream graph or warped image. In other words, it has been found that the lower-resolution, high-level abstraction of the stream information found in the output of the intermediate layer of the stream decoder contains both moving and non-moving pixel information, which improves the residual decoder's ability to distinguish between moving and non-moving pixel information, thereby facilitating this in a computationally efficient manner without significantly slowing down the runtime of the compression pipeline.

[0230] although Figure 7 Not shown, but envisioned to enhance the ability of the residual decoder output soft mask m, the residual decoder 212 may include one or more convolutional layers and / or activation layers that receive the output 700 along with... As input, but not dependent on any other streaming information (such as...) In other words, in some implementations, the residual decoder 212 can be said to have two parallel threads: one for computing the soft mask m, which takes the output 700 of the intermediate layer of the stream decoder (also referred to as stream-side information in this paper) and the residual information r as input, and the other thread for computing... It takes a distorted previously decoded image. The outputs of these two parallel threads are then combined as described above to produce the final reconstructed image, i.e., r. This method allows the computation of soft masks to be performed independently (e.g., in separate modules and separate neural networks) and in parallel with the residual decoder decoding residual information into intermediate output images, thereby allowing the soft mask computation to leverage the parallelization capabilities of the hardware on which it runs to facilitate acceleration.

[0231] Alternatively, in other implementations, soft mask estimation can be interleaved with the operations of the residual decoder (e.g., one or more outputs of one or more convolutional or activation layers of the residual decoder can be fed into one or more convolutional or activation layers for computing the soft mask, or vice versa, where the residual decoder can use one or more intermediate outputs of the layer estimating the soft mask). In this case, the soft mask neural network and the residual decoder can be viewed as integrated with each other as... A single neural network with mutual outputs from m.

[0232] In a specific, non-limiting implementation example, the residual decoder 212 may accordingly include one or more convolutional layers and one or more activation layers that process the input stream-side information 700 and the residual information r to produce a soft mask m. For example, the residual decoder 212 may include a first convolutional layer and a second convolutional layer, each having a predetermined number of input channels, a kernel size and stride, and an activation layer combining these, for example, using a floating-point function add-ReLU operation. Then, before applying a final convolutional layer to the output of the ReLU layer, another convolutional layer may be applied to the output of the add-ReLU operation, the output of which is then fed to another activation function, such as a ReLU layer. Then, optionally, a pixel shuffling operation may be performed on the output of the final convolutional layer before applying the final hardtanh activation layer to produce the final soft mask output. It will be understood that the input channel dimensions, kernel size, and stride will be determined by the type and dimensions of the input image forming part of the frame sequence being encoded and decoded. It will also be understood that this is only an illustrative implementation, and other example implementations are envisioned. It will also be understood that the weights and biases of these layers are learnable, and it is envisioned that these can be trained end-to-end using the rest of the compressed pipeline described here.

[0233] It is also envisioned that the intermediate information from the stream decoder is only one type of information containing a sufficiently strong signal to facilitate the learning of a network that outputs a soft mask. Therefore, although the above methods have been described in the context of using stream-side information, it is envisioned that other information could be additionally or alternatively used; for example, the latent representation of the stream before it is input into the stream decoder could be used to train the network to generate a soft mask. Additionally or alternatively, information obtained separately, such as information indicating the differences between the first and second images (i.e., direct comparison of patches or partial patches), such as information identified by methods including comparison of bounding boxes or regions of interest, could be used.

[0234] Finally, it is envisioned that the final output image does not need to be obtained simply by combining the masked previously decoded image and the output of the masked residual decoder. Instead, it is envisioned that the residual decoder output can be masked to produce a modified output image, and then used in some further, other way to obtain the final output image.

[0235] Concept 2: Encoder Mask Another problem in AI-based video and image compression is that for the neural network on the encoder side of an AI-based compression pipeline, learning how best to allocate bits to the information transmitted in the bitstream is challenging when it doesn't know how the information will be processed on the decoder side. A non-limiting example of this challenge can be illustrated by considering the effect of using the soft masking described in Concept 1 above. That is, if the encoder allocates many bits to certain pixels of the input image that will be soft-masked on the decoder side anyway, then it's a waste of bits that could be better used elsewhere or might not be needed at all. Therefore, this challenge presents an opportunity to further reduce the bitstream size and thus improve compression performance (e.g., in bits per pixel or in any other metric indicating compression performance).

[0236] The inventors have recognized that introducing a learned mask on the encoding side, modifying some or all of the data before encoding the data entropy into a bitstream, can result in a significant reduction in bitstream size without correspondingly reducing reconstruction accuracy.

[0237] Consider a pipeline example where soft masking (or virtually any masking or mechanism where information is discarded on the decoding side) has led to the reconstruction of the image. The information of all or most of the pixels that cause the reconstructed image comes from the previously decoded image. Instead of the output from the residual decoder (For example, because x) t-1 and x t There is no motion or low motion between them, and the network has learned to use soft masks to utilize more of the previously decoded image instead of the reconstructed current image, resulting in better rate and / or distortion. In this case, transmission with x in the bitstream t Most of the bits used for the associated information are wasted because they are either not used or only partially used on the decoding side.

[0238] The inventors examined the reconstruction of the current frame before applying the soft mask. It was discovered that the soft-masked area (i.e., the area previously decoded from frames) was... Regions that contribute more to the output frame have become more blurred and inaccurate in the pipelined network. That is, the introduction of a soft-masking architecture unexpectedly causes the network to implicitly learn to reduce the bitstream size without affecting reconstruction accuracy by allocating fewer bits to regions unused after the application of the soft-masking architecture components. Very generally, the network is learning to become implicitly aware of soft masks. However, this effect is very subtle and slight. Concept 2 aims to leverage this effect by introducing an explicit architectural mechanism that the network can directly use to mask information before generating and sending the bitstream. This applies not only to soft masks on the decoding side as described above, but also to any mechanism that discards information on the decoding side.

[0239] In other words, if the encoding side has an architectural mechanism to know when the decoding side might not use some or all of the transmitted information, and knows which information might not be used, then that information can be omitted from the bitstream. The resulting bitstream is therefore smaller, and the impact on the decoding side in terms of reconstruction accuracy will be minimal, because the omitted information will not be used anyway.

[0240] Concept 2, as described here, introduces such a mechanism in the form of encoder-side learning masks, which completely or partially mask part or all of the information to be transmitted before transmission. Very generally, it provides an architectural mechanism for encoder-side networks to explicitly reduce the entropy of the distributions of latent representations, hyperlatent representations, and / or hyperhyperlatent representations, thereby allowing these to be entropily encoded into smaller bitstream sizes.

[0241] For example, the encoder-side learning mask may include a tensor that, before the latent representation, hyperlatent representation, or hyperhyperlatent representation is entropy-encoded into a bitstream, is applied to one or all of the latent representation, hyperlatent representation, or hyperhyperlatent representation generated by the encoder neural network of the I-frame portion 201 and / or P-frame portion 202 (e.g., as...). Figure 3 , 6 (And / or as shown in 7). The values ​​of the tensors can be generated by a dedicated encoder mask neural network separate from the neural networks of the corresponding I-frame portion 201 and / or P-frame portion 202, or they can be generated by existing neural networks of the corresponding I-frame portion 201 and / or P-frame portion 202 to not only output the residual decoder described in Concept 1. Furthermore, it outputs the soft mask m directly in the same way. In both cases, imagine that the neural network(s) that generate the encoder-side learned mask is trained together with other neural networks in an AI-based compressed pipeline to allow the network to learn how best to use the architectural mechanisms to generate the encoder-side learned mask, which best minimizes the loss of whatever loss function (e.g., rate-distortion loss function) is used during training.

[0242] The pseudocode illustrating the exemplary training loop is shown below. Generally speaking, neural networks Includes one or more layers (or defined as) An independent neural network (for output mask m) and an accompanying configuration specifying the location where the encoder mask m is to be applied are initialized. This configuration may, for example, specify that the mask m is applied to the latent representation tensor, hyperlatent representation tensor, and / or hyperhyperlatent representation tensor outputs of one or more of the encoders, superencoders, and / or hyperhyperlatent encoders of the pipeline's I-frame section 201 and / or P-frame section 202 at some point before the latent representation tensor, hyperlatent representation tensor, and / or hyperhyperlatent representation tensor are encoded into a bitstream, such as after the normalization step, as will be described in more detail below. Neural networks f and g are initialized, for example, using random values, and the grad flag is set to true for both the weights and / or biases and the noise of neural networks f and g, indicating that they are trainable. Then, training for n steps is initiated by repeatedly performing forward passes through f and g by adding the mask m to the output specified in the configuration. For example, the loss is calculated according to a rate-distortion loss function (e.g., as described above in the introduction section), and backpropagation is performed through the elements where grad is set to true. An optimizer O, such as SGD, Adam, or some other optimization algorithm, is applied to update the weights and / or biases of f and g. Training terminates after some predetermined conditions are met, such as after a predetermined number of epochs, after training steps, after a loss plateau, or after convergence in some other way. It is envisioned that this training loop can be performed in a high-precision space such as fp32. After training, the implementation of f and / or g can be converted to perform inference in a lower-precision space such as int8, whereby the weights and biases can be converted to int8 values, and the operations used during inference (such as matmuls, etc.) can be specified to operate fully or partially in the applicable lower-precision space. During inference using the training implementation of f and g, it is envisioned that a mask m is applied to the same encoder output as in training.

[0243] refer to Figure 8 An exemplary implementation of this concept is illustrated. Figure 8 The diagram illustrates an AI-based compression pipeline streaming encoder / decoder network and a residual encoder / decoder network, including I-frame portion 201 and P-frame portion 202. For example, this could be similar to... Figure 3 The type of AI-based compression pipeline shown is illustrated.

[0244] The stream encoder neural network acquires the current image x t And previous image x t-1These are encoded into a streaming latent representation, which is optionally quantized and entropy-encoded and transmitted as a bitstream. On the decoding side, the bitstream is received, its entropy is decoded into a streaming latent representation, and the streaming decoder neural network uses this streaming latent representation as input to produce a representation of the stream. For example, a flow graph. (The flow graph is then used in this context.) Applied to previously decoded images To generate a distorted version of the previously decoded image Then, the distorted version of the previously decoded image... Together with the current image x t The data is fed into the residual encoder network to produce a residual latent representation, which is optionally quantized and entropy-encoded and transmitted as a bitstream. On the decoding side, the bitstream is received and its entropy is decoded back into the residual latent representation r, which is then combined by the residual decoder neural network with a distorted version from the previously decoded image. The information used to generate reconstructed images .exist Figure 8 In the case of distorted previously decoded images The associated information may optionally be processed first by module T, referred to below as the synthesis adapter, for example, by downsampling and / or padding it before feeding it together with the residual latent representation of the entropy decoder into the residual decoder to produce the final reconstructed image. Then we can work on x. t+1 This process is repeated to encode, transmit, and decode the sequence of frames.

[0245] like Figure 8 As further shown, the entropy encoding that generates the corresponding bitstreams 204, 208, and 212 is supported by the corresponding supernetwork, which in turn is supported by a super-supernetwork. The supernetwork of the I-frame module 201 includes a neural network 801 (called a super encoder) and a neural network 802 (called a super decoder). (See diagram for further details.) Figure 2The super encoder neural network 801 encodes the latent representation generated by the encoder neural network 203, and the super decoder neural network 802 generates entropy parameters that define the distribution of the latent representation. These parameters are used to generate the bitstream 204 during entropy encoding. The encoded latent representation generated by the super encoder 801 is itself entropily encoded into a bitstream using an entropy model that uses entropy parameters obtained by encoding the encoded latent representation using the super encoder 803 and decoding the encoded latent representation using the super decoder 804. This nested structure can be extended as needed, but super networks and super-super networks have been found to achieve good performance for runtime trade-offs. The entropy parameters used to generate the bitstream associated with the super network (or whatever the highest level of the nested structure is) can optionally be fixed, parameterized, or generated by, for example, the encoder / decoder neural network pair in the super network.

[0246] As in Figure 8 As can be seen, this nested hypernetwork and super-hypernetwork method is also provided in the optical flow module 206 in the form of a super encoder neural network 805, a super decoder neural network 806, a super-hyper encoder neural network 807, and a super-hyper decoder neural network 808. The same nested hypernetwork and super-hypernetwork method is also provided in the residual module 210 in the form of a super encoder neural network 809, a super decoder neural network 810, a super-hyper encoder neural network 811, and a super-hyper decoder neural network 812.

[0247] Figure 9 and Figure 10 A more detailed illustration is provided, for example... Figure 3 , 6 Exemplary implementations of supernetworks and super-supernetwork architectures of 7 and 8. Figure 9 The illustration shows the architecture of super encoder 901 and super super encoder 902 that support potential representation encoder 903, such as I-frame portion 201 or P-frame portion 202. Figure 10 The diagram illustrates the architecture of superdecoder 901 and super-superdecoder 902 that support potential representation decoder 903, such as I-frame portion 201 or P-frame portion 202.

[0248] by Figure 9 To begin, input image x t (Or, in the case of a streaming and / or residual module of a P-frame module, one or more input images x from different times of an image sequence) t ,...,x t−nThe latent representation (and / or its various representations) is fed into a first neural network 904, which outputs the latent representation of the input. The latent representation generated by the first neural network 904 (encoder neural network) is then fed into a second neural network 905 (superencoder neural network), which outputs a super-latent representation (i.e., a latent representation of a latent representation). The super-latent representation is then decoded by a third neural network 906 (superdecoder neural network) to produce entropy parameters 907 indicating the distribution of the latent representation. These entropy parameters 907 represent tensors of position and scaling parameters σ, μ, and δ, which define, for a given set of pixel values, what their zeros or mean are, what their scaling is, and, for example, the variance of the distribution. The entropy parameter 907 allows the distribution to be normalized 908 (in the case of latent representation) by making the distribution centered at a zero and scaling it around that zero, to produce a tensor with a value of minimum entropy, which can then be losslessly encoded very efficiently by applying subsequent entropy encoding operations 909 to the normalized distribution, such as using rANS or any other entropy encoding operation to produce a first bitstream portion 910 that includes latent representation information.

[0249] Although not shown, the superdecoder neural network 906 can also generate one or more quantization parameters (e.g., bin size and bin width) that allow the latent representation to be quantized in a way that minimizes information loss more effectively than if the learned quantization parameters were not used, prior to performing normalization 908 and / or entropy encoding 909. In other words, using the learned quantization parameters generated in the supernetwork facilitates better quantization than using naive quantization parameters.

[0250] Return to Figure 9The hyperlatent representation output by the superencoder neural network 905 is not only fed into the superdecoder neural network 906 to generate entropy parameters 907 for entropy encoding of the latent representation, but is also fed into the superencoder neural network 911 of the super-super network 902 to generate a hyper-hyperlatent representation (i.e., a latent representation of a latent representation of the input latent representation). The hyper-hyperlatent representation is then fed into the super-superdecoder neural network 912 to generate a set of entropy parameters 913, which indicate the distribution of the hyperlatent representation. That is, the entropy parameters 913 represent tensors of position and scaling parameters σ, μ, and δ, which define, for a given set of pixel values, what their zeros or mean are, what their scaling is, and, for example, the variance of the distribution. In the same manner as for the hypernetwork 901, the entropy parameter 913 allows the hyperlatent representation to be normalized 914 by making the distribution centered at a zero and scaling it around that zero, to produce a tensor with a value of minimum entropy, which can then be losslessly encoded very efficiently by applying a subsequent entropy encoding operation 915 to the normalized distribution, such as using rANS or any other entropy encoding operation, to produce a second bitstream portion 916 that includes the hyperlatent representation information used to encode the latent representation entropy into a first bitstream portion 910.

[0251] As in the hypernetwork 901, the hyperdecoder neural network 912 can also generate output quantization parameters (e.g., bin size and bin width). All hyperlatent representations are quantized before performing normalization operation 914 and / or entropy encoding operation 915 to facilitate quantization with reduced information loss compared to using naive quantization parameters.

[0252] Finally, the hyper-hyper-latent representation generated by the hyper-hyper-encoder neural network 911 can be normalized 917 using a naive (e.g., fixed, unlearned) entropy parameter 918, or, if a hyper-hyper-network (or some other network used to generate the entropy parameter) exists, the entropy parameter generated by it is used. An entropy encoding operation 919 is then performed to produce a third bitstream portion 920 that includes hyper-hyper-latent information used to entropy-encode the hyper-latent representation into a second bitstream portion 916.

[0253] The first, second, and third bitstream portions 910, 916, and 920 are then ready to be transmitted to the decoding side of the AI-based compression pipeline. Actually, now... Figure 10 The diagram in the middle is shown.

[0254] Now let's return to the concept of the encoder mask m that we've learned. Figure 9 The mask generator 921 g(ϕ) used to generate the mask m is further described. In Figure 9In this diagram, the mask generator 921 is shown as a standalone neural network, but it is also envisioned that it can be integrated with one or more of the following neural networks: a base encoder 904, a super encoder 905, a super decoder 906, a super-super encoder 911, and / or a super-super decoder 912. The output of the mask generator 921 is a mask m, which can be applied to one or more normalized distributions generated by normalization operations 908, 914, and 917. Figure 9 In the diagram, the mask m is shown as composed of different components m applied respectively to the normalized latent representation, the normalized hyperlatent representation, and / or the normalized hyperhyperlatent representation. i 922, m j 923 and m k Composed of 924. However, note that it is conceivable that the mask can be applied alternatively to only one or some of these, and as... Figure 9 As shown in the diagram, applying the mask to all is optional.

[0255] Therefore, we can use it as input (such as x when used in I-frame section 201). t Or when used in P-frame section 202, x t and x t-1 The operation of the mask generator g(ϕ) is defined by the weight ϕ: If we define the latent representation as y, the hyperlatent representation as z, and the hyperhyperlatent representation as w, and define the corresponding normalized versions of these as η y η z and η w Then, the entropy coding (EE) operations 909, 915, and 919 can be referred to as the encoding of mη. y mη z and mη w One or more of the executions are given: As mentioned above, it is assumed that the mask does not need to be applied to all y, z, and w; for example, it is also assumed that the mask can be applied only to η. y Applicable only to η z Applicable only to η w Applicable only to η y and η z Applicable only to η y and η w Applicable only to η w and η z , and / or any combination thereof.

[0256] The mask m can include a matrix or tensor of values ​​that are modified by, for example, multiplication or some other operation, the latent representation, hyperlatent representation, and / or hyperhyperlatent representation matrix or tensor to which it is applied.

[0257] The specific form of g(ϕ) is not important; for example, it can include a simple MLP, a convolutional neural network, or any other network with trainable parameters. In one example, g(ϕ) consists of alternating sequences of convolutional layers followed by activation layers, all or some of which have learnable weights, and optionally has one or more shaping layers to ensure that the shape of the output mask matches the mask shape of the latent representation, hyperlatent representation, and / or hyperhyperlatent representation to which it is to be applied.

[0258] The inventors have discovered that introducing this architectural mechanism allows AI compression pipelines to better learn during training which information (e.g., which pixels in latent representations, hyperlatent representations, and / or hyperhyperlatent representations) can be fully or partially masked on the encoding side without significantly increasing the distortion score. In other words, it provides the network with an architectural mechanism to explicitly learn the behavior implicitly learned when, for example, a soft-masking architectural mechanism is introduced on the decoding side.

[0259] Therefore, although Concept 2 is envisioned as a standalone architectural component, the inventors have discovered a synergistic effect between the mask generated by the mask generator g(ϕ) and the soft mask described in Concept 1 above. That is, when a pipeline with both the soft mask architectural component and the encoder mask architectural component is trained, the network on the encoder side learns to use the mask generator g(ϕ) to mask portions of the latent representation, hyperlatent representation, and / or hyperhyperlatent representation that would otherwise have been soft-masked on the decoding side. In effect, the network learns to use g(ϕ) to closely track the effect of the soft mask. As a result, combining the two components produces an architecture that minimizes bit transmission better than using either component alone, which would otherwise waste bits when masking on the decoding side.

[0260] Now let's return to the decoding side. Figure 10 The diagram shows what can be done with Figure 9 An exemplary decoder, superdecoder, and super-superdecoder architecture component coupled in an architecture. For example... Figure 10As shown, the bitstream consisting of the first, second, and third portions 910, 916, and 920 is received from the encoding side of the compression pipeline on the decoding side. The hyper-hyper-network portion 1001 starts running first, followed by the hyper-network 1002, and then the base network 1003. That is, the third bitstream portion 920 is entropy decoded 1004 and denormalized 1005 using naive entropy parameters 1006 (e.g., fixed and / or unlearned), or, if a hyper-hyper-hyper-network or some other network used to generate the entropy parameters exists, the entropy parameters generated by it are used. These correspond to those used on the encoding side for normalizing and entropy encoding of the hyper-hyper-latent representation, such as... Figure 9 As shown in the diagram. That is, the variance parameter σ is used for entropy decoding (e.g., rANS decoding or performing some other entropy decoding operation) to reconstruct the normalized hyper-hyper-latential representation η. w The normalized hyper-hyper-potential representation η w It is then denormalized (i.e., inverted with zero-point μ and scaled by δ) to reconstruct the hyper-hyper-latent representation w.

[0261] The hyper-hyper-latent representation w is then fed into the hyper-hyper-decoder 1007, which can substantially correspond to, in terms of weights and / or biases, as follows: Figure 9 The super-decoder 912 present on the encoding side is shown. Therefore, the output of the super-decoder 1007 on the decoding side includes the reconstruction of the entropy parameter 1008 for normalizing and entropy encoding the hyperlatent representation on the encoding side, and can thus be used to perform a reverse operation on the second bitstream portion 916 on the decoding side. That is, the variance parameter σ generated by the super-decoder neural network 1007 is used to perform rANS decoding (or other entropy decoding) 1009 on the second bitstream portion 916 to produce a normalized hyperlatent representation, and the reconstructed zeros and scaling parameters μ and σ generated by the super-decoder 1007 are used to denormalize the normalized hyperlatent representation 1010 to reconstruct the hyperlatent representation z on the decoding side.

[0262] The reconstructed hyperlatent representation z is then fed into the superdecoder neural network 1011, which can substantially correspond to, in terms of weights and / or biases, as shown in the figure. Figure 9The superdecoder 906 present on the encoding side, as shown, generates the reconstruction of entropy parameters 1012 on the encoding side for normalizing and entropy encoding of the latent representation y. These can therefore be used to perform the inverse operation on the first bitstream portion 910 on the decoding side. That is, the variance parameter σ generated by the superdecoder neural network 1011 is used to perform rANS decoding (or other entropy decoding) operation 1013 on the first bitstream portion 910 to generate a normalized latent representation, and the reconstruction zero-point and scaling parameters μ and σ generated by the superdecoder 1011 are used to denormalize the normalized latent representation 1014 to reconstruct the latent representation y on the decoding side.

[0263] The latent representation y can then be fed into the base decoder neural network to reconstruct the input. Or some representation thereof, or information associated with it. For example, if used in a streaming module of a streaming-residual architecture, the output could be a reconstruction representing information about the flow between two or more input images, rather than the images themselves. That is, Figure 9 and Figure 10 The architecture can be used in any part of an AI-based compression pipeline, where base networks, supernetworks, and / or super-supernetworks are used. For example, Figure 10 Previous decoded frames are also shown. It can be introduced at the 1015 level of the basic decoder neural network, such as, for example Figure 3 , 6 As shown in 7 and 8, either as part of the soft masking architecture described in Concept 1 above, or using, Figure 3 The simple residual mechanism shown, or using, as Figure 6 and Figure 8 The synthesis adapter shown, and / or any other mechanism that uses previously decoded images or image sequences in some way through the final reconstructed output.

[0264] More generally, the inventors have discovered that introducing encoder masks into the residual module of the stream-residual component of an AI-based compression pipeline results in a particularly significant reduction in bit rate due to compression of high-motion or low-motion scenes. This is understood to be a result of such scenes causing spikes in the bits required to compress frames of such scenes. Therefore, the masking allows the compression pipeline to selectively mask simply the parts of the scene that cause such spikes, thereby reducing compression rate spikes, surprisingly without causing a significant increase in distortion (i.e., without causing a decrease in image reconstruction accuracy). Furthermore, in many cases, the pipeline's neural network learns to use masking mechanisms to mask certain types of image content that are typically expensive to encode (in bits), not just content that might be masked by, for example, soft-masking architecture components. For example, it has been observed that the network masks parts that result in a potential representation with a very low "scaling" entropy parameter. In layman's terms, in such cases, the network has learned that it is best to allow a slight increase in distortion for such images to avoid a much more significant deterioration in compression rate.

[0265] It will also be understood that encoder-side masks can have different bit-saving effects when used on latent representations y, hyper-latent representations z, and / or hyper-hyper-latent representations w. For example, the inventors have observed that when encoder-side masks are used in conjunction with soft masks in the residual module of a compression pipeline, the bit savings for z are proportionally greater than those for y. The explanation for these kinds of differences is as follows. Consider a portion of the latent representation y corresponding to the region completely masked on the decoding side by the soft mask (i.e., where previously decoded image information is used entirely in the reconstruction of this portion, rather than the received transmitted information). Any bits spent on this portion are completely wasted. Therefore, the pipeline's neural network improves rate-distortion performance by attempting to minimize those bits (i.e., without wasting bits on sending information associated with those portions). The superencoder and superdecoder work together to produce entropy model parameters that define the distribution of the latent representation y in the final bitstream. For example, if the latent representation y is modeled using a discrete Laplace distribution, the superdecoder outputs parameters that parameterize that distribution. The Laplace distribution that can be efficiently entropy encoded has values ​​for the latent distribution y in the interval (μ-δ / 2, μ+δ / 2), where the bit cost is proportional to the scaling parameter σ. However, the problem is that the residual (basic) encoder has no information about what μ and σ will be, because those require hyperlocal information, which only appears in the hyperlatent representation z after the superencoder has acted on the latent representation y. Therefore, the responsibility of minimizing the wasted bits in y actually falls on z. From observing the decomposition plots of y, z, and w, it can be empirically determined that the latent representation y is the most expensive of the three to transmit in the residual module. It can be inferred that even without the encoder mask, the network has already tried to put as many bits as possible into z (and w) to reduce the bits spent on the disproportionately larger y. The same plot also shows that, when this is possible for a given input image, the neural network of the residual module attempts to do so by getting as close as possible to the ideal Laplace distribution mentioned above.

[0266] Therefore, when an encoder mask is introduced, it forces the underlying encoder to produce a potential representation y with ideal position parameters μ and scaling δ, regardless of z. This means there is no additional bit cost in z to make μ or δ “correct.” To make the Laplace distribution as cheap as possible in terms of bits, there is still a bit cost for the scaling parameter σ. Therefore, in this case, the bit savings of the encoder mask are primarily in z, not y.

[0267] In other words, despite this, some bits are still saved in y. This is because if the base encoder produces a distribution of μ and δ that is far from the ideal Laplace distribution (e.g., this might appear in very fine detail in some frames), the mask modifies that distribution to push it closer to the ideal distribution, resulting in fewer bits being needed to transmit that distribution after entropy encoding.

[0268] Finally, we will understand. Figure 9 The concept of encoder masks introduced in this paper for masking normalized latent representations, hyperlatent representations, and / or hyperhyperlatent representations can be applied at different points in the information stream. That is, while the inventors have found that introducing the masking operation precisely before the entropy coding step allows the masking step to act as a final filter to avoid the transmission of unwanted bits, the masking can also, or alternatively, be introduced before the normalization steps 908, 914, and 917. In this case, the entropy parameters generated by the encoder-side decoders 906 and 912 will differ from the entropy parameters when the masking is applied after the normalization step. Specifically, they ultimately model the entropy of the already masked distribution, which can be advantageous in some cases because the masked portion of the distribution contributes very little to the total entropy of the distribution and can produce a very simple distribution after normalization. In turn, these already have very low entropy and can be losslessly compressed into a small bitstream size without requiring further masking steps.

[0269] Concept 3: Mini-frames for distortion correction Certain scene types, such as scene transitions, occluded scenes (where objects move and reveal a highly complex pixel-structured background behind them), significant frame changes due to high motion, failure to describe object misalignment within frames caused by high motion, and other non-motion-compensable changes in scene content, can be challenging for AI-based compression pipelines to handle well. Non-motion-compensable changes refer to variations between a reference frame and the current frame, where any amount of optical flow-based motion compensation cannot model the changes because the reference frame does not contain sufficient or any information useful for predicting the current frame. Such scenes contain frame sequences that can cause the AI-based compression pipeline network to spend disproportionately many bits to compress, resulting in artifacts and other failure modes in the reconstructed frames. It will be understood that failure modes of a neural network occur when the output of a given input deviates significantly from the output the network would expect to produce from the input. For example, for many scene types, an AI-based compression pipeline may produce reconstructed frames with a certain target rate score and / or distortion score, but for other scene types, it suffers from failure modes that cause, for example, large spikes in the distortion score, indicating that the output frame does not look like the input frame at all.

[0270] Generally, this poor performance can be partly attributed to the operation of compression pipelines based on flow residuals. That is, they produce outputs from the residual portion 210 used to reconstruct x. t Reference frame As described above, this reference frame Motion compensation based on optical flow is achieved by warping past frames using a flow graph generated by flow portion 206. This is generated. However, there are several situations where frames generated by motion compensation are not a useful basis for the residual part 210, such as scene transitions, significant changes in frames due to high motion, or failure to describe object misalignment in frames caused by high motion, etc. In fact, this results in the image content at a given location being... and x t The image content changes drastically between these conditions, and the residual component 210 strives to correct the image content without artifacts. In these cases, the inventors have recognized that when the final reference frame for generating the residual component 210 is... In such cases, applying coarse, inexpensive corrections to these regions is often bit-rate efficient and aesthetically pleasing. Concept 3 addresses this type of correction.

[0271] Specifically, Concept 3 involves a distorted reference frame. The residuals are corrected before being passed to the pipeline, thereby reducing the difficulty of the tasks to be performed by the residuals 210 and reducing the final output image size. The number of artifacts in the image is reduced, and the residual portion is reduced by 210 to produce the final image. The number of bits spent. Generally, this correction can be achieved by introducing a new output tensor on the decoding side of the streaming section 206, the new output tensor including information about the current frame x. t Information and how to use this information to correct Information.

[0272] For example, a stream superdecoder can be configured to output not only the entropy parameter (e.g., as in example...) Figure 10 (The position, scaling parameters, etc. shown), and also output x t The learning representation, referred to below as For example, a tensor having luminance and chrominance channels and a predetermined resolution, which can be related to x. t The same resolution or a lower resolution. Then, the basic stream decoder can be configured to output not only the stream graph, but also the learned occlusion mask m. occ The occlusion mask m occ Specify by using newly generated Replace or modify the distorted reference frame The pixels are used to correct the distorted reference frame. The location and extent of the fault. Very generally, the stream section 206 on the decoding side of the pipeline now includes an additional output that the network can learn to use to correct problematic fault modes caused by fault modes in the stream section. This ensures that the residual portion 210 is not burdened by this task.

[0273] More formally, the corrected distorted reference frame It can be defined as: Where m occ It is the learned mask output by the basic stream decoder. The input frame x is fed into the encoding side of the stream section 206. t The learning representation, and (1−m) occ ) is the mask m occ The inverse. In one implementation, the mask m occ It can be a binary mask, and therefore specifies Which pixels are taken from And which ones are taken from Alternatively, the mask does not have to be binary and can be specified instead. and Pixel-by-pixel blending.

[0274] During training, the pipeline's network learning uses m occ and To construct Any part, when there is very high motion, mainly from Construction, and when low motion is present, mainly from Construction. This means the network is learning by using data from... Pixel replacement pixels to correct Any unwanted flow artifacts. Meanwhile, online learning in It basically includes only those pixels that might be used for the correction. As a result, Capture x more accurately t and x t-1 The actual movement between them, and therefore the difficulty of the task that the residual part 210 must perform is reduced, and the final reconstructed frame is obtained. It contains fewer artifacts and uses fewer bits in the bitstream.

[0275] Configure the network of stream section 206 to output m occ and The stream portion 206 is allowed to operate effectively as an I-frame module simultaneously, thus learning during training, which in some cases may require direct access from x. t Reconstruct relevant information (in) (in the form) and discard Some or all of the pixels, whenever x t and x t-1 When non-motion-compensable changes occur, such as when the scene content changes in x t and x t-1 The drastic differences between them will result in peak values ​​in bit rate and / or distortion score. In extreme cases, such as when there is a scene change or scene switching, m occ and Even allowing the flow portion of the 206 network to choose completely from To construct from pixels Therefore, discard All pixels in the pixel. In which the mask m occ In the toy example of binary masking, this means m occ Including all zeros, and It operates as a scene-switching detector for learning. When the residual part 210 receives the corrected distorted frame... At that time, its task was simply to adjust. To get as close as possible to x t This attempts to fill severely distorted, artifact-laden images with fewer bits per second. As close to x as possible t It's much simpler.

[0276] As described above, one or more of the stream decoder, super decoder, and / or ultra-super decoder can be configured to generate tensors. and m occ Additional outputs in the form of [other outputs], and then using those outputs to modify the flow portion 206 of the pipeline in the manner described above. Generally, this may include introducing one or more output layers into the network, and the tensors output by those layers (i.e., [other outputs]) and m occ ) can be passed (if applicable) to the execution The correction function.

[0277] Because the use of these new outputs directly affects the bit rate and / or distortion during the forward pass, the weights of the streaming decoder, superdecoder, and / or super-superdecoder networks are updated during training, making... and m occ The computational value of the tensor minimizes the bit rate and / or distortion of the rate-distortion loss function. This is done while monitoring during training. and m occ When considering pixel values, it can be seen that the network tends to learn to use these outputs in the manner described above. That is, it serves as a mechanism for using reference frames to correct undesirable distortions caused by fault modes in the streaming network.

[0278] For example, visualization in a graph and m occ show The pixel coordinates are essentially empty (e.g., this emptiness represents a gray pixel or some other color, which is the same for all pixels within it that have not moved or have low motion, allowing for efficient compression of very few bits), while simultaneously containing x. t The representation is that as long as x exists t and x t-1 The non-motion between them can compensate for the changes. Furthermore, m occ Ultimately, this serves as a mask that covers the empty pixel portion of the overlay frame but allows non-motion-compensating variations to pass through. (x from the toy example scene) t-1 1100, x t 1101 1102 and m occ The visualization of 1103 Figure 11 As shown in the image. In this case, we will use high motion as an example of non-motion-compensable changes. First frame x t-1 With the second frame x t There exists high motion, in the form of a character jumping sideways against a low-motion background with a sun and trees. When x t 1101 and x t-1 1100 is fed through the streaming section 206 of the pipeline and decoding is performed, in addition to their usual output, the streaming superdecoder now... Output x directly in the form of 1102 t The representation of m, and the output of the stream decoder. occ 1103. In Figure 11 As can be seen, 1103 has an area with empty pixel 1104 and patch 1105, where x t Most of the reconstruction corresponds to locations with high motion and non-motion compensable changes. This also applies to m. occ Because there exists a corresponding The masked region 1106 of the empty pixels, and the corresponding Chinese x t The unmasked region 1107 of the largely reconstructed region 1105, where high motion exists when the following combination 1203 is used: The result is from Low motion pixels and from A frame constructed from high-motion pixels. This frame is larger than... The uncorrected version captures high motion better.

[0279] Figure 12 This illustrates the introduction of the aforementioned architectural mechanism into... Figure 8 In the pipeline. References with the same number refer to elements with the same number, and this will not be repeated here. However, with Figure 8 Unlike in the previous example, both the stream-based decoder 209 and the stream superdecoder 806 are configured to output the corresponding tensor (i.e., m). occ 1201 and 1202), the tensor and Combination 1203, This is achieved by applying the twist operation w to the previously decoded frame. (Using the reconstructed stream also output by the underlying stream decoder as described above) This is produced by [the process]. The result of these steps is [the process]. Then it is passed to the residual section 210 in the usual manner as described above, and the residual section 210 uses it to finally reconstruct the current frame. .

[0280] Although the above examples use a stream superdecoder and a stream fundamental decoder to generate 1102 and m occ However, the 1203 operation can be performed by introducing the corresponding new outputs from those networks and using those outputs as described above. To select any network in the flow section 206 for this purpose. During training, the network will learn to use the new outputs as a mechanism to correct the poor flow estimation as described above by predicting the values ​​of these outputs based on the given input frames that produce this effect. This can be achieved by, as... Figure 11 Visualize m in the way shown occ and To confirm. For example, imagine that any one of the stream-based decoder 209, super decoder 806, and super-super decoder 808 can output m in any combination. occ or Any one of them.

[0281] Given The goal is to make it usable for correction The pixel source of the "bad" pixels, where m occ The inventors have recognized that specifying the location to be corrected does not necessarily require a coordinate with x. t and x t-1 The same higher resolution generation and / or m occ This is at least because the residual portion 210 effectively compensates for poor resolution artifacts, provided the pixels are approximately correct. and / or m occ The generation does not necessarily need to be at full resolution. Therefore, and m occ It can be relative to x t and x t-1 Downsampling, lower resolution space (e.g., h / 8, w / 8 or h / 16, w / 16 or below x) t and x t-1 Efficiently generated in h and w resolutions (and some other resolutions), and then upsampled to before performing correction. The resolution is [missing information]. Alternatively, the inventors have discovered that generating m [missing information] at higher resolutions (e.g., full resolution) is possible. occ It has an advantage. This is because m occ Final determination The position to be corrected, and because of m occ High-entropy shapes can be generated very cheaply (with a few bits) because such shapes share boundaries with the content being transmitted in the optical flow information, thus allowing the utilization of mutual information between them, which in turn means Fewer bits can be spent on creating accurate shapes, while more bits can be spent on accurate colors. Therefore, even when generating m at full resolution... occ It is also relatively inexpensive and promotes more accurate and better calibration. .

[0282] For the sake of completeness, although Figure 12 Not shown, but it is envisioned that it may include from Figure 7 soft masking mechanisms and from Figure 9 The encoder mask.

[0283] The following pseudocode will now describe more details of the illustrative training implementation.

[0284] In other words, selecting a training dataset The learning rate η and the number of training steps or epochs E are defined, and the stream part g is defined. ϕ and residual part h ψ Network architecture f θ For example, Figure 3 As shown in the figure. Network parameters θ, ϕ ψ is initialized (e.g., randomly), and then the training loop begins. For the training data... Each batch in, via f θ Perform the forward pass. As mentioned above, this includes using the flow portion of the architecture, g. ϕ Predict optical flow information and use it to warp previously decoded frames x. t-1 To generate and the use of the stream part g ϕ To predict the mask m occ and the current frame The associated representation. Then use the mask m. occ To combine and To generate Then combine the distorted previously decoded frames. With the current frame x t Together they are fed into the residual part h of the architecture. ψ China-Israel calculation The total loss is then calculated by combining the distortion term D, the rate term R, and any other loss terms (not shown). A backpropagation is then performed to compute the gradient based on the loss, and an optimizer (such as stochastic gradient descent SGD or some other known optimizer) is used to optimize the parameters θ. ϕ 、ψ.

[0285] Optionally, a validation loss can be calculated. The learning rate, batch size, and / or number of epochs can be optimized during training, for example, using a learning rate scheduler or some other hyperparameter optimization method. More generally, hyperparameters can be optimized experimentally.

[0286] One problem that may arise during training is f θ Network (including the streaming part g) ϕ and residual part h ψ It has the ability to generate occlusion masks m through learning. occ The tendency to "cheat" by finding a local rather than a global minimum loss, the occlusion mask m occ It has the ability to completely mask all input frames. The value (i.e., the value that leads to) This means that the benefit of the architecture mechanism acting as a corrective measure to compensate for the effects of bad flow and bad distortion is not realized. To prevent this, annealing or regularization scheduling can be introduced, which changes which loss terms contribute to the overall loss, and / or uses a method already trained to produce high-quality... and m occ Teachers will monitor and learn through a network, gradually increasing the number of teachers. occ It can have an impact on the overall loss.

[0287] For example, for the sake of argumentation, assume that in addition to the usual rate and distortion terms, the loss function includes the following additional loss term M: Where mse is a function that estimates the mean squared error between its input independent variables, where m teacher Calculated by the teacher network, where m occ and (use The corrected reference frame is computed by the network being trained, i.e., by the network with the streaming part g. ϕ and residual part h ψ f θ Calculation. The teacher network can include f θ Networks with the same or similar architecture, including the streaming component g ϕ and residual part h ψ However, it has been trained using a training scheme that allows it to produce high-quality occlusion masks and microframes, but not necessarily in a way that makes it feasible to deploy on end devices. For example, the teacher's architecture may be structurally similar to the student's architecture, but much larger than the architecture that can run on the device, and the unsupervised training schedule may take much longer than iterative development is feasible. By way of a non-limiting example, if the student training schedule has 200,000 training steps, the teacher training schedule may have 1,000,000 or more steps.

[0288] During the forward pass, the new total loss can therefore be calculated as: However, at the beginning of the training, from and mse(m occ ,m teacher There is a conflict between the gradients of (). This is when the gradients used to generate microframes The quality is poor, therefore Punishment occlusion mask m occ It is not all 1s (i.e., a full mask is applied, and therefore poor-quality microframes are not used at all). On the other hand, the teacher's occlusion mask m teacher Students typically have more zero areas (i.e., more use of microframes) than students. (This is because it assumes the use of high-quality microframes from the teacher.) And therefore they encourage students to set those regions of the mask to 0. To ensure that the mask is learned efficiently, gradients can be prevented from shifting at the beginning of training. Flow direction mask m occ This is achieved by gradually introducing the mask using only annealing or regularization scheduling. Two ways to achieve this are to separate the mask in the function that generates distorted reference frames for this part of the training, or to use the teacher's occlusion mask m. teacher The latter option is preferred because it performs slightly better, but both options are possible.

[0289] One reason for favoring the latter option is that once supervised training is introduced, the student's occlusion mask m occ It quickly becomes almost entirely 1 (i.e., the operation completely masks the microframe). However, the microframe The only gradient received during initial training is through Therefore, if a student mask of all 1s is used, m occ Then microframe They are essentially not involved in training until the student mask improves. This results in undertrained microframes. This is detrimental to performance. Using the teacher's mask m in the warp operation... teacher This means microframes More frequently used (i.e., via teacher mask m) teacher More frequent zero regions), and therefore receive useful gradients throughout training.

[0290] Returning to annealing or regularized scheduling, we define the total loss as: in: in It uses microframes and the separated mask m occ The corrected distorted reference frame prevents gradients from passing through. Flow direction mask m occ And among them It uses microframes and mask m occ The corrected distorted reference frame, without separation operation, allows the gradient to flow through to the occlusion mask m. occ .

[0291] This will be understood as the occlusion mask m occ The gradient on therefore has the following form: This facilitates the use of regularization or annealing parameters α and β to gradually switch from The gradient.

[0292] At the start of training scheduling, α=0, indicating that gradients cannot be drawn from the training table. Propagation is done through a mask, allowing the network to focus on learning. and m occ As a separate task. At the end of the training schedule, α=1, indicating that the gradient can be drawn from... Full propagation through the mask allows for better alignment of the task of learning two tensors (i.e., minimizing). At this point, the value of β can be set to 0, making the occlusion mask m occ Fully adaptable microframes generated by the network Instead of from mse(m occ ,m teacherThe mask of the miniframes received as encouragement is matched to the hypothetical teacher. The value of α can be progressively moved from 0 to 1 according to some predefined schedule based on, for example, the number of training steps or some other criterion (such as how close the overall loss converges to a certain predetermined loss value, or some other requirement).

[0293] For completeness, note that annealing or regularization scheduling, as shown above, promotes m occ and It is particularly effective when used in conjunction with the encoder mask of Concept 2 above. This is because the inventors have discovered that using m occ and Along with the encoder mask, the network is more easily "cheated" during training because the encoder mask and the occlusion mask m occ Both are used collaboratively by the network to control what information is sent on the encoding side and / or what information is masked on the decoding side. When used together, they may reach a local minimum during training, where the network masks all input frames on the decoding side. The encoder mask then learns to mask out anything that will be used on the encoding side. The above annealing schedule is highly effective in overcoming this, and therefore helps to facilitate the integration of the encoder mask of Concept 2 with the m of this concept. occ and The architecture and mechanisms are combined.

[0294] As will be understood, although the above example uses a streaming superdecoder 806 and a streaming base decoder 209 to predict m on the decoding side... occ 1201 and 1202, but this is merely illustrative. At the most general level, the decoding side is correcting the distortion of previous decoded frames using some reference containing useful information to perform the correction. In the example above, this additional functionality is directed to the network that primarily serves other purposes (i.e., the network of the flow section 206 of the pipeline). However, additionally or alternatively, it is envisioned that a dedicated network could be introduced into the pipeline, the primary purpose of which is to generate a network for correction. Reference information. An illustrative example of this is... Figure 13 As shown in the image.

[0295] Figure 13 Corresponding to Figure 12 In addition to now using a dedicated encoder 1301 and decoder 1302 to generate bitstream portion 1303 to compute from x t (and / or x) t-1 (not shown) 1202. Therefore, the dedicated encoder 1301 and decoder 1302 can be said to be similar in some ways to the I-frame module 201 of the compression pipeline, since it does not depend on time information and produces x t The representation of . Alternatively, it can also be represented by including x. t-1 The timing information is used as input. Furthermore, the dedicated encoder 1301 and decoder 1302 pair can operate at resolutions much lower than those of the I-frame module. As previously stated, The generation of 1202 does not need to be performed at full resolution because its primary purpose is to serve as an information source to correct artifacts introduced by the streaming network when predicting poor streaming patterns in fault modes. Even in This objective can still be achieved even when the information is generated at low resolution and therefore upsampled to the resolution of the reference frame during correction, which is coarse. That is, even when the representation of x is approximate... t Even a very coarse representation of the information is better than the artifact-laden creation of a streaming network suffering from catastrophic failure modes.

[0296] Therefore, one or more downsampling operations can be introduced into encoder 1301 to adjust its input x. t This allows the encoder 1301 and decoder 1302 pair to operate in, for example, lower resolution spaces such as 1 / 2, 1 / 4, 1 / 8, 1 / 16, 1 / 32, or some other lower resolution space compared to the I-frame module. By operating in a lower resolution space, the computational overhead of the dedicated encoder 1301 and decoder 1302 pair can be kept small, thereby facilitating real-time or near-real-time operation.

[0297] Although in addition to optical flow information, Figure 13 The basic stream decoder 209 is still used to predict the occlusion mask m. occ 1201, but this function can also be offloaded to one or more dedicated networks (not shown). For example, this function can be offloaded to a dedicated encoder 1301 decoder 1302 pair or one or more other dedicated networks, where the input includes x t and x t-1 Both.

[0298] Furthermore, despite Figure 13 As not shown, the dedicated encoder 1301 and decoder 1302 may also be composed of an associated supernetwork and / or ultra-supernetwork to communicate with... Figure 13 It supports other networks in the same way.

[0299] m occ and The advantage of offloading computation to one or more dedicated networks is that, in many cases, training stability is improved because the network does not attempt to learn to simultaneously serve its primary purpose (e.g., the basic stream decoder and the superstream decoder) and also serve the purpose of generating m. occ and A set of weights for secondary purposes.

[0300] The subject matter and functional operations described in this specification can be implemented in digital electronic circuits, tangibly implemented computer software or firmware, or computer hardware, including the structures disclosed in this specification and their structural equivalents, or combinations thereof. The subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier, for execution by or control of the operation of a data processing device. Alternatively or additionally, program instructions can be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, generated to encode information for transmission to a suitable receiver device for execution by the data processing device. Computer storage media can be machine-readable storage devices, machine-readable storage substrates, random or serial access memory devices, or combinations thereof. However, computer storage media are not propagation signals.

[0301] The term "data processing apparatus" encompasses all kinds of devices, apparatuses, and machines used for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The apparatus may include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0302] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script, or code) can be written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but does not need to, correspond to a file in a file system. A program may be stored as a portion of a file that holds other programs or data, for example, as one or more scripts stored in a markup language document, as a single file dedicated to the program in question, or as a plurality of coordinated files, such as a file storing portions of one or more modules, subroutines, or code. A computer program can be deployed to execute on a single computer or on multiple computers located at a site or distributed across multiple sites and interconnected by a communication network.

[0303] The processes and logic flows described in this specification can be executed by one or more programmable computers, which execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by dedicated logic circuits, and the apparatus can be implemented as dedicated logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits).

[0304] A computer suitable for executing computer programs includes (as an example, based on) a general-purpose or special-purpose microprocessor or both, or any other kind of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory or both. The basic components of a computer are the central processing unit for executing or carrying out instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include, or be operatively coupled to, one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, to receive data from or transfer data to, or both. However, a computer does not need to have such a device. Furthermore, a computer can be embedded in another device, such as a mobile phone, personal digital assistant (PDA), mobile audio or video player, VR headset, game console, GPS receiver, server, mobile phone, tablet computer, laptop computer, music player, e-book reader, laptop or desktop computer, PDA, smartphone, or other fixed or portable device including one or more processors and computer-readable media, or portable storage devices such as Universal Serial Bus (USB) flash memory, to name just a few.

[0305] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented or incorporated therein by dedicated logic circuitry.

[0306] The subject matter described in this specification can be implemented in a computing system that includes back-end components (e.g., as a data server), middleware components (e.g., an application server), front-end components (e.g., a client computer with a graphical user interface or web browser through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), such as the Internet.

[0307] A computing system can include clients and servers. Clients and servers are typically geographically separated and usually interact through a communication network. The client-server relationship arises from computer programs running on their respective computers, and they have a client-server relationship with each other.

[0308] While this specification contains many specific implementation details, these should be interpreted as descriptions of features that may be specific to particular examples of a particular invention. Some features described in this specification within the context of separate examples may also be implemented in combination in a single example. Conversely, various features described in the context of a single example may also be implemented separately or in any suitable sub-combination in multiple examples.

[0309] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to obtain the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various system modules and components in the above examples should not be construed as requiring such separation in all examples, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

Claims

1. A method for encoding, transmitting, and decoding lossy images or videos, the method comprising the following steps: Receive the first image and the second image at the first computer system; Optical flow information is generated using a first neural network with a first image and a second image, the optical flow information indicating the difference between the first image and the second image; A second neural network is used to generate residual information using the first image and optical flow information, the residual information indicating the difference between the first image and the second image; The residual information is transmitted to the second computer system; The residual information is decoded using a third neural network to generate an intermediate output image; as well as A portion of the previously decoded image is combined with a portion of the intermediate output image to produce a final output image, wherein the final output image is an approximation of the first image.

2. The method of claim 1, further comprising extracting information from one or more intermediate layers of a first neural network, wherein said portion is based on the extracted information.

3. The method of claim 2, wherein the extracted information includes one or more outputs from one or more convolutional layers and / or activation layers of the first neural network.

4. The method according to any one of claims 1 to 3, comprising estimating the portion by applying a mask to a previously decoded image and / or an intermediate output image.

5. The method of claim 4 when referring to claim 2, wherein the mask is based on the extracted information.

6. The method of claim 4 or 5, wherein applying the mask comprises applying the mask to one of the previously decoded images or intermediate output images, and applying the inverse of the mask to the other of the previously decoded images or intermediate output images.

7. The method according to any one of claims 4 to 6, wherein the mask comprises a non-binary mask.

8. The method according to any one of claims 4 to 7, comprising using a third neural network to generate a mask.

9. The method of claim 8, wherein using a third neural network to generate the mask comprises: One or more convolutional layers and / or activation layers are applied to the residual information and the extracted information.

10. The method according to any one of claims 4 to 7, comprising using a fourth neural network to generate a mask.

11. The method of claim 10, wherein using a fourth neural network to generate the mask comprises: One or more convolutional layers and / or activation layers are applied to the residual information and the extracted information.

12. The method according to any one of claims 4 to 11, wherein the mask specifies the degree of contribution of the previously decoded image to the final output image compared to the degree of contribution of the intermediate output image to the final output image.

13. The method according to any one of claims 4 to 12, wherein the mask is applied on a pixel-by-pixel basis to the previously decoded image and / or intermediate output image to perform pixel-by-pixel partial frame skipping of the first image.

14. The method according to any one of claims 1 to 13, wherein the optical flow information output by the first neural network is used to decode the residual information using the third neural network.

15. The method according to any one of claims 1 to 14, wherein the previously decoded image is an approximation of the second image.

16. The method according to any one of claims 1 to 13, comprising repeating the generation, transmission, decoding, and combination of a sequence of video frames to reconstruct the sequence of frames at a second computer system to mitigate image artifacts in the reconstructed sequence of frames.

17. A data processing system configured to perform the method of any one of claims 1 to 16.

18. A method for receiving and decoding lossy images or videos, the method comprising the following steps: A first neural network is used at a first computer system to decode optical flow information, which indicates the difference between a first image and a second image; A second neural network is used at a first computer system to decode residual information to produce an intermediate output image, the residual information indicating the difference between a first image and a second image; A portion of the previously decoded image is combined with a portion of the intermediate output image to produce a final output image, wherein the final output image is an approximation of the first image.

19. A data processing apparatus configured to perform the method of claim 18.

20. A computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 18.

21. A computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 18.

22. A method for training one or more neural networks for use in lossy image or video encoding, transmission, and decoding, the method comprising the steps of: Receive the first image and the second image at the first computer system; Optical flow information is generated using a first neural network with a first image and a second image, the optical flow information indicating the difference between the first image and the second image; A second neural network is used to generate residual information using the first image and optical flow information, the residual information indicating the difference between the first image and the second image; The residual information is decoded using a third neural network to generate an intermediate output image; as well as A portion of the previously decoded image is combined with a portion of the intermediate output image to produce a final output image, wherein the final output image is an approximation of the first image; The function is evaluated based on the difference between the first image and the output image; The parameters of the first, second, and / or third neural networks are updated based on the evaluated function; as well as Repeat the above steps using the first set of input images to generate the first, second, and / or third trained neural networks.

23. A method for encoding, transmitting, and decoding lossy images or videos, the method comprising the following steps: Receive the first image and the second image at the first computer system; Optical flow information is generated using a first neural network with a first image and a second image, the optical flow information indicating the difference between the first image and the second image; Extract information from one or more intermediate layers of the first neural network; A second neural network is used to generate residual information using the first image and optical flow information, the residual information indicating the difference between the first image and the second image; The residual information is transmitted to the second computer system; The residual information is decoded using a third neural network to generate an intermediate output image; as well as An intermediate output image is modified based on information extracted from one or more intermediate layers to produce a modified output image, wherein the modified output image is an approximation of the first image.

24. The method of claim 23, further comprising using output extracted from one or more intermediate layers to generate a mask.

25. The method of claim 24, wherein modifying the intermediate output image includes applying a mask to the intermediate output image.

26. The method according to any one of claims 23 to 25, wherein the mask comprises at least three distinct values ​​between a minimum value and a maximum value.

27. The method according to any one of claims 23 to 26, comprising using one or more layers of a third neural network to generate a mask.

28. The method according to any one of claims 23 to 27, comprising using a fourth neural network to generate a mask.

29. The method according to any one of claims 23 to 28, wherein the information extracted from one or more intermediate layers of the first neural network includes one or more outputs from one or more convolutional layers and / or activation layers of the first neural network.

30. The method according to any one of claims 23 to 29, comprising repeating the steps for a sequence of video frames to reconstruct the sequence of video frames at a second computer system.

31. The method of claim 30, wherein the step is repeated for a sequence of video frames to mitigate image artifacts in the reconstructed sequence of frames.

32. A data processing system configured to perform the method of any one of claims 23 to 31.

33. A method for receiving and decoding lossy images or videos, the method comprising the following steps: A first neural network is used at a first computer system to decode optical flow information, which indicates the difference between a first image and a second image; Extract information from one or more intermediate layers of the first neural network; A second neural network is used at a first computer system to decode residual information to produce an intermediate output image, the residual information indicating the difference between a first image and a second image; An intermediate output image is modified based on information extracted from one or more intermediate layers of a first neural network to produce a modified output image, wherein the modified output image is an approximation of the first image.

34. A data processing apparatus configured to perform the method of claim 33.

35. A computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 23 to 33.

36. A computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 23 to 33.

37. A method for encoding, transmitting, and decoding lossy images or videos, the method comprising the following steps: Receive the first image at the first computer system; The first image is encoded using a first neural network to generate a latent representation; At least partially mask the underlying representation; Transmit the masked potential representation to a second computer system; A second neural network is used to decode the latent representation of the mask to produce an output image, which is an approximation of the first image.

38. The method of claim 37, wherein at least partially masking the latent representation reduces the entropy of the distribution of the latent representation.

39. The method of claim 37 or 38, wherein the masking comprises generating a mask based on a first image.

40. The method of claim 39, further comprising using a third neural network to generate a mask.

41. The method according to any one of claims 38 to 40, comprising entropy encoding a masked latent representation into a bit stream, wherein the transmission masked latent representation comprises a transmission bit stream, and wherein the step of at least partially masking the latent representation reduces the size of the bit stream after entropy encoding compared to a bit stream generated without the step of at least partially masking.

42. The method according to any one of claims 38 to 41, wherein at least partially masking the latent representation includes at least partially masking the normalized representation of the latent representation.

43. The method of claim 42, further comprising using one or more first-learned scaling and / or zero-point parameters to generate a normalized representation of the latent representation.

44. The method of claim 43, further comprising encoding the latent representation using a fourth neural network to generate a hyperlatent representation, and decoding the hyperlatent representation using a fifth neural network to generate one or more first-learned scaling and / or zero-point parameters.

45. The method of claim 44, further comprising at least partially masking the hyperpotential representation to reduce the entropy of the distribution of the hyperpotential representation.

46. ​​The method according to any one of claims 44 to 45, comprising entropy encoding the masked hyperlatent representation and transmitting it to a second computer system.

47. The method of claim 46, wherein at least partially masking the hyperpotential representation includes at least partially masking the normalized representation of the hyperpotential representation.

48. The method of claim 47, further comprising using one or more second-learned scaling and / or zero-point parameters to generate a normalized representation of the hyperlatent representation.

49. The method of claim 48, further comprising encoding the hyperlatent representation using a sixth neural network to generate a hyperhyperlatent representation, and decoding the hyperhyperlatent representation using a seventh neural network to generate second learned scaling and / or zero-point parameters.

50. The method of claim 49, further comprising at least partially masking the hyper-hyper-latent representation to reduce the entropy of the distribution of the hyper-hyper-latent representation.

51. The method of claim 50, wherein at least partially masking the hyper-hyper-latent representation includes at least partially masking the normalized representation of the hyper-hyper-latent representation.

52. The method of claim 50 or 51, further comprising encoding the masked hyper-hyper-latent representation entropy and transmitting it to a second computer system.

53. A method for encoding and transmitting lossy images or videos, the method comprising the following steps: Receive the first image at the first computer system; The first image is encoded using a first neural network to generate a latent representation; At least partially mask the underlying representation; The masked potential representation is transmitted to a second computer system.

54. A method for receiving and decoding lossy images or videos, the method comprising the following steps: A masked latent representation is received at a second computer system, the masked latent representation being generated by encoding the first image using a first neural network and by at least partially masking the latent representation; A second neural network is used to decode the latent representation of the mask to produce an output image, which is an approximation of the first image.

55. A data processing apparatus configured to perform the method of any one of claims 37 to 54.

56. A computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 37 to 54.

57. A computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 37 to 54.

58. A method for encoding, transmitting, and decoding lossy images or videos, the method comprising the following steps: Receive the first image and the second image at the first computer system; Optical flow information is generated using a first neural network with a first image and a second image, the optical flow information indicating the difference between the first image and the second image; A second neural network is used to generate residual information using the first image and optical flow information, the residual information indicating the difference between the first image and the second image; At least partially masking residual information; The masked residual information is transmitted to the second computer system; The masked residual information is decoded using a third neural network to generate an output image, wherein the output image is an approximation of the first image.

59. The method of claim 58, wherein at least partially masking the residual information reduces the entropy of the distribution of the residual information.

60. The method of claim 58 or 59, wherein the masking comprises generating a mask based on a first image and / or a second image.

61. The method of claim 60, further comprising using a fourth neural network to generate a mask.

62. The method according to any one of claims 58 to 61, comprising entropy encoding masked residual information into a bit stream, wherein transmitting the masked residual information comprises transmitting the bit stream, and wherein the step of at least partially masking the residual information reduces the size of the bit stream after entropy encoding compared to a bit stream generated without the step of at least partially masking.

63. The method according to any one of claims 58 to 62, wherein at least partially masking the residual information includes at least partially masking a normalized representation of the residual information.

64. The method of claim 63, further comprising using one or more first-learned scaling and / or zero-point parameters to generate a normalized representation of the residual information.

65. The method of claim 64, further comprising encoding the residual information using a fifth neural network to generate a hyperlatent representation of the residual information, and decoding the hyperlatent representation of the residual information using a sixth neural network to generate one or more first-learned scaling and / or zero-point parameters.

66. The method of claim 65, further comprising at least partially masking the hyperlatent representation of the residual information to reduce the entropy of the distribution of the hyperlatent representation of the residual information.

67. The method of claim 66, further comprising entropy encoding the masked hyperlatent representation of the residual information and transmitting it to a second computer system.

68. The method according to any one of claims 66 to 67, wherein the hyperlatent representation that at least partially masks the residual information comprises a normalized representation of the hyperlatent representation that at least partially masks the residual information.

69. The method of claim 68, further comprising using one or more second-learned scaling and / or zero-point parameters to generate a normalized representation of the hyperlatent representation of the residual information.

70. The method of claim 69, further comprising encoding the hyper-latent representation of the residual information using a seventh neural network to generate a hyper-hyper-latent representation of the residual information, and decoding the hyper-hyper-latent representation of the residual information using an eighth neural network to generate second-learned scaling and / or zero-point parameters.

71. The method of claim 70, further comprising at least partially masking the hyper-hyper-latent representation of the residual information to reduce the entropy of the distribution of the hyper-hyper-latent representation of the residual information.

72. The method of claim 71, wherein the hyper-hyper-latent representation that at least partially masks the residual information comprises a normalized representation of the hyper-hyper-latent representation that at least partially masks the residual information.

73. The method according to claim 71 or 72, comprising encoding the masked hyper-hyper-latent representation of the residual information by entropy encoding and transmitting it to a second computer system.

74. A method for encoding and transmitting lossy images or videos, the method comprising the following steps: Receive the first image and the second image at the first computer system; Optical flow information is generated using a first neural network with a first image and a second image, the optical flow information indicating the difference between the first image and the second image; A second neural network is used to generate residual information using the first image and optical flow information, the residual information indicating the difference between the first image and the second image; At least partially masking residual information; as well as The masked residual information is transmitted to the second computer system.

75. A method for receiving and decoding lossy images or videos, the method comprising the following steps: The masked residual information is received at the second computer system. The masked residual information is generated by using a first neural network to generate optical flow information using a first image and a second image, and by using a second neural network to generate residual information using the first image and the optical flow information, and by at least partially masking the residual information. The residual information indicates the difference between the first image and the second image. as well as The masked residual information is decoded using a third neural network to generate an output image, wherein the output image is an approximation of the first image.

76. A data processing apparatus configured to perform the method of any one of claims 58 to 75.

77. A computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 58 to 75.

78. A computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 58 to 75.

79. A method for encoding, transmitting, and decoding lossy images or videos, the method comprising the following steps: Receive the first and second images of the image sequence at the first computer system; Optical flow information is generated using a first neural network with a first image and a second image, the optical flow information indicating the difference between the first image and the second image; At least partially masking optical flow information; The representation of the masked optical flow information is transmitted to the second computer system; A second neural network is used to decode the representation of the masked optical flow information to generate an approximation of the optical flow information. as well as An output image is generated using optical flow information through a third neural network, wherein the output image is an approximation of the first image.

80. The method of claim 79, wherein at least partially masking the optical flow information reduces the entropy of the distribution of the optical flow information.

81. The method of claim 80, wherein the masking comprises generating a mask based on a first image and / or a second image.

82. The method of claim 80 or 81, further comprising using a fourth neural network to generate a mask.

83. The method according to any one of claims 79 to 82, comprising entropy encoding masked optical flow information into a bit stream, wherein transmitting the masked optical flow information comprises transmitting the bit stream, and wherein the step of at least partially masking the optical flow information reduces the size of the bit stream after entropy encoding compared to a bit stream generated without the step of at least partially masking.

84. The method according to any one of claims 79 to 83, wherein at least partially masking the optical flow information includes at least partially masking a normalized representation of the optical flow information.

85. The method of claim 84, further comprising using one or more first-learned scaling and / or zero-point parameters to generate a normalized representation of the optical flow information.

86. The method of claim 85, further comprising encoding optical flow information using a fifth neural network to generate a hyperlatent representation of the optical flow information, and decoding the hyperlatent representation of the optical flow information using a sixth neural network to generate one or more first-learned scaling and / or zero-point parameters.

87. The method of claim 86, further comprising at least partially masking the hyperpotential representation of optical flow information to reduce the entropy of the distribution of the hyperpotential representation of optical flow information.

88. The method of claim 87, further comprising entropy encoding a masked hyperlatent representation of optical flow information and transmitting it to a second computer system.

89. The method of claim 87 or 88, wherein the hyperlatent representation that at least partially masks the optical flow information comprises a normalized representation of the hyperlatent representation that at least partially masks the optical flow information.

90. The method of claim 89, further comprising using one or more second-learned scaling and / or zero-point parameters to generate a normalized representation of the hyperlatent representation of optical flow information.

91. The method of claim 90, comprising encoding a hyper-latent representation of optical flow information using a seventh neural network to generate a hyper-hyper-latent representation of optical flow information, and decoding the hyper-hyper-latent representation of optical flow information using an eighth neural network to generate second learned scaling and / or zero-point parameters.

92. The method of claim 91, further comprising at least partially masking the hyper-hyper-latent representation of optical flow information to reduce the entropy of the distribution of the hyper-hyper-latent representation of optical flow information.

93. The method of claim 92, wherein the hyper-hyper-latent representation that at least partially masks the optical flow information comprises a normalized representation of the hyper-hyper-latent representation that at least partially masks the optical flow information.

94. The method according to claim 92 or 93, comprising entropy encoding of the masked ultra-hyper-latent representation of optical flow information and transmitting it to a second computer system.

95. A method for encoding and transmitting lossy images or videos, the method comprising the following steps: Receive the first and second images of the image sequence at the first computer system; Optical flow information is generated using a first neural network with a first image and a second image, the optical flow information indicating the difference between the first image and the second image; At least partially masking optical flow information; as well as The representation of the masked optical flow information is transmitted to the second computer system.

96. A method for receiving and decoding lossy images or videos, the method comprising the following steps: The representation of masked optical flow information received at the second computer system, the masked optical flow information being generated using a first neural network with a first image and a second image, and being generated by at least partially masking the optical flow information; A second neural network is used to decode the representation of the masked optical flow information to generate an approximation of the optical flow information. as well as An output image is generated using optical flow information through a third neural network, wherein the output image is an approximation of the first image.

97. A data processing apparatus configured to perform the method of any one of claims 79 to 96.

98. A computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 79 to 96.

99. A computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 79 to 96.

100. A method for encoding, transmitting, and decoding lossy images or videos, the method comprising the following steps: Receive the first and second images of the image sequence at the first computer system; Optical flow information is generated using a first neural network with a first image and a second image, the optical flow information indicating the difference between the first image and the second image; The optical flow information is transmitted to the second computer system; A second neural network is used to decode the representation of optical flow information to generate an approximation of optical flow information. The representation of the previously decoded second image is distorted using an approximation of optical flow information; Combine the distorted representation of the previously decoded second image with the representation of the first image; as well as An output image is generated using a third neural network with a combined, distorted representation of the previously decoded second image, wherein the output image is an approximation of the first image.

101. The method of claim 100, further comprising using a second neural network to generate a representation of the first image.

102. The method of claim 101, further comprising using a second neural network to generate a mask, and using the mask to combine a representation of a first image with a distorted representation of a previously decoded second image.

103. The method of claim 100, further comprising using a fourth neural network to generate a representation of the first image from the first image.

104. The method of claim 103, further comprising using a fourth neural network to generate a mask from a first image, and using the mask to combine a representation of the first image with a distorted representation of a previously decoded second image.

105. The method of claim 102 or 104, wherein the combination comprises performing a weighted sum of the representation of the first image and the representation of the previously decoded distorted second image using weights based on the values ​​of the mask.

106. The method according to any one of claims 100 to 105, wherein the combination reduces (i) the difference between the distorted representation of the previously decoded second image and (ii) the first image.

107. The method according to any one of claims 100 to 106, wherein the combination reduces the difference between (i) the output image generated by using the combined distorted representation of the previously decoded second image with a third neural network and (ii) the first image.

108. The method according to any one of claims 100 to 107, wherein the combination corrects artifacts in the representation of the previously decoded second image that are distorted.

109. A method for encoding and transmitting lossy images or videos, the method comprising the following steps: Receive the first and second images of the image sequence at the first computer system; Optical flow information is generated using a first neural network with a first image and a second image, the optical flow information indicating the difference between the first image and the second image; and The optical flow information is transmitted to the second computer system.

110. A method for receiving and decoding lossy images or videos, the method comprising the following steps: The optical flow information received at the second computer system is generated by the first neural network using a first image and a second image of an image sequence and indicates the difference between the first image and the second image. A second neural network is used to decode the representation of optical flow information to generate an approximation of optical flow information. The representation of the previously decoded second image is distorted using an approximation of optical flow information; Combine the distorted representation of the previously decoded second image with the representation of the first image; as well as An output image is generated using a third neural network with a combined, distorted representation of the previously decoded second image, wherein the output image is an approximation of the first image.

111. A method for training one or more neural networks for use in lossy image or video encoding, transmission, and decoding, the method comprising the steps of: Receive the first and second images of the image sequence at the first computer system; Optical flow information is generated using a first neural network with a first image and a second image, the optical flow information indicating the difference between the first image and the second image; The optical flow information is transmitted to the second computer system; A second neural network is used to decode the representation of optical flow information to generate an approximation of optical flow information. The representation of the previously decoded second image is distorted using an approximation of optical flow information; Combine the distorted representation of the previously decoded second image with the representation of the first image; An output image is generated using a third neural network with a combined, distorted representation of the previously decoded second image, wherein the output image is an approximation of the first image; The function is evaluated based on the difference between the first image and the output image; The parameters of the first, second, and / or third neural networks are updated based on the evaluated function; as well as Repeat the above steps using the first set of input images to generate the first, second, and / or third trained neural networks.

112. The method of claim 111, further comprising using a second neural network to generate a representation of the first image.

113. The method of claim 112, further comprising using a second neural network to generate a mask, and using the mask to combine a representation of a first image with a distorted representation of a previously decoded second image.

114. The method of claim 111, further comprising using a fourth neural network to generate a representation of the first image from the first image.

115. The method of claim 114, further comprising using a fourth neural network to generate a mask from the first image, and using the mask to combine a representation of the first image with a distorted representation of a previously decoded second image.

116. The method according to claim 112 or 114, wherein the function is further based on the difference between (i) the representation of the first image generated by the second neural network or the fourth neural network and (ii) the representation of the first image generated by the teacher neural network.

117. The method of claim 113 or 115, wherein the function is further based on the difference between (i) a mask generated by a second neural network or a fourth neural network and (ii) a mask generated by a teacher neural network.

118. The method of any one of claims 111 to 117, wherein evaluating the function comprises regularizing the propagation of gradients through one or more terms of the function.

119. The method of claim 118 when referring to claim 117, wherein evaluating the function comprises regularizing the propagation of gradients through the function associated with the difference between (i) a mask generated by a second neural network or a fourth neural network and (ii) a mask generated by a teacher neural network.

120. The method of claim 118 or 119, further comprising controlling the regularization based on a predetermined regularization schedule.

121. A data processing apparatus configured to perform the method of any one of claims 100 to 120.

122. A computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 100 to 120.

123. A computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 100 to 120.

Images