Super-resolution using time-space-frequency tokens

Abstract

A computing device includes a processor configured to receive input video data including a plurality of input images. Each of the input images may include a plurality of input pixels. For each input image, the processor may also be configured to perform up-sampling on the input image and divide the up-sampled input image into a respective plurality of tiles. For each tile, the processor may also be configured to generate a plurality of time-space frequency tokens. The plurality of time-space frequency tokens generated for the chunks may be indexed by a time step, a spatial location, and a frequency. At least in part at the trained machine learning model, the processor may be further configured to generate a plurality of super-resolution output images based at least in part on the temporal-spatial frequency tokens. The processor may also be configured to output a super-resolution output image.

Description

Background Art

[0001] Video data is often stored and transmitted in compressed form. Compressing video data reduces the amount of data included in the video, which can allow for easier storage or transmission of the video. However, when video data is compressed, the image quality may be significantly reduced. Compressing video data may accordingly result in a degraded user experience and the loss of image details (e.g., text or facial features) in the video that may be relevant to the user.

[0002] Video super-resolution (VSR) is the task of constructing a sequence of high-resolution (HR) frames of video data from a sequence of low-resolution (LR) frames. Thus, VSR can be used to recover HR video data from LR video data received in compressed form. By using VSR, it is possible to achieve a reduction in video file size by compression while at least partially avoiding degradation of image quality.

[0003] In other examples, VSR can also be used to enhance the image quality of uncompressed video. For example, VSR can be applied to uncompressed video data collected by a low-resolution camera to effectively increase the camera resolution. Therefore, using VSR on uncompressed data can allow the use of smaller, cheaper cameras while maintaining the quality of the collected video data. Summary of the invention

[0004] According to one aspect of the present disclosure, a computing device is provided, including a processor configured to receive input video data including multiple input images. Each of the multiple input images may include multiple input pixels. For each of the multiple input images, the processor may also be configured to perform upsampling on the input image and divide the upsampled input image into corresponding multiple blocks. For each of the multiple blocks, the processor may also be configured to generate multiple time-space frequency tokens. The multiple time-space frequency tokens generated for the blocks may be indexed by time step, spatial position and frequency. At least in part at a trained machine learning model, the processor may also be configured to generate multiple super-resolution output images based at least in part on the multiple time-space frequency tokens. The processor may also be configured to output multiple super-resolution output images.

[0005] This summary is provided to introduce a selection of concepts in simplified form, which will be further described in the following detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all of the disadvantages mentioned in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1An example computing device is schematically illustrated at which output video data including a plurality of super-resolved output images is generated from input video data according to an example embodiment.

[0007] Figure 2 Shown according to Figure 1 An example pre-processing phase of an example may be executed at a processor included in a computing device.

[0008] Figure 3 Schematically shows the Figure 2 Example of the generation of multiple spatio-temporal frequency tokens in the preprocessing stage.

[0009] Figure 4 Schematically shows the Figure 1 An example of an example trained machine learning model configured to be executed at a processor.

[0010] Figure 5 Schematically shows the Figure 1 An example post-processing stage of the example is configured to be executed at a processor.

[0011] Figure 6 Schematically shows the Figure 1 An example of a computing device during training of a trained machine learning model.

[0012] Fig. 7A Shown according to Figure 1 A flowchart of an example method for use with a computing device to generate a plurality of super-resolved output images.

[0013] FIG. 7B to FIG. 7E Additional steps of the method of FIG. 7 are shown that may be performed in some examples.

[0014] Figure 8 Shows where you can instantiate Figure 1 Schematic diagram of an example computing environment of a computing device. DETAILED DESCRIPTION

[0015] Existing methods of VSR generally fail to produce super-resolved video data that accurately matches the true appearance of the scene depicted in the input video data. For example, existing VSR methods often do not distinguish between compression artifacts and physical textures in the imaged scene. Therefore, such VSR methods may fail to achieve the desired improvement in image quality.

[0016] To address the above shortcomings of existing VSR methods, the apparatus and method discussed below are provided. The apparatus and method discussed below utilize a tokenization scheme in which frequency data as well as spatial and temporal data are encoded in tokens, which are used as input to a trained machine learning model. The trained machine learning model can be a transformer network, as discussed in further detail below. As used herein, "frequency" refers to the spatial frequency at which repetitive textures occur in an input image.

[0017] The devices and methods discussed below may also use a loop structure via which information about optical flow between frames of input video data may be propagated to frames of output video data. Information about the optical flow may be expressed in hidden states of a trained machine learning model. The loop structure may allow the trained machine learning model to utilize the optical flow data when generating a high-resolution image from a low-resolution image, wherein the optical flow data allows the trained machine learning model to more accurately predict pixels of the high-resolution image from the low-resolution image data.

[0018] Figure 1 An example computing device 10 according to one embodiment is schematically shown. The computing device 10 may include a processor 12 configured to execute instructions to perform a computing process. For example, the processor 12 may include one or more central processing units (CPUs), graphics processing units (GPUs), field programmable gate arrays (FPGAs), dedicated hardware accelerators, and / or other types of processing devices. The computing device 10 may also include a memory 14 communicatively coupled to the processor 12. The memory 14 may, for example, include one or more volatile memory devices and / or one or more non-volatile memory devices.

[0019] Other components may also be included in computing device 10, such as user input devices 16 and / or user output devices. One or more input devices 16 may, for example, include a keyboard, a mouse, a touch screen, a microphone, an accelerometer, an optical sensor, and / or other types of input devices. One or more output devices may include a display device 18, where output video data 60 generated at processor 12 may be displayed, as discussed in further detail below. One or more other types of output devices, such as speakers, may additionally or alternatively be included in computing device 10.

[0020] Computing device 10 may be instantiated in a single physical computing device or in multiple communicatively coupled physical computing devices. For example, computing device 10 may include a physical or virtual server computing device located at a data center. In an example where computing device 10 is a virtual server computing device, the functionality of processor 12 and / or memory 14 may be distributed between multiple physical computing devices. In some examples, computing device 10 may be instantiated at least in part at one or more client computing devices. In such an example, one or more client computing devices may be configured to communicate with one or more server computing devices over a network. For example, a client computing device may be configured to offload processing of input video data to a server computing device, where a trained machine learning model is executed. The client computing device may also be configured to receive output video data from a server computing device.

[0021] like Figure 1 As shown in the example of FIG. 1 , the processor 12 may be configured to receive input video data 20 including a plurality of input images 22. Each of the plurality of input images 22 may include a plurality of input pixels 24. The plurality of input images 22 may be indicated as The input images 22 each have a height H and a width W. The plurality of input images 22 is a sequence of T images. In some examples, the input images 22 may be compressed images generated from real video data. The real video data may be indicated as High-resolution video data.

[0022] The processor 12 may be configured to generate output video data 60 including a plurality of super-resolved output images 62. Each of the plurality of super-resolved output images 62 may include a plurality of output pixels 64. The plurality of super-resolved output images 62 may be indicated as Each of the super-resolved output images 62 may have a height αH and a width αW, where α represents an upsampling scale factor. The upsampling performed on the input image 22 is discussed in further detail below.

[0023] The processor 12 may also be configured to pre-process the plurality of input images 22 at a pre-processing stage 30. At the pre-processing stage 30, the processor 12 may be configured to generate an input to a trained machine learning model 40. The input generated at the pre-processing stage 30 may include a plurality of time-space frequency tokens τ (t,i,f) . Multiple time-space-frequency tokens τ (t,i,f)It can be indexed by time step, spatial position and frequency. In addition, in the pre-processing stage 30, the processor 12 can also be configured to calculate the corresponding deformed hidden state for each input image in the plurality of input images 22 except the first input image Deformation hidden state Optical flow data may be encoded on a plurality of input images 22 .

[0024] The processor 12 may also be configured to execute the trained machine learning model 40. In some examples, the trained machine learning model 40 may be a transformer network. In such an example, the plurality of time-space-frequency tokens τ received at the trained machine learning model 40 (t,i,f) A plurality of query tokens, a plurality of key tokens, and a plurality of value tokens may be included. In addition, as discussed in further detail below, the plurality of query tokens, the plurality of key tokens, and the plurality of value tokens may include a subset of tokens that is based at least in part on a plurality of deformed hidden states. The trained machine learning model 40 may be configured to perform inference on a plurality of query tokens, a plurality of key tokens, and a plurality of value tokens at one or more attention heads. Thus, the trained machine learning model 40 may be configured to generate a machine learning model output 42. The machine learning model output 42 may be the output of an attention head in one or more attention heads.

[0025] The processor 12 may also be configured to post-process the machine learning model output 42 at a post-processing stage 50. At the post-processing stage 50, the processor 12 may be configured to generate a plurality of super-resolved output images 62 based at least in part on the machine learning model output 42 of the trained machine learning model 40. In addition, the processor 12 may also be configured to generate a recurrent hidden state H at the post-processing stage 50. T . The recurrent hidden state H T can be used to generate corresponding deformed hidden states for use in processing one or more subsequent input images 22

[0026] Thus, the trained machine learning model 40 at the pre-processing stage 30, and the processor 12 at the post-processing stage 50, may be configured to generate output video data 60 from the input video data 20. The processor 12 may also be configured to output a plurality of super-resolved output images 62 included in the output video data 60, the super-resolved output images being used for display at the display device 18.

[0027] Figure 2Schematically, additional details of the pre-processing stage 30 are shown when the input image 22 is pre-processed. The pre-processing stage 30 may be performed for each input image 22 of the plurality of input images 22. At the pre-processing stage 30, the processor 12 may be configured to perform upsampling on the input image 22. Performing upsampling on the input image 22 may multiply both the height and width of the input image by an upsampling scale factor α, where α>1. Therefore, when a super-resolved output image 62 is generated from the input image 22, the resolution of the super-resolved output image 62 may be greater than the resolution of the input image 22.

[0028] In some examples, the processor 12 may be configured to perform upsampling on the input image 22 at least in part by performing bicubic interpolation 70 on the input image 22 to generate a first upsampled image 72. In addition, performing upsampling may also include performing an upsampling neural network. The input image 22 is processed at 74 to generate a second upsampled image 74. For example, the upsampling neural network Can be a Super Resolution Convolutional Neural Network (SRCNN) or BasicVSR network. Bicubic interpolation70 and upsampling neural network Both may be configured to scale the height and width of the input image 22 by the same upsampling scale factor α. Thus, both the first upsampled image 72 and the second upsampled image 74 may have a height αH and a width αW. Generating two different upsampled images from the input image 22 may allow the processor 12 to note the differences between the first upsampled image 72 and the second upsampled image 74 at the trained machine learning model 40. Noting the differences between the first upsampled image 72 and the second upsampled image 74 may allow the trained machine learning model 40 to more accurately super-resolve features of the upsampled image having a higher level of detail.

[0029] In the pre-processing stage 30, the processor 12 may also be configured to divide each of the upsampled input images into a corresponding plurality of partitions. Thus, the processor 12 may be configured to generate a plurality of first partitions 76 from the first upsampled image 72 and a plurality of second partitions 78 from the second upsampled image 74. Each of the first partitions 76 and the second partitions 78 may have a height of B input pixels 24 and a width of B input pixels 24.

[0030] During the pre-processing stage 30, the processor 12 may also be configured to generate a plurality of spectrograms D of a plurality of blocks. LR Spectrum D LRThe corresponding frequencies of the component textures included in the input image 22 may be encoded according to the spatial positions on the horizontal axis and the vertical axis on the input image 22. In an example where the processor 12 is configured to generate a plurality of first blocks 76 and a plurality of second blocks 78, the processor 12 may be configured to generate a first spectrum map D for each of the plurality of first blocks 76 and the plurality of second blocks 78. LR，1 and the second spectrum D LR,2 .

[0031] Spectrum D LR Each can be generated by performing a discrete cosine transform (DCT) on the corresponding block. The DCT can be configured to perform a projection operation of the image on a set of cosine components corresponding to a two-dimensional frequency. Using an upsampling neural network The generated spectrum D LR，1 So it can be expressed as: Spectrogram D generated using bicubic interpolation 70 LR,2 can be similarly generated via DCT. In the above equation, u∈[0, B-1] and v∈[0, B-1] are spatial indices of the two-dimensional frequency within the block. For a block P with B×B dimensions, the DCT function can be given as follows: In the above equation, x and y are the 2D indices of the pixels. c(·) is a normalization scale factor that enforces orthogonality, where: The spectrogram of the block P generated by the above equation for DCT may also have B×B dimensions.

[0032] The processor 12 may be configured to perform DCT on each of the plurality of first partitions 76 to generate a plurality of first spectrum graphs D LR,1 , and performing DCT on each of the plurality of second blocks 78 to generate a plurality of second spectrum graphs D LR,2 The first spectrogram D generated from the input video data 20 LR,1 The sequence and the second spectrum D LR,2 The sequences can each have Dimension, where F is the number of dimensions of the frequency space, C is the number of frequency bands, is the height of the spectrogram, and is the width of the spectrum. The number of frequency dimensions F can be given by F = B 2 Given.

[0033] The processor 12 may also be configured to convert the spectrogram D into LREach spectrum map in the first spectrum map is divided into a corresponding plurality of spatial frequency domain blocks. The plurality of spatial frequency domain blocks may include LR，1 The plurality of first spatial frequency domain blocks 80 generated, and the second spectrum map D LR，2 The generated plurality of second spatial-frequency domain blocks 82. The first spatial-frequency domain block 80 and the second spatial-frequency domain block 82 may each have a kernel size of K×K for their respective spatial dimensions.

[0034] The processor 12 may also be configured to: divide each of the spatial frequency domain blocks 80 and the spatial frequency domain blocks 82 into a plurality of temporal spatial frequency tokens τ (t，i，f) , to generate multiple time-space frequency tokens τ for each of the multiple blocks 76 and 78 (t，i，f) Processor 12 may be configured to divide spatial frequency domain blocks 80 and spatial frequency domain blocks 82 into temporal spatial frequency tokens τ according to spatial locations within blocks 76 and 78. (t，i，f) . Multiple time-space-frequency tokens τ (t，i，f) It can be indexed by time step, spatial position and frequency. Therefore, the time-space frequency tokens τ generated for multiple input images 22 are (t，i，f) The set of can be given by: T={τ (t，i，f) ,t∈[1,T],i∈[1,N],f∈[1,F]} In the above equation, N is the value for the spectrum graph D LR Each spectrum D in LR The number of blocks generated, and i is the index into N blocks. Therefore, i is the index of the spatial position of the token. Time Space Frequency Token τ (t，i，f) Each time-space frequency token in may have dimensions 1×1×C×K×K. The processor 12 may be configured to generate F time-space frequency tokens τ for each block (t，i，f) . Time-space-frequency token τ (t，i，f) The total number can be given by 2×T×F×N, where 2 is due to the use of two different upsampling techniques.

[0035] Figure 3 Description Figure 2 Tokenization of the first upsampled image 72 or the second upsampled image 74 of the example. Figure 3 As shown, the first upsampled image 72 or the second upsampled image 74 may be divided into a plurality of first partitions 76 or second partitions 78. Each of the first partitions 76 or second partitions 78 may then be divided into a respective set of first spatial frequency domain blocks 80 or second spatial frequency domain blocks 82. Those blocks may then also be divided into temporal spatial frequency tokens τ(ti，f) A collection of. Figure 3 In the example of , the corresponding dimensions of the upsampled image, tiles, blocks and time-space frequency tokens are also shown.

[0036] return Figure 2 The example preprocessing stage 30 shown, a plurality of time-space frequency tokens τ (t，i，f) Can include multiple query tokens Multiple key tokens and multiple value tokens In order to take into account the temporal information encoded across multiple input images 22, the set of query tokens Q can be obtained from the first spectrogram generated for the Tth input image 22 The set of value tokens K and the set of value tokens V can be extracted from the second spectrum graph Extract from, where t∈[1,T-1]. For the input image The resulting set of query, key, and value tokens can be given accordingly: Time-space-frequency token τ (t，，i，f) From the spectrum diagram D LR is extracted along the time, space and frequency dimensions and may be generated for each of the input images 22. As discussed in further detail below, Figure 2 The key token shown in and value token There may be a plurality of first key tokens and a plurality of first value tokens, respectively, which are configured to be used as inputs to a first attention head of the trained machine learning model 40.

[0037] Figure 2 The example preprocessing stage 30 also shows the previous input image and the recurrent hidden state H T-1 . The previous input image It can be immediately after the input image The input image 22 before, the multiple time-space frequency tokens τ for it (t，，i，f) The generation of Figure 2 The cyclic hidden state H T-1 It can be carried over from the processing of the previous input image T-1 to the processing of the current input image T.

[0038] In the pre-processing stage 30, the processor 12 may also be configured to calculate a plurality of optical flow maps between consecutive pairs 22 of the input images. T, consecutive pairs 22 of input images are included in the input video data 20. Figure 2 The input image is shown With the previous input image The optical flow graph O T Calculation of optical flow graph O T It can be a vector field, where the corresponding multiple optical flow vectors are related to the input image associated with the location.

[0039] For each input image in the plurality of input images 22 except the first input image 22, the processor 12 may also be configured to calculate a corresponding deformed hidden state The processor 12 may be configured to at least partially transform the optical flow map O T Applied to the corresponding recurrent hidden state H T-1 To calculate the deformed hidden state The loop hides the state H T-1 is generated for a previous input image 22 among the multiple input images 22. T Applied to the recurrent hidden state H T-1 It can include the optical flow map O T The magnitude and direction of the vector indicated in the loop are included in the hidden state H T-1 Deformation of features in .

[0040] Figure 4 A trained machine learning model 40 according to one example is shown. Figure 4 In the example of , the trained machine learning model 40 is a transformer network. Figure 4 As shown in the example of , the trained machine learning model can be configured to receive multiple query tokens Multiple First Key Tokens and multiple first value tokens As input. In addition, the trained machine learning model 40 can be configured to receive the deformation hidden state

[0041] The trained machine learning model 40 may include a spatial frequency attention head 90. The spatial frequency attention head 90 may be configured to receive a plurality of query tokens Multiple First Key Tokens and multiple first value tokens Query token used as input at spatial frequency attention head 90 may be a query token for the current time step T in the input image sequence 22 .

[0042] At the spatial frequency attention head 90, the processor 12 may be configured to calculate a plurality of spatial frequency attention weights 92 between corresponding spatial frequency domain blocks generated for an input image 22 of the plurality of input images 22. Thus, the spatial frequency attention weights 92 may indicate the degree to which the spatial frequency attention head 90 pays attention to different regions of the input space, the input space being parameterized according to spatial position and frequency. The spatial frequency attention head 90 may be configured to calculate the spatial frequency attention head output R T .

[0043] The trained machine learning model 40 may also include a time-frequency attention head 94. The time-frequency attention head 94 may be configured to receive a plurality of query tokens Multiple Second Key Tokens and multiple second value tokens The query token used as input at the time-frequency attention head 94 may be a query token generated for an earlier time step t∈[1, T-1] in the input image sequence 22. The processor 12 may be configured to generate a query token at least in part by transforming the deformed hidden state Add to the spatial frequency attention head output R T To generate multiple second key tokens and multiple second value tokens

[0044] At the temporal frequency attention head, the processor 12 may be configured to calculate a plurality of temporal frequency attention weights 96 between corresponding spatial frequency domain blocks generated for shared spatial locations in consecutive input images 22 of the plurality of input images 22. Thus, the temporal frequency attention weights 96 may indicate the degree to which the temporal frequency attention head 94 pays attention to frequency features occurring across a plurality of time steps in the input image sequence 22. The temporal frequency attention head 94 may be configured to generate a temporal frequency attention head output P T , the time-frequency attention head output P T may be transmitted to a post-processing stage 50 .

[0045] like Figure 4 As shown, when spatial frequency attention and temporal frequency attention are combined, the temporal frequency attention head outputs P T It can be the joint time-space-frequency attention output, which can be expressed as follows: The temporal spatial frequency attention can be calculated according to the following equation: In the above equation, d kis the dimension of each key vector in the key vectors, and SM is the softmax function.

[0046] The processor 12 may also be configured to generate a plurality of super-resolved output images 62 based at least in part on the plurality of spatial frequency attention weights 92 and the plurality of temporal frequency attention weights 96, at least in part on the trained machine learning model 40. Figure 4 In the example of FIG. 1 , the temporal frequency attention head 94 is downstream of the spatial frequency attention head 90, but in other examples, the order of the temporal frequency attention head 94 and the spatial frequency attention head 90 may be reversed. In such an example, the spatial frequency attention head 90 may be configured to receive the temporal frequency attention head output P T Furthermore, in such an example, the spatial frequency attention head outputs R T It can be time-space-frequency attention Λ T×S .

[0047] like Figure 4 As shown in the example of T×S Compared with the calculation of the time-space-frequency attention at a single attention head, the calculation of the time-space-frequency attention Λ T×S Therefore, dividing the computation of the temporal spatial frequency attention between the spatial frequency attention head 90 and the temporal frequency attention head 94 can reduce the computation time of the temporal spatial frequency attention Λ T×S In addition, using the spatial frequency attention head 90 and the temporal frequency attention head 94 can allow the spectrogram D LR The feature size of is kept consistent between training and inference at the Transformer network.

[0048] Figure 5 The post-processing stage 50 is schematically shown as being configured to be performed downstream of the trained machine learning model 40. The post-processing stage 50 is shown when a super-resolved output image 62 included in the output video data 60 is generated. The post-processing stage 50 may be configured to Figure 4 The trained machine learning model 40 depicted in the example of FIG. 4 receives the time-frequency attention head output P T In examples where the spatial frequency attention head 90 is located downstream of the temporal frequency attention head 94, the post-processing stage 50 may alternatively be configured to receive the spatial frequency attention head output R T Additionally, the post-processing stage 50 may be configured to receive the deformed hidden state The post-processing stage 50 may also be configured to receive The first spectrum D generatedLR,1 and the second spectrum D LR,2 .

[0049] In the post-processing stage 50, the processor 12 may be configured to output the time-frequency attention head P T The input image corresponding to and The associated multiple first spectrograms D LR,1 and a plurality of second spectrograms D LR,2 The processor 12 may also be configured to add the spliced ​​result to the plurality of first spectrograms D LR，1 and a plurality of second spectrograms D LR,2 The processor 12 may also be configured to perform an inverse DCT (rDCT) on the result of the above addition. The output of the rDCT may be a super-resolved output image 62. Thus, at least in part at the trained machine learning model 40 and post-processing stage 50, the processor 12 may be configured to perform a super-resolved output image 62 based at least in part on the plurality of time-space frequency tokens τ (t，i，f) and the deformed hidden state To generate multiple super-resolution output images 62.

[0050] In the post-processing stage 50, the processor 12 may also be configured to prepare the recurrent hidden state H for the time step T T , so that the loop hidden state H T It can be used in the subsequent time step T+1. Prepare the recurrent hidden state H T This can include outputting the time-frequency attention head P T , the first spectrum D generated for the time step T LR，1 and the second spectrogram D generated for time step T LR，2 Processor 12 may also be configured to append the result of the splicing to the deformed hidden state And the spatial frequency attention head output R T to obtain the recurrent hidden state H T .

[0051] Figure 6 The computing device 10 is schematically shown when the machine learning model 140 is trained at the processor 12 to generate the trained machine learning model 40. Figure 6 In the example of , the machine learning model 140 is trained at the computing device 10, but in other examples, the machine learning model 140 may be trained at some other computing device and stored in the memory 14 of the computing device 10.

[0052] like Figure 6As shown, as a preliminary step for training the machine learning model 140, the processor 12 may be configured to compress the uncompressed training video data 100 including a plurality of uncompressed training images 102 so as to generate compressed training video data 110 including a plurality of compressed training images 112. The processor 12 may be configured to use the uncompressed training video data 100 as real data when training the machine learning model 140. The uncompressed training video data 100 and the compressed training video data 110 may be organized into a plurality of uncompressed training videos and a corresponding plurality of uncompressed training videos, each of which includes a corresponding plurality of images. When a plurality of compressed training videos are generated, a plurality of different compression rates may be used.

[0053] The processor 12 may also be configured to pass the compressed training video data 110 through the pre-processing stage 30, the machine learning model 140, and the post-processing stage 50 to generate candidate super-resolved video data 120. The candidate super-resolved video data 120 may include a plurality of candidate super-resolved images 122.

[0054] The processor 12 may also be configured to input the uncompressed training video data 100 and the candidate super-resolution video data 120 into a loss function 130, in which the processor 12 may be configured to calculate the loss of the machine learning model 140. For example, the following Charbonnier penalty loss function may be used: In the above equation, T is the number of frames in the uncompressed training video, is the uncompressed training image 102 at the t-th frame, is the candidate super-resolved image 122 at the t-th frame, and ∈ is a constant value. For example, in some examples ∈=10 -3 Alternatively, some other value of ∈ may be used. In other examples, some other type of loss function may be used when training the machine learning model 140.

[0055] Based at least in part on the value of the loss function 130, the processor 12 may also be configured to update the parameters of the machine learning model 140. For example, the processor 12 may be configured to update the parameters via stochastic gradient descent. The parameters of the machine learning model 140 may be updated in multiple training iterations. Thus, the trained machine learning model 40 may be generated by training the machine learning model 140.

[0056] Fig. 7AA flow chart of an example method 200 for use with a computing device to generate a plurality of super-resolved output images is shown. At step 202, the method 200 may include receiving input video data including a plurality of input images. Each of the plurality of input images may include a plurality of input pixels.

[0057] Step 204, step 206 and step 208 of method 200 can each be performed in a preprocessing stage for each input image in a plurality of input images. At step 204, method 200 may also include performing upsampling on the input image. When upsampling is performed on the input image, both the height and width of the input image may be increased by the upsampling scaling factor. At step 206, method 200 may also include dividing the upsampled input image into a corresponding plurality of blocks. At step 208, method 200 may also include generating a plurality of time-space frequency tokens for each of the plurality of blocks. The plurality of time-space frequency tokens generated for the blocks may be indexed by a time step, a spatial position and a frequency. The frequency of the indexed time-space frequency token is the spatial frequency of the repeated texture features within the input image.

[0058] At step 210, method 200 may also include: generating multiple super-resolution output images at least in part at the trained machine learning model, at least in part based on multiple time-space frequency tokens. The trained machine learning model can be a transformer network. In addition to the trained machine learning model, multiple super-resolution output images can be generated at least in part in a post-processing stage, and the post-processing stage is performed subsequent to the trained machine learning model. The post-processing stage can be configured to further process the machine learning model output of the trained machine learning model to generate multiple super-resolution output images. For example, in an example where the trained machine learning model is a transformer model, the machine learning model output can include multiple attention head outputs.

[0059] At step 212, method 200 may also include outputting a plurality of super-resolution output images. The plurality of super-resolution output images may be output to a display device for display as a super-resolution video. In some examples, the plurality of super-resolution output images may be output to a separate computing device. For example, the plurality of super-resolution output images may be generated at a server computing device, and the plurality of super-resolution output images may be output for display at a client computing device.

[0060] Figure 7BAdditional steps of the method 200 of FIG. 7 that may be performed in some examples are shown. At step 204A, performing upsampling the input image at step 204 may include performing bicubic interpolation on the input image to generate a first upsampled image. Additionally, at step 204B, performing upsampling at step 204 may also include processing the input image at an upsampling neural network to generate a second upsampled image.

[0061] At step 206A, generating the plurality of tiles at step 206 may also include dividing the first upsampled image and the second upsampled image into a plurality of first tiles and a plurality of second tiles, respectively. The plurality of first tiles and the plurality of second tiles may be further processed during a preprocessing stage to generate a separate set of time-space-frequency tokens. When generating the time-space-frequency tokens, using two different upsampled images may allow the trained machine learning model to more accurately reconstruct small-scale textures by noting the differences between the upsampled images.

[0062] Figure 7C Additional steps of method 200 that may be performed in some examples are shown. At step 208A, generating a plurality of time-space frequency tokens at step 208 may include generating a plurality of spectrograms of a plurality of blocks. Figure 7B As shown, in an example in which two different upsampling techniques are used, step 208A may also include: at step 208B, performing DCT on each of the plurality of first blocks to generate a plurality of first spectrograms. In addition, step 208A may also include: at step 208C, performing DCT on each of the plurality of second blocks to generate a plurality of second spectrograms.

[0063] In step 208D, generating a plurality of time-space frequency tokens may further include dividing each of the spectrograms into a corresponding plurality of space-frequency domain blocks. In an example where two different upsampling techniques are used to generate a separate set of spectrograms, the plurality of first spectrograms and the plurality of second spectrograms may be divided into a first plurality of space-frequency domain blocks and a second plurality of space-frequency domain blocks, respectively.

[0064] Step 208 may also include, at step 208E, dividing each of the spatial frequency domain blocks into a plurality of time-space frequency tokens according to the spatial position. In an example in which the trained machine learning model is a transformer model, the plurality of time-space frequency tokens may include a plurality of query tokens generated from a plurality of first spatial frequency domain blocks. The plurality of time-space frequency tokens may also include a plurality of key tokens, and a plurality of value tokens generated from a plurality of second spatial frequency domain blocks.

[0065] In examples where DCT is performed during a pre-processing stage, as shown in steps 208B and 208C, method 200 may also include performing an inverse DCT (rDCT) in a post-processing stage downstream of the trained machine learning model. For example, the rDCT may be performed on an intermediate post-processing result rather than on the direct output of the trained machine learning model. In some examples, the rDCT may be the final post-processing step at which multiple super-resolved output images are generated.

[0066] Fig.7D Additional steps of method 200 are shown that can be performed in an example in which the trained machine learning model is a transformer network. At step 210A, executing the trained machine learning model at step 210 may include calculating multiple spatial frequency attention weights between corresponding spatial frequency domain blocks, and the corresponding spatial frequency domain blocks are generated for input images in multiple input images. Step 210A can be performed at a spatial frequency attention head included in the trained machine learning model. The spatial frequency attention head can be configured to receive a plurality of query tokens, a plurality of first key tokens, and a plurality of first value tokens as input, and the plurality of query tokens, the plurality of first key tokens, and the plurality of first value tokens are included in the plurality of temporal spatial frequency tokens generated in the preprocessing stage. The query tokens used as input to the spatial frequency attention head may include: a subset of the query tokens generated for the current frame of the input video data.

[0067] At step 210B, executing the trained machine learning model may also include: calculating multiple time-frequency attention weights between corresponding spatial frequency domain blocks, and the corresponding spatial frequency domain blocks are generated for shared spatial positions of consecutive input images in multiple input images. The time-frequency attention weights can be calculated at the time-frequency attention head included in the trained machine learning model. The time-frequency attention head can be configured to receive multiple query tokens, multiple second key tokens, and multiple second value tokens included in multiple time-space frequency tokens as input. The query token used as input to the time-frequency attention head may include: a subset of query tokens generated for previous frames of input video data. Multiple second key tokens and multiple second value tokens can be calculated based at least in part on the output of the spatial frequency attention head. In other examples, the order of the spatial frequency attention head and the time-frequency attention head can be reversed.

[0068] At step 210C, step 210 may also include: generating multiple super-resolved output images, at least in part at the trained machine learning model, based at least in part on multiple spatial frequency attention weights and multiple temporal frequency attention weights. As described above, when generating multiple super-resolved output images, post-processing may also be applied to the output of the trained machine learning model.

[0069] Fig. 7E Additional steps of method 200 by which a recurring structure may be implemented are shown. At step 216, method 200 may also include calculating a plurality of optical flow maps between consecutive pairs of input images included in the input video data. At step 218, method 200 may also include calculating a corresponding deformed hidden state for each input image in the plurality of input images other than the first input image. The deformed hidden state may be generated at least in part by applying an optical flow map in the plurality of optical flow maps to a corresponding recurring hidden state, the corresponding recurring hidden state being generated for a previous input image in the plurality of input images. Steps 216 and 218 may be performed during a preprocessing stage.

[0070] At step 210D, executing the trained machine learning model at step 210 may also include generating, at least in part at the trained machine learning model, a plurality of super-resolved output images based at least in part on the plurality of deformed hidden states. In some examples, the plurality of deformed hidden states may be used to generate a plurality of second key tokens and a plurality of second value tokens, the plurality of second key tokens and the plurality of second value tokens being received as inputs to the time-frequency attention head.

[0071] At step 220, method 200 may also include calculating a cyclic hidden state for each of the plurality of input images. In some examples, calculating the cyclic hidden state may include adding the spatial frequency attention head output to the deformed hidden state. The sum of the spatial frequency attention head outputs may be appended to the concatenation of the temporal frequency attention head outputs, the first spectrogram, and the second spectrogram to obtain a cyclic hidden state for the current time step. The cyclic hidden state may be utilized at a trained machine learning model at a subsequent time step.

[0072] Using the apparatus and methods discussed above, super-resolution may be performed on input video data. VSR performed as discussed above may have higher image quality when super-resolution is performed on small-scale features of an input image compared to previous VSR methods. Thus, when the apparatus and methods discussed above are applied to compressed or uncompressed video data, an improved viewer experience may be allowed.

[0073] In some embodiments, the methods and processes described herein may be bound to a computing system of one or more computing devices. In particular, such methods and processes may be implemented as computer applications or services, application programming interfaces (APIs), libraries, and / or other computer program products.

[0074] Figure 8A non-limiting embodiment of a computing system 300 that can perform one or more of the methods and processes described above is schematically illustrated. The computing system 300 is shown in simplified form. The computing system 300 can embody the above and Figure 1 Components of computing system 300 may be instantiated in one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, video gaming devices, mobile computing devices, mobile communication devices (e.g., smart phones), and / or other computing devices, as well as wearable computing devices (such as smart watches and head-mounted augmented reality devices).

[0075] The computing system 300 includes a logic processor 302, a volatile memory 304, and a non-volatile storage device 306. The computing system 300 may optionally include a display subsystem 308, an input subsystem 310, a communication subsystem 312, and / or Figure 3 Other components not shown.

[0076] Logical processor 302 includes one or more physical devices configured to execute instructions. For example, a logical processor may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform tasks, implement data types, transform the state of one or more components, implement technical effects, or otherwise achieve a desired result.

[0077] The logical processor may include one or more physical processors (hardware) configured to execute software instructions. Additionally or alternatively, the logical processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. The processor of the logical processor 302 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel and / or distributed processing. The various components of the logical processor may optionally be distributed between two or more separate devices, which are remotely located and / or configured for coordinated processing. Various aspects of the logical processor may be virtualized and executed by a remotely accessible networked computing device configured in a cloud computing configuration. It should be understood that in this case, these virtualized aspects run on different physical logical processors of various different machines.

[0078] The non-volatile storage device 306 includes one or more physical devices that are configured to store instructions that can be executed by a logical processor to implement the methods and processes described herein. When implementing such methods and processes, the state of the non-volatile storage device 306 can be transformed, for example, to store different data.

[0079] The non-volatile storage device 306 may include a removable and / or built-in physical device. The non-volatile storage device 306 may include an optical storage device (e.g., CD, DVD, HD-DVD, Blu-ray disc, etc.), a semiconductor memory (e.g., ROM, EPROM, EEPROM, flash memory, etc.), and / or a magnetic storage device (e.g., a hard disk drive, a floppy disk drive, a tape drive, MRAM, etc.) or other mass storage device technology. The non-volatile storage device 306 may include a non-volatile, dynamic, static, read / write, read-only, sequential access, location addressable, file addressable, and / or content addressable device. It should be understood that the non-volatile storage device 306 is configured to save instructions even when power is removed from the non-volatile storage device 306.

[0080] Volatile memory 304 may include physical devices including random access memory. Volatile memory 304 is typically utilized by logical processor 302 to temporarily store information during the processing of software instructions. It should be understood that when power to volatile memory 304 is cut off, volatile memory 304 typically does not continue to store instructions.

[0081] Aspects of the logic processor 302, volatile memory 304, and non-volatile storage device 306 may be integrated together into one or more hardware logic components. For example, such hardware logic components may include field programmable gate arrays (FPGAs), programmable and application specific integrated circuits (PASIC / ASIC), program and application specific standard products (PSSP / ASSP), systems on chips (SOCs), and complex programmable logic devices (CPLDs).

[0082] The terms "module," "program," and "engine" may be used to describe aspects of a computing system 300 that is typically implemented in software by a processor to perform a specific function using portions of volatile memory that involves transform processing that specifically configures the processor to perform the function. Thus, a module, program, or engine may be instantiated via a logical processor 302 executing instructions held by a non-volatile storage device 306 using portions of volatile memory 304. It should be understood that different modules, programs, and / or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same module, program, and / or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module," "program," and "engine" may encompass individual or groups of executable files, data files, libraries, drivers, scripts, database records, and the like.

[0083] When included, the display subsystem 308 can be used to represent a visual representation of the data stored by the non-volatile storage device 306. The visual representation can take the form of a graphical user interface (GUI). As the methods and processes described herein change the data held by the non-volatile storage device, and thus transform the state of the non-volatile storage device, the state of the display subsystem 308 can also be transformed to visually represent the changes in the underlying data. The display subsystem 308 may include one or more display devices utilizing almost any type of technology. Such a display device can be combined with the logical processor 302, the volatile memory 304, and / or the non-volatile storage device 306 in a shared housing, or such a display device can be a peripheral display device.

[0084] When included, the input subsystem 310 may include or interface with one or more user input devices, such as a keyboard, mouse, touch screen, or game controller. In some embodiments, the input subsystem may include or interface with selected natural user input (NUI) components. Such components may be integrated or peripheral, and the conversion and / or processing of input actions may be handled on-board or off-board. Example NUI components may include microphones for speech and / or voice recognition; infrared, color, stereo, and / or depth cameras for machine vision and / or gesture recognition; head trackers, eye trackers, accelerometers, and / or gyroscopes for motion detection and / or intent recognition; and electric field sensing components for assessing brain activity; and / or any other suitable sensors.

[0085] When included, the communication subsystem 312 can be configured to communicatively couple the various computing devices described herein to each other and to communicatively couple with other devices. The communication subsystem 312 may include wired and / or wireless communication devices compatible with one or more different communication protocols. As a non-limiting example, the communication subsystem can be configured to communicate via a wireless telephone network, or a wired or wireless local area network or a wide area network (such as an HDMI connected via Wi-Fi). In some embodiments, the communication subsystem can allow the computing system 300 to send messages to other devices and / or receive messages from other devices via a network such as the Internet.

[0086] The following paragraphs discuss several aspects of the present disclosure. According to one aspect of the present disclosure, a computing device is provided, including a processor configured to receive input video data including multiple input images. Each of the multiple input images may include multiple input pixels. For each of the multiple input images, the processor may also be configured to perform upsampling on the input image and divide the upsampled input image into corresponding multiple blocks. For each of the multiple blocks, the processor may also be configured to generate multiple time-space frequency tokens. The multiple time-space frequency tokens generated for the blocks may be indexed by time step, spatial position, and frequency. At least in part at a trained machine learning model, the processor may also be configured to generate multiple super-resolution output images based at least in part on the multiple time-space frequency tokens. The processor may also be configured to output multiple super-resolution output images.

[0087] According to this aspect, the processor may also be configured to perform upsampling on the input image at least in part by performing bicubic interpolation on the input image to generate a first upsampled image. Performing upsampling may also include processing the input image at an upsampling neural network to generate a second upsampled image. The processor may also be configured to divide the first upsampled image and the second upsampled image into a plurality of first blocks and a plurality of second blocks, respectively.

[0088] According to this aspect, the processor is configured to generate a plurality of time-space frequency tokens at least in part by generating a plurality of block-wise multiple spectrograms. Generating a plurality of time-space frequency tokens may also include dividing each of the spectrograms in the spectrograms into a corresponding plurality of space-frequency domain blocks. Generating a plurality of time-space frequency tokens may also include dividing each of the space-frequency domain blocks in the space-frequency domain blocks into a plurality of time-space frequency tokens according to spatial position.

[0089] According to this aspect, the processor can be configured to generate the plurality of spectrograms at least in part by performing a discrete cosine transform (DCT) on each of the plurality of first partitions to generate the plurality of first spectrograms. Generating the plurality of spectrograms can also include performing a DCT on each of the plurality of second partitions to generate the plurality of second spectrograms.

[0090] According to this aspect, the trained machine learning model can be a transformer network.

[0091] According to this aspect, the trained machine learning model may include a spatial frequency attention head, at which the processor may be configured to calculate multiple spatial frequency attention weights between corresponding spatial frequency domain blocks generated for an input image in a plurality of input images. The trained machine learning model may also include a temporal frequency attention head, at which the processor may be configured to calculate multiple temporal frequency attention weights between corresponding spatial frequency domain blocks generated for shared spatial positions in consecutive input images in the plurality of input images. At least in part at the trained machine learning model, the processor may also be configured to generate multiple super-resolved output images based at least in part on the multiple spatial frequency attention weights and the multiple temporal frequency attention weights.

[0092] According to this aspect, the spatial frequency attention head can be configured to receive as input a plurality of query tokens, a plurality of first key tokens, and a plurality of first value tokens included in a plurality of temporal spatial frequency tokens. The processor can be configured to generate the plurality of query tokens based at least in part on a plurality of first spatial frequency domain blocks generated from a first spectrogram. The processor can also be configured to generate a plurality of first key tokens and a plurality of first value tokens based at least in part on a plurality of second spatial frequency domain blocks generated from a second spectrogram.

[0093] According to this aspect, the time-frequency attention head can be configured to receive as input a plurality of query tokens, a plurality of second key tokens, and a plurality of second value tokens included in a plurality of time-space frequency tokens. The processor can be configured to generate a plurality of second key tokens and a plurality of second value tokens based at least in part on an output of the space-frequency attention head and one or more recurrent hidden states.

[0094] According to this aspect, the processor may be further configured to generate a plurality of super-resolved output images at least in part by performing an inverse DCT (rDCT) in a post-processing stage executed downstream of the trained machine learning model.

[0095] According to this aspect, for each input image of the plurality of input images, the processor may be further configured to output a corresponding recurrent hidden state.

[0096] According to this aspect, the processor may also be configured to calculate a plurality of optical flow maps between consecutive pairs of input images included in the input video data. For each input image other than the first input image in the plurality of input images, the processor may also be configured to calculate a corresponding deformed hidden state at least in part by applying an optical flow map in the plurality of optical flow maps to a corresponding recurrent hidden state, the corresponding recurrent hidden state being generated for a previous input image in the plurality of input images. At the trained machine learning model, the processor may also be configured to generate a plurality of super-resolved output images based at least in part on the plurality of deformed hidden states.

[0097] According to another aspect of the present disclosure, a method for use with a computing device is provided. The method may include receiving input video data including a plurality of input images. Each of the plurality of input images may include a plurality of input pixels. For each of the plurality of input images, the method may also include upsampling the input image and dividing the upsampled input image into a corresponding plurality of blocks. For each of the plurality of blocks, the method may also include generating a plurality of time-space frequency tokens. The plurality of time-space frequency tokens generated for the blocks may be indexed by time step, spatial position, and frequency. At least in part at a trained machine learning model, the method may also include generating a plurality of super-resolution output images based at least in part on the plurality of time-space frequency tokens. The method may also include outputting a plurality of super-resolution output images.

[0098] According to this aspect, performing upsampling on the input image may include performing bicubic interpolation on the input image to generate a first upsampled image, and processing the input image at an upsampling neural network to generate a second upsampled image. The method may also include dividing the first upsampled image and the second upsampled image into a plurality of first blocks and a plurality of second blocks, respectively.

[0099] According to this aspect, generating multiple time-space frequency tokens may include generating multiple block-by-block multiple spectrograms. Generating multiple time-space frequency tokens may also include dividing each spectrogram in the spectrogram into a corresponding multiple space-frequency domain blocks. Generating multiple time-space frequency tokens may also include dividing each space-frequency domain block in the space-frequency domain block into multiple time-space frequency tokens according to spatial position.

[0100] According to this aspect, generating a plurality of spectrograms may include performing a discrete cosine transform (DCT) on each of the plurality of first blocks to generate a plurality of first spectrograms. Generating a plurality of spectrograms may also include performing a DCT on each of the plurality of second blocks to generate a plurality of second spectrograms.

[0101] According to this aspect, the trained machine learning model can be a transformer network.

[0102] According to this aspect, at a spatial frequency attention head included in the trained machine learning model, the method may also include calculating a plurality of spatial frequency attention weights between corresponding spatial frequency domain blocks generated for an input image in a plurality of input images. At a temporal frequency attention head included in the trained machine learning model, the method may also include calculating a plurality of temporal frequency attention weights between corresponding spatial frequency domain blocks generated for shared spatial positions in consecutive input images in a plurality of input images. At least in part at the trained machine learning model, the method may also include generating a plurality of super-resolved output images based at least in part on the plurality of spatial frequency attention weights and the plurality of temporal frequency attention weights.

[0103] According to this aspect, the method may also include calculating a plurality of optical flow maps between consecutive pairs of input images contained in the input video data. For each input image in the plurality of input images other than the first input image, the method may also include calculating a corresponding deformed hidden state at least in part by applying an optical flow map in the plurality of optical flow maps to a corresponding recurrent hidden state generated for a previous input image in the plurality of input images. At least in part at the trained machine learning model, the method may also include generating a plurality of super-resolved output images based at least in part on the plurality of deformed hidden states. The method may also include calculating a recurrent hidden state for each input image in the plurality of input images.

[0104] According to another aspect of the present disclosure, a computing device is provided, including a processor configured to receive input video data including a plurality of input images. Each of the plurality of input images may include a plurality of input pixels. For each of the plurality of input images, the processor may also be configured to perform upsampling on the input image and divide the upsampled input image into a corresponding plurality of partitions. For each of the plurality of partitions, the processor may also be configured to generate a plurality of time-space frequency tokens. The plurality of time-space frequency tokens generated for the partitions are indexed by time steps, spatial positions, and frequencies. The plurality of time-space frequency tokens may include a plurality of query tokens, a plurality of first key tokens, and a plurality of first value tokens. The processor may also be configured to calculate a plurality of optical flow maps between consecutive pairs of input images included in the input video data. For each of the plurality of input images other than the first input image, the processor may also be configured to calculate a corresponding deformation hidden state at least in part by applying an optical flow map in the plurality of optical flow maps to a cyclic hidden state generated for a previous input image in the plurality of input images. The processor may also be configured to calculate a plurality of second key tokens and a plurality of second value tokens based at least in part on the deformation hidden state. At least in part at the transformer network, the processor may also be configured to generate a plurality of super-resolved output images based at least in part on a plurality of query tokens, a plurality of first key tokens, a plurality of first value tokens, a plurality of second key tokens, and a plurality of second value tokens generated for a plurality of input images, respectively. The processor may also be configured to output the plurality of super-resolved output images.

[0105] According to this aspect, the transformer network may include a spatial frequency attention head, at which the processor may be configured to receive multiple query tokens, multiple first key tokens, and multiple first value tokens. At the spatial frequency attention head, the processor may also be configured to generate a spatial frequency attention head output. At the transformer network, the processor may also be configured to generate multiple second key tokens and multiple second value tokens based at least in part on the spatial frequency attention head output and the deformed hidden state. The transformer network may also include a temporal frequency attention head, at which the processor may also be configured to receive multiple query tokens, multiple second key tokens, and multiple second value tokens. At the temporal frequency attention head, the processor may also be configured to generate a temporal frequency attention head output. The processor may also be configured to generate multiple super-resolved images based at least in part on the temporal frequency attention head outputs generated for multiple input images respectively.

[0106] It should be understood that the configuration and / or method described herein are exemplary in nature, and these specific embodiments or examples should not be considered restrictive, because many variations are possible. The specific routine or method described herein can represent one or more of any number of processing strategies. Therefore, the various actions shown and / or described can be performed in the order shown and / or described, in other orders, in parallel, or omitted. Equally, the order of the above process can be changed.

[0107] The subject matter of the present disclosure includes all novel and nonobvious combinations and subcombinations of the various processes, systems and configurations, and other features, functions, acts and / or properties disclosed herein, as well as any and all equivalents thereof.

Claims

1. A computing device, comprising: The processor is configured as: receiving input video data comprising a plurality of input images, wherein each input image of the plurality of input images comprises a plurality of input pixels; as well as For each input image of the plurality of input images: Performing upsampling on the input image; Dividing the upsampled input image into a corresponding plurality of blocks; as well as For each of the plurality of partitions, generating a plurality of time-space-frequency tokens, wherein the plurality of time-space-frequency tokens generated for the partition are indexed by time step, spatial position, and frequency; generating, at least in part at the trained machine learning model, a plurality of super-resolved output images based at least in part on the plurality of time-space-frequency tokens; as well as The plurality of super-resolution output images are output.

2. The computing device of claim 1 , wherein the processor is configured to: The upsampling of the input image is performed at least in part by: performing bicubic interpolation on the input image to generate a first upsampled image; processing the input image at an upsampling neural network to generate a second upsampled image; as well as The first upsampled image and the second upsampled image are divided into a plurality of first blocks and a plurality of second blocks, respectively.

3. The computing device of claim 2, wherein the processor is configured to generate the plurality of time-space-frequency tokens at least in part by: generating a plurality of spectrograms of the plurality of blocks; dividing each of the spectrograms into a corresponding plurality of spatial frequency domain blocks; Each of the space-frequency domain blocks is divided into the plurality of time-space-frequency tokens according to spatial positions.

4. The computing device of claim 3, wherein the processor is configured to generate the plurality of spectrograms at least in part by: performing a discrete cosine transform (DCT) on each of the plurality of first blocks to generate a plurality of first spectrograms; and The DCT is performed on each of the plurality of second blocks to generate a plurality of second frequency spectrograms.

5. The computing device of claim 4, wherein the trained machine learning model is a transformer network.

6. The computing device of claim 5, wherein: The trained machine learning model includes: a spatial frequency attention head, the processor being configured at the spatial frequency attention head to: calculate a plurality of spatial frequency attention weights between corresponding spatial frequency domain blocks, the corresponding spatial frequency domain blocks being generated for an input image of the plurality of input images; a temporal frequency attention head, the processor being configured at the temporal frequency attention head to: calculate a plurality of temporal frequency attention weights between corresponding spatial frequency blocks, the corresponding spatial frequency blocks being generated for shared spatial locations in consecutive ones of the plurality of input images; At least in part at the trained machine model, the processor is further configured to generate the plurality of super-resolved output images based at least in part on the plurality of spatial frequency attention weights and the plurality of temporal frequency attention weights.

7. The computing device of claim 6, wherein: The spatial-frequency attention head is configured to receive as input a plurality of query tokens, a plurality of first key tokens, and a plurality of first value tokens included among the plurality of temporal-spatial-frequency tokens; The processor is configured to: generating the plurality of query tokens based at least in part on a plurality of first spatial-frequency domain blocks generated from the first spectrogram; as well as The plurality of first key tokens and the plurality of first value tokens are generated based at least in part on a plurality of second spatial-frequency domain blocks generated from the second spectrogram.

8. The computing device of claim 7, wherein: The time-frequency attention head is configured to receive as input the plurality of query tokens, a plurality of second key tokens, and a plurality of second value tokens included in the plurality of time-space frequency tokens; The processor is configured to generate the plurality of second key tokens and the plurality of second value tokens based at least in part on an output of the spatial-frequency attention head and one or more recurrent hidden states.

9. A computing device according to claim 4, wherein the processor is further configured to generate the multiple super-resolved output images at least in part by performing an inverse DCT (rDCT) at a post-processing stage executed downstream of the trained machine learning model.

10. The computing device of claim 1, wherein for each input image of the plurality of input images, the processor is further configured to output a corresponding recurrent hidden state.

11. The computing device of claim 10, wherein the processor is further configured to: computing a plurality of optical flow maps between consecutive pairs of input images included in the input video data; for each input image other than the first input image of the plurality of input images, computing a corresponding deformed hidden state at least in part by applying an optical flow map of the plurality of optical flow maps to the corresponding recurrent hidden state, the corresponding recurrent hidden state being generated for a previous input image of the plurality of input images; and At the trained machine learning model, the plurality of super-resolved output images are generated based at least in part on a plurality of deformed hidden states.

12. A method for use with a computing device, the method comprising: receiving input video data comprising a plurality of input images, wherein each input image of the plurality of input images comprises a plurality of input pixels; as well as For each input image of the plurality of input images: Performing upsampling on the input image; Dividing the upsampled input image into a corresponding plurality of blocks; as well as For each of the plurality of partitions, generating a plurality of time-space-frequency tokens, wherein the plurality of time-space-frequency tokens generated for the partition are indexed by time step, spatial position, and frequency; generating, at least in part at the trained machine learning model, a plurality of super-resolved output images based at least in part on the plurality of time-space-frequency tokens; as well as The plurality of super-resolution output images are output.

13. The method according to claim 12, wherein: Performing the upsampling on the input image comprises: performing bicubic interpolation on the input image to generate a first upsampled image; processing the input image at an upsampling neural network to generate a second upsampled image; and The method further includes dividing the first upsampled image and the second upsampled image into a plurality of first blocks and a plurality of second blocks, respectively.

14. The method of claim 12, wherein the trained machine learning model is a transformer network.

15. The method according to claim 12, further comprising: computing a plurality of optical flow maps between consecutive pairs of input images included in the input video data; for each input image other than the first input image in the plurality of input images, computing a corresponding deformed hidden state at least in part by applying an optical flow map in the plurality of optical flow maps to a corresponding recurrent hidden state, the corresponding recurrent hidden state being generated for a previous input image in the plurality of input images; as well as generating, at least in part at the trained machine learning model, the plurality of super-resolved output images based at least in part on the plurality of deformed hidden states; as well as The recurrent hidden state is calculated for each input image of the plurality of input images.

Images