Video super-resolution based on compressed information

Abstract

Exemplary aspects of the present disclosure are directed to systems and methods featuring machine-learned video super-resolution (VSR) models trained using a bidirectional training approach. In particular, the present disclosure provides a compression-information-based (e.g., compression-aware) super-resolution model that can perform well on real-world videos with different compression levels. Specifically, the exemplary model described herein can include three modules for robustly restoring missing information caused by video compression. First, a bidirectional recurrent module can be used to reduce accumulated warping errors from random positions of intra-frames from compressed video frames. Second, a detail-aware flow estimation module can be added to enable restoration of high-resolution (HR) flows from compressed low-resolution (LR) frames. Finally, a Laplacian enhancement module can add high-frequency information to warped HR frames that have been washed out by video encoding.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to and the benefit of U.S. Provisional Patent Application No. 63 / 179,795, filed April 26, 2021. U.S. Provisional Patent Application No. 63 / 179,795 is incorporated herein by reference in its entirety.

[0002] TECHNICAL FIELD This disclosure relates generally to systems and methods for performing compression-informed video super-resolution, and more particularly, to systems and methods featuring machine-learned video super-resolution models trained using a bidirectional training approach. [Background technology]

[0003] Super-resolution is a fundamental research problem in computer vision with numerous applications. Systems performing super-resolution aim to reconstruct detailed high-resolution (HR) images from low-resolution (LR) inputs. When the input is a single image, the reconstruction process typically uses learned image priors to restore high-resolution details of a given image, which can be called single-image super-resolution (SISR). When multiple frames in a video are available, some reconstruction processes can use both image priors and inter-frame information to produce a temporally smooth high-resolution result, which can be called video super-resolution (VSR).

[0004] While there has been great progress in the field of super-resolution, existing SISR and VSR methods rarely take compression into account. Specifically, certain previous works have used "uncompressed" data to enhance high-quality, low-compression videos. Thus, previous methods tend to produce significant artifacts when operated on heavily compressed input videos.

[0005] In particular, most digital videos (e.g., those present on the Internet or on mobile devices such as smartphones) are stored and / or streamed at different compression levels to achieve a selected visual quality level. For example, a typical compression ratio (Constant Rate Factor (CRF)) for H.264 encoding is 23 as a trade-off between quality and file size. Existing techniques designed and optimized for the application of VSR to uncompressed video data do not work well when applied to such compressed video.

[0006] One possible solution is to apply a denoising model to remove compression artifacts, followed by one of the state-of-the-art VSR models. At first glance, this is attractive, since the VSR model is fed high-quality frames, similar to using the evaluation data directly. However, experiments have shown that such a setting does not improve the final performance, and in fact may even make it worse. In preprocessing, the denoising model in the first step is likely to modify the degradation kernels implicitly used in the VSR model training. Thus, in essence, the VSR model is being applied to more challenging data.

[0007] Another possible solution is to train existing state-of-the-art VSR models on compressed frames, which can bring additional compression information to the model training. However, experiments have shown that simply using compressed frames in training only brings slight improvements. In fact, without specific modifications to the design of the network modules, such training data may even degrade the overall performance. Summary of the Invention [Problem to be solved by the invention]

[0008] Therefore, there is a need for improved systems, methods, model architectures, and training approaches that provide improved VSR on compressed video data. [Means for solving the problem]

[0009] One or more computer systems may be configured to perform particular operations or actions by installing software, firmware, hardware, or any combination thereof on the system that, during operation, causes the system to perform the actions. One or more computer programs may be configured to perform particular operations or actions by including instructions that, when executed by a data processing device, cause the device to perform the actions.

[0010] One general aspect includes a computer-implemented method for bidirectionally training a machine-learned video super-resolution (VSR) model using compressed video data. The computer-implemented method includes obtaining, by a computing system including one or more computing devices, a set of ground truth video data that may include a plurality of ground truth high resolution (HR) video frames and a plurality of low resolution (LR) video frames, where the plurality of LR video frames each correspond to a plurality of ground truth HR video frames, and the plurality of ground truth HR video frames and the plurality of LR video frames are arranged in a time sequence corresponding to the compressed video. The method also includes performing, by the computing system, a forward temporal prediction to generate a forward predicted HR video frame for a current position in the time sequence based on one or more video frames associated with one or more previous positions in the time sequence. The method also includes performing, by the computing system, a backward temporal prediction to generate a backward predicted HR video frame for a current position in the time sequence based on one or more video frames associated with one or more subsequent positions in the time sequence. The method also includes evaluating, by the computing system, a loss function of the machine-learned VSR model, the loss function comparing the ground truth HR video frame to the forward predicted HR video frame and comparing the ground truth HR video frame to the backward predicted HR video frame. The method also includes modifying, by the computing system, one or more values ​​of one or more parameters of the machine-learned VSR model based on the loss function. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0011] Another example aspect is directed to a computing system including one or more processors and one or more non-transitory computer-readable media that collectively store a machine-learned video super-resolution (VSR) model and instructions that, when executed by the one or more processors, cause the computing system to use the machine-learned VSR model to super-resolve compressed video.

[0012] The machine-learned video super-resolution (VSR) model may include a flow estimation portion configured to process a previous or subsequent LR video frame and a current LR video frame to generate a low resolution (LR) flow estimate and a high resolution (HR) flow estimate, warp the previous or subsequent LR video frame according to the LR flow estimate to generate a predicted LR video frame for the current position in the time sequence, and warp the previous or subsequent HR video frame according to the HR flow estimate to generate an intermediate HR video frame for the current position in the time sequence, a Laplacian enhancement portion configured to enhance the intermediate HR video frame, and a frame generation portion configured to process the intermediate HR video frame and the current LR video frame to generate a predicted HR video frame for the current position in the time sequence.

[0013] Implementations of the described techniques may include hardware, a method or process on a computer-accessible medium, or computer software.

[0014] A detailed description of the embodiments, directed to persons skilled in the art, is set forth herein with reference to the accompanying drawings. [Brief description of the drawings]

[0015] [Figure 1A] FIG. 1 is a graphical diagram of an example process for bidirectionally training a machine-learned video super-resolution model, according to an example embodiment of the present disclosure. [Figure 1B] FIG. 1 is a graphical diagram of an example process for bidirectionally training a machine-learned video super-resolution model, according to an example embodiment of the present disclosure. [Diagram 2] FIG. 1 is a graphical diagram of an example architecture of an example machine-learned video super-resolution model, according to an example embodiment of the present disclosure. [Figure 3A] FIG. 1 is a block diagram of an exemplary computing system in accordance with an exemplary embodiment of the present disclosure. [Figure 3B] FIG. 2 is a block diagram of an exemplary computing device in accordance with an exemplary embodiment of the present disclosure. [Figure 3C] FIG. 2 is a block diagram of an exemplary computing device in accordance with an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Reference numbers repeated among the figures are intended to identify like features in various implementations.

[0017] overview Exemplary aspects of the present disclosure are directed to systems and methods featuring machine-learned video super-resolution (VSR) models trained using a bidirectional training approach. In particular, the present disclosure provides a compression-informed (e.g., compression-aware) super-resolution model that can perform well on real-world videos with different compression levels. Specifically, the exemplary model described herein can include three modules for robustly restoring missing information caused by video compression. First, a bidirectional recurrent module can be used to reduce accumulated warping errors from random positions of intra-frames from compressed video frames. Second, a detail-aware flow estimation module can be added to enable restoration of high-resolution (HR) flows from compressed low-resolution (LR) frames. Finally, a Laplacian enhancement module can add high-frequency information to warped HR frames that have been washed out by video encoding. An exemplary implementation of the proposed model may be referred to as COMpression-Informed video Super-Resolution (COMISR) in some examples.

[0018] In U.S. Provisional Patent Application No. 63 / 179,795, which is included in and forms a part of this disclosure, the effectiveness of an exemplary implementation of COMISR using three modules is demonstrated by an ablation study. In particular, extensive experiments were performed on several VSR benchmark datasets using videos compressed with different CRF values. The experiments showed that the COMISR model achieves significant performance gains in compressed videos (e.g., CRF23) while maintaining competitive performance in uncompressed videos. In addition, U.S. Provisional Patent Application No. 63 / 179,795 presents evaluation results based on different combinations of state-of-the-art VSR models and off-the-shelf video denoisers. Finally, U.S. Provisional Patent Application No. 63 / 179,795 demonstrates the robustness of the COMISR model simulating streaming YouTube® videos compressed with a proprietary encoder.

[0019] Therefore, one exemplary aspect of the present disclosure is directed to a compression-informed model for super-resolution of real-world compressed videos for practical applications. Another exemplary aspect includes three modules that are new to VSR to effectively improve critical components for video super-resolution on compressed frames. Finally, extensive experiments are performed on the state-of-the-art VSR model on compressed benchmark datasets.

[0020] The systems and methods of the present disclosure provide several technical effects and benefits. As an example, the models described herein can perform improved image processing, such as improved image super-resolution (e.g., increasing image resolution through image synthesis). For example, by performing bidirectional training of the VSR model, the VSR model can be better equipped / trained to take into account temporal artifacts introduced by the compression process.

[0021] Specifically, one common technique used in video compression is to apply different algorithms to compress and encode frames at different positions in a video stream. Typically, a codec randomly selects some reference frames, known as intra frames, and compresses them independently without using information from other frames. It then compresses other frames by exploiting the consistency and encoding differences from the intra frames. As a result, intra frames usually require more bits to encode than other frames and have fewer compression artifacts. In video super-resolution, since the positions of intra frames are not known in advance, the proposed bidirectional approach can be used to enforce forward and backward consistency of the LR warp input and the HR predicted frames to effectively reduce the accumulated error from the unknown positions of the intra frames.

[0022] The systems and methods of the present disclosure can be used for several applications. In one example, the models described herein can be used to enhance the resolution of compressed video. For example, compressed video can be transmitted or streamed in compressed form and then super-resolved at the end device that displays the video. This can provide technical benefits of saving network bandwidth and storage space, since compressed video may require less computational resources to transmit and / or store. By way of example, the compressed video can be a compressed video conferencing video stream, compressed user-generated content video, and / or any other type of video.

[0023] Referring now to the drawings, exemplary embodiments of the present disclosure will be described in more detail.

[0024] Example Model Training and Inference The exemplary COMISR model is designed using a recurrent formulation that feeds previous information to the current frame, which is similar to state-of-the-art video SR methods. Recurrent designs typically consume less memory and can be applied to many inference tasks in videos.

[0025] The exemplary model architecture described herein can include three novel parts, namely, bidirectional recurrent warping, detail-aware flow estimation, and Laplacian enhancement parts, to make the model robust to compressed video. Given a LR ground truth frame, the model can apply forward and backward recurrent modules (see Fig. 1A-1B) to generate HR frame predictions and calculate content loss for the HR ground truth frame in both directions. The recurrent modules can predict the flow and generate warped frames in both LR and HR, and train the network end-to-end using the LR and HR ground truth frames.

[0026] Example Bidirectional Recurrent Module One technique used in video compression is to apply different algorithms to compress and encode frames at different positions in a video stream. Typically, a codec randomly selects some reference frames, known as intraframes, and compresses them independently without using information from other frames. It then compresses other frames by exploiting the consistency and encoding differences from the intraframes. As a result, intraframes usually require more bits to encode than other frames and have fewer compression artifacts. In video super-resolution, the positions of intraframes are not known in advance, so to effectively reduce the accumulated error from the unknown positions of intraframes, this disclosure proposes a bidirectional recurrent network to enforce forward and backward consistency of the LR warp input and the HR predicted frames.

[0027] Specifically, a bidirectional recurrent network can include symmetric modules for the forward and reverse directions. In the forward direction, the model first performs the LR flow

[0028]

number

[0029] and HR Flow

[0030]

number

[0031] Both the LR frame

[0032]

number

[0033] and

[0034]

number

[0035] The model can then apply different operations separately for the LR and HR streams. For the LR stream, the model estimates the LR frame using the

[0036]

number

[0037] of,

[0038]

number

[0039] Warp to time t using

[0040]

number

[0041] can be obtained, which will be used in a later stage.

[0042]

number

[0043] In the HR stream, the model predicts the previous frame

[0044]

number

[0045] of,

[0046]

number

[0047] Warp to time t using

[0048]

number

[0049] followed by a Laplacian enhancement module to generate the accurate HR warped frame.

[0050]

number

[0051] The model then:

[0052]

number

[0053] We apply a spatial-depth operation to the

[0054]

number

[0055] Then, the concatenated frame is fed to the HR frame generator to generate the final HR prediction.

[0056]

number

[0057] The training process is carried out as follows to measure the loss:

[0058]

number

[0059] Ground truth HR

[0060]

number

[0061] can be compared to:

[0062] Similarly, the model can apply a backward symmetric operation to obtain a warped LR frame and a predicted HR frame. In this case, the detail-aware flow estimation module can generate a backward flow from time t to t-1, and the warping can be performed by applying the backward flow to a frame at time t to estimate a frame at time t-1.

[0063] 1A and 1B show an example VSR model used for forward and backward temporal prediction, respectively. In some implementations of the present disclosure, forward temporal prediction may be performed in both training and inference, while backward temporal prediction may be performed only during training.

[0064] Specifically, in some implementations, to train the VSR model, the computing system may obtain several sets of ground truth training data. Training iterations may be performed on batches of training videos, with each batch including one or more sets of ground truth video data.

[0065] In particular, the set of ground truth video data may include a plurality of ground truth high resolution (HR) video frames and a plurality of low resolution (LR) video frames. The plurality of LR video frames correspond to a plurality of ground truth HR video frames, respectively. For example, each LR frame may be a relatively low resolution version of a corresponding one of the HR frames. In one example, the frames of the HR video may be downsampled and / or compressed to generate the LR frames. The HR frames may themselves be compressed or uncompressed.

[0066] The ground truth HR video frames and the LR video frames may be arranged in a time sequence. As an example, the time sequence may correspond to sequentially numbered frames captured in consecutive order by an image capture device.

[0067] The model can be trained for one or more of the positions in the time sequence, for example, training can be performed for all positions in the time sequence.

[0068] Specifically, the VSR model can be used to perform forward temporal prediction to generate a forward predicted HR video frame for a current position in a time sequence based on one or more video frames associated with one or more previous positions in the time sequence. An example of forward temporal prediction is shown in FIG. 1A. In addition, according to one aspect of the present disclosure, the VSR model can also be used to perform backward temporal prediction to generate a backward predicted HR video frame for a current position in a time sequence based on one or more video frames associated with one or more subsequent positions in the time sequence. An example of backward temporal prediction is shown in FIG. 1B.

[0069] In some implementations, the forward and backward models are symmetric and share weights. A different idea is that the same model can be used for each of the forward and backward passes, but applied to the frames in a different (e.g., opposite) order or sequence. For example, the order of the frames can simply be reversed.

[0070] After performing forward and / or backward temporal prediction, the computing system can evaluate a loss function of the machine-learned VSR model. As an example, the loss function can both (1) compare the ground truth HR video frames with the forward predicted HR video frames generated by forward temporal prediction, and (2) compare the ground truth HR video frames with the backward predicted HR video frames generated by backward temporal prediction. The loss function can be jointly evaluated for both (1) and (2) above, or (1) and (2) can be evaluated separately and then summed or otherwise treated together (e.g., as a batch).

[0071] The computing system can modify one or more values ​​of one or more parameters of the machine-learned VSR model based on the loss function, for example, by using backpropagation of error to update the values ​​of the parameters of the machine-learned VSR model according to the gradient of the loss function.

[0072] In particular, with specific reference to FIG. 1A , in some implementations, performing forward temporal prediction may include processing, by a computing system, a previous HR video frame 14 associated with a previous position in the time sequence, a previous LR video frame 16 associated with a previous position in the time sequence, and a current LR video frame 18 associated with the current position in the time sequence to generate a forward predicted HR video frame 20 for the current position in the time sequence using a machine-learned VSR model 12.

[0073] Similarly, with specific reference to FIG. 1B , performing backward temporal prediction may include processing, by a computing system, a subsequent HR video frame 24 associated with a subsequent position in the time sequence, a subsequent LR video frame 26 associated with a subsequent position in the time sequence, and a current LR 28 associated with the current position in the time sequence, to generate a backward predicted HR video frame 30 for the current position in the time sequence using the machine-learned VSR model 12.

[0074] The previous HR video frame 14 may be a previous predicted HR video frame or may be a previous ground truth HR video frame. Similarly, the subsequent HR video frame 24 may be a subsequent predicted HR video frame or may be a subsequent ground truth HR video frame.

[0075] Example Recurrent Model Details 2, there is shown an example architecture of an example VSR model 200. The model 200 may include a flow estimation portion 202, a Laplacian enhancement portion 204, and / or a frame generation portion 206.

[0076] The flow estimation portion 202 may be configured to process a previous or subsequent low resolution (LR) video frame (e.g., previous LR video frame 16) and a current LR video frame 18 to generate an LR flow estimate 210 and a high resolution (HR) flow estimate 212. The flow estimation portion 202 may warp the previous or subsequent LR video frame (e.g., 16) according to the LR flow estimate 210 to generate a predicted LR video frame 214 for the current position in the time sequence. The flow estimation portion 202 may warp the previous or subsequent HR video frame (e.g., previous HR frame 14) according to the HR flow estimate 212 to generate an intermediate HR video frame 216 for the current position in the time sequence.

[0077] The Laplacian enhancement portion 204 can be configured to enhance the intermediate HR video frames 216 .

[0078] The frame generation portion 206 may be configured to process the intermediate HR video frame 216 (eg, after enhancement) and the current LR video frame 18 to generate a predicted HR video frame 20 for a current position in the time sequence.

[0079] Similarly, performing backward temporal prediction (not specifically shown) by the computing system may include processing, by the computing system, a subsequent LR video frame and a current LR video frame using the flow estimation portion 202 of the machine-learned VSR model to generate an LR backward flow estimate and a HR backward flow estimate, warping, by the computing system, the subsequent LR video frame according to the LR backward flow estimate to generate a backward predicted LR video frame for the current position in the time sequence, and warping, by the computing system, the subsequent HR video frame according to the HR backward flow estimate to generate a backward-intermediate HR video frame for the current position in the time sequence.

[0080] Similarly, performing backward temporal prediction by the computing system may include applying, by the computing system, a Laplacian enhancement filter to the backward-intermediate HR video frame, and after applying the Laplacian enhancement filter, processing, by the computing system, the backward-intermediate HR video frame and the current LR video frame using a frame generation portion of the machine-learned VSR model to generate a backward predicted HR video frame for the current position in the time sequence.

[0081] In some implementations, the loss function may further compare (3) a forward predicted LR video frame 214 for the current position with a current LR video frame 18 associated with the current position in the time sequence, and / or (4) a backward predicted LR video frame (not specifically shown) for the current position with a current LR video frame associated with the current position in the time sequence.

[0082] A previous position in the time sequence may be the immediately preceding position in the time sequence or may be a non-immediately adjacent position in time. Similarly, a subsequent position in the time sequence may be the immediately following position in the time sequence or may be a non-immediately adjacent position in time.

[0083] After training, the machine-learned VSR model can be used to super-resolve the additional compressed video. For example, using the machine-learned VSR model to super-resolve the additional compressed video can include performing only forward temporal prediction on video frames of the additional compressed video.

[0084] The training techniques described herein may be performed on multiple compressed training videos for multiple training iterations each. The multiple compressed training videos may have been compressed using the same compression algorithm or multiple different compression algorithms. One exemplary compression algorithm is the H.264 codec.

[0085] Exemplary Detailed Recognition Flow Estimation In the proposed recurrent model, the model can explicitly estimate both LR and HR flows between adjacent frames and pass this information forward and backward.

[0086] Figure 2 shows the forward direction for illustration. The backward operation is similarly applied. The model first divides two adjacent LR frames

[0087]

number

[0088] and

[0089]

number

[0090] Concatenate them and pass them through the LR flow estimation network to get the LR flow

[0091]

number

[0092] It is possible to generate LR flows.

[0093]

number

[0094] Instead of directly upsampling, the model can apply several additional deconvolution layers on top of the bilinearly upsampled LR flow. Thus, detailed residual maps can be encouraged to be learned during end-to-end training, so that the model can better preserve high-frequency details in the predicted HR flow.

[0095] Exemplary Laplacian Enhancement Module The Laplacian residual is widely used in many vision tasks, including image blending, super-resolution, and restoration. It is particularly useful for finding fine details from video frames, and such details can be smoothed during video compression. In some examples of the proposed recurrent VSR model, the warped predicted HR frame retains the information and some details learned from the previous frame. Such details can easily be missing from the upscaling network. Therefore, some exemplary implementations include the Laplacian residual on the predicted HR frame to highlight the details.

[0096] The Laplacian boosted image can be computed by a Gaussian kernel blur G(·,·) with width σ.

[0097]

number

[0098] During the ceremony,

[0099]

number

[0100] is the intermediate result of the predicted HR frame, and α is a weighting factor that controls the residual power. By utilizing the Laplacian, the model can add details to the warped HR frame. This is followed by a spatial-depth operation that rearranges blocks of spatial data into the depth dimension, which can then be concatenated with the LR input frame. The model can pass it to the HR frame generator to generate the final HR prediction.

[0101] Example loss function During training, there are typically two streams: HR and LR frames. A loss can be designed considering the use of both streams. For the loss of HR frames, we can calculate the L2 distance between the final output and the HR frame. t denotes the ground truth frame,

[0102]

number

[0103] Let t denote the frame generated at time t. For each recurrent step, the predicted HR frame can be used to compute the loss. The L2 loss can optionally be combined as follows:

[0104]

number

[0105] Each of the warped LR frames from t-1 to t can also be penalized for its L2 distance to the current LR frame as follows:

[0106]

number

[0107] One exemplary total loss can be the sum of the HR loss and the LR loss,

[0108]

number

[0109] where β and γ are the weights for each loss.

[0110] Exemplary Devices and Systems 3A illustrates a block diagram of an exemplary computing system 100 for performing video super-resolution, according to an exemplary embodiment of the present disclosure. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 communicatively coupled via a network 180.

[0111] The user computing device 102 may be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0112] The user computing device 102 includes one or more processors 112 and memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or multiple processors operably connected. The memory 114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 may store data 116 and instructions 118 that are executed by the processor 112 to cause the user computing device 102 to perform operations.

[0113] In some implementations, the user computing device 102 may store or include one or more machine-learned VSR models 120. For example, the machine-learned VSR models 120 may be or otherwise include various machine-learned models, such as neural networks (e.g., deep neural networks) or other types of machine-learned models including nonlinear and / or linear models. The neural networks may include feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. Some exemplary machine-learned models may leverage attention mechanisms such as self-attention. For example, some exemplary machine-learned models may include multi-headed self-attention models (e.g., transformer models). An exemplary machine-learned VSR model 120 is described with reference to FIGS. 1A, 1B, and 2.

[0114] In some implementations, one or more machine-learned VSR models 120 may be received from the server computing system 130 over the network 180, stored in the user computing device memory 114, and then used or otherwise implemented by one or more processors 112. In some implementations, the user computing device 102 may implement multiple parallel instances of a single machine-learned VSR model 120 (e.g., to perform parallel video super-resolution across multiple instances of a low-resolution video).

[0115] Additionally or alternatively, one or more machine-learned VSR models 140 may be included in or otherwise stored and implemented by a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the machine-learned VSR models 140 may be implemented by the server computing system 140 as part of a web service (e.g., a video super-resolution service). Thus, one or more models 120 may be stored and implemented in the user computing device 102 and / or one or more models 140 may be stored and implemented in the server computing system 130.

[0116] The user computing device 102 may also include one or more user input components 122 that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). The touch-sensitive component may serve to implement a virtual keyboard. Other exemplary user input components include a microphone, a traditional keyboard, or other means by which a user may provide user input.

[0117] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or multiple processors operably connected. The memory 134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 may store data 136 and instructions 138 that are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0118] In some implementations, the server computing system 130 includes or is otherwise implemented by one or more server computing devices. When the server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0119] As described above, the server computing system 130 can store or otherwise include one or more machine-learned VSR models 140. For example, the models 140 can be or otherwise include various machine-learned models. Exemplary machine-learned models include neural networks or other multi-layer nonlinear models. Exemplary neural networks include feed-forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some exemplary machine-learned models can leverage attention mechanisms such as self-attention. For example, some exemplary machine-learned models can include multi-headed self-attention models (e.g., transformer models). Exemplary models 140 are described with reference to FIGS. 1A, 1B, and 2.

[0120] The user computing device 102 and / or the server computing system 130 can train the models 120 and / or 140 through interaction with a training computing system 150 that is communicatively coupled via a network 180. The training computing system 150 may be separate from the server computing system 130 or may be part of the server computing system 130.

[0121] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or multiple processors operably connected. The memory 154 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 154 may store data 156 and instructions 158 that are executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0122] The training computing system 150 may include a model trainer 160 that trains the machine-learned models 120 and / or 140 stored in the user computing device 102 and / or the server computing system 130 using various training or learning techniques, such as, for example, backpropagation. For example, a loss function may be backpropagated through the model to update one or more parameters of the model (e.g., based on the gradient of the loss function). Various loss functions may be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques may be used to iteratively update the parameters over a number of training iterations.

[0123] In some implementations, performing backpropagation may include performing truncated backpropagation through time. The model trainer 160 can perform several generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the model being trained.

[0124] In particular, the model trainer 160 can train the machine-learned VSR model 120 / 140 based on a set of training data 162. The training data 162 can include, for example, ground truth video data. For example, the ground truth video data can include videos in both a higher resolution format and a corresponding lower resolution format.

[0125] In some implementations, the training data may include the REDS and / or Vimeo datasets for training. The REDS dataset includes over 200 video sequences for training, each of which has 100 frames with a resolution of 1280x720. The Vimeo-90K dataset includes approximately 65k video sequences for training, each of which has 7 frames with a resolution of 448x256. One main difference between these two datasets is that the REDS dataset has much greater motion between consecutive frames captured from a handheld device. To train and evaluate the COMISR model, the frames may first be smoothed by a Gaussian kernel with a width of 1.5 and downsampled by 4x.

[0126] In some implementations, the COMISR model can be evaluated on the Vid4 and REDS4 datasets (clip numbers 000, 011, 015, 020). All test sequences have more than 30 frames.

[0127] In some implementations, the following compression methods can be used. One example follows the most common settings for the H.264 codec at different compression ratios (i.e., different CRF values). The recommended CRF values ​​are 18-28, with a default of 23 (although the values ​​range from 0-51). In some examples, CRFs of 15, 25, and 35 can be used to evaluate video super-resolution at a wide range of compression ratios. In some implementations, the same degradation method to generate LR sequences before compression. Finally, such compressed LR sequences are fed to the VSR model for inference.

[0128] In some implementations, the following training process can be used: In some implementations, for each input frame, the training process can randomly crop a patch (e.g., a 128x128 patch) from the mini-batch as input. Each mini-batch can contain several samples (e.g., 16 samples). The α, β, and γ parameters can be set to 1, 20, 1, respectively. The model training can be supervised by losses described elsewhere herein. The Adam optimizer can be used with β_1=0.9 and β_2=0.999. The learning rate can be set to 5x10^(-5). Video compression can be optionally employed as an additional data augmentation method to the training pipeline with a probability of 50% on the input batch.

[0129] In some implementations, if the user has given consent, the training examples may be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 may be trained by the training computing system 150 on user-specific data received from the user computing device 102. In some examples, this process may be referred to as personalizing the model.

[0130] The model trainer 160 includes computer logic that is utilized to provide the desired functionality. The model trainer 160 can be implemented in hardware, firmware, and / or software that controls a general-purpose processor. For example, in some implementations, the model trainer 160 includes program files that are stored in a storage device, loaded into memory, and executed by one or more processors. In other implementations, the model trainer 160 includes one or more sets of computer-executable instructions stored in a tangible computer-readable storage medium, such as RAM, a hard disk, or an optical or magnetic medium.

[0131] Network 180 may be any type of communications network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications over network 180 may occur over any type of wired and / or wireless connections, using a wide variety of communications protocols (e.g., TCP / IP, HTTP, SMTP, FTP, etc.), encodings or formats (e.g., HTML, XML), and / or security schemes (e.g., VPN, Secure HTTP, SSL, etc.).

[0132] 3A illustrates one exemplary computing system that can be used to implement the present disclosure. Other computing systems can also be used. For example, in some implementations, the user computing device 102 can include a model trainer 160 and a training dataset 162. In such implementations, the model 120 can be trained and used locally on the user computing device 102. In some such implementations, the user computing device 102 can implement the model trainer 160 to personalize the model 120 based on user-specific data.

[0133] 3B illustrates a block diagram of an exemplary computing device 10 for performing according to an exemplary embodiment of the present disclosure. The computing device 10 may be a user computing device or a server computing device.

[0134] Computing device 10 includes several applications (e.g., applications 1-N). Each application includes its own machine learning library and machine-learned models. For example, each application may include a machine-learned model. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc.

[0135] 3B, each application may communicate with several other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application may communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0136] 3C illustrates a block diagram of an exemplary computing device 50 for performing according to an exemplary embodiment of the present disclosure. The computing device 50 may be a user computing device or a server computing device.

[0137] Computing device 50 includes several applications (e.g., applications 1-N). Each application communicates with a central intelligence layer. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and models stored therein) using an API (e.g., a common API across all applications).

[0138] The central intelligence layer includes several machine-learned models. For example, as shown in FIG. 3C, each machine-learned model may be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications may share a single machine-learned model. For example, in some implementations, the central intelligence layer may provide a single model for all applications. In some implementations, the central intelligence layer is included within or otherwise implemented by the operating system of the computing device 50.

[0139] The central intelligence layer can communicate with a central device data layer. The central device data layer can be a centralized repository of data for the computing device 50. As shown in FIG. 3C, the central device data layer can communicate with several other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0140] Additional Disclosures The technology described herein refers to servers, databases, software applications, and other computer-based systems, as well as actions taken and information transmitted to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functions among components. For example, the processes described herein can be implemented using a single device or component, or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0141] Although the present subject matter has been described in detail with respect to various specific exemplary embodiments thereof, each example is provided as an explanation and not a limitation of the present disclosure. Those skilled in the art, upon achieving the above understanding, can easily generate modifications, variations, and equivalents of such embodiments. Thus, the present disclosure does not exclude the inclusion of such modifications, variations, and / or additions to the present subject matter as would be readily apparent to those skilled in the art. For example, features illustrated or described as part of one embodiment can be used with another embodiment to obtain yet a further embodiment. Thus, the present disclosure is intended to cover such modifications, variations, and equivalents. [Explanation of symbols]

[0142] 10. Computing Devices 12 Machine learning VSR model 14 Previous HR Video Frames 16 Previous LR Video Frames 18 Current LR video frames 20 forward predicted HR video frames 24 Subsequent HR video frames 26 subsequent LR video frames 28 Current LR 30 backward predicted HR video frames 50 Computing Devices 100 Computing Systems 102 User Computing Devices 112 processors 114 Memory 116 Data 118 Command 120 Machine-learned VSR models 122 User Input Components 130 Server Computing System 132 processors 134 Memory 136 Data 138 Command 140 Machine-learned VSR model, server computing system 150 Training Computing System 152 processors 154 Memory 156 Data 158 Command 160 Model Trainer 162 Training data, training data set 180 Network 200 VSR model 202 Flow Estimation Part 204 Laplacian Enhancement 206 Frame Generation Part 210 LR Flow Estimation 212 High Resolution (HR) Flow Estimation 214 Predicted LR Video Frames 216 Medium HR Video Frames

Claims

1. 1. A computer-implemented method for bidirectionally training a machine-learned video super-resolution (VSR) model using compressed video data, comprising: obtaining, by a computing system comprising one or more computing devices, a set of ground truth video data comprising a plurality of ground truth high resolution (HR) video frames and a plurality of low resolution (LR) video frames, each of the plurality of LR video frames corresponding to the plurality of ground truth HR video frames, and the plurality of ground truth HR video frames and the plurality of LR video frames being arranged in a time sequence corresponding to a compressed video; For each of one or more positions in the time sequence, performing, by the computing system, a forward temporal prediction to generate a forward predicted HR video frame for a current position in the time sequence based on one or more video frames associated with one or more previous positions in the time sequence; performing, by the computing system, backward temporal prediction to generate a backward predicted HR video frame for the current position in the time sequence based on one or more video frames associated with one or more subsequent positions in the time sequence; evaluating, by the computing system, a loss function of the machine-learned VSR model, the loss function comparing the ground truth HR video frames to the forward predicted HR video frames and comparing the ground truth HR video frames to the backward predicted HR video frames; modifying, by the computing system, one or more values ​​of one or more parameters of the machine-learned VSR model based on the loss function; 4. A computer-implemented method comprising:

2. performing the forward temporal prediction includes processing, by the computing system, a previous HR video frame associated with a previous position in the time sequence, a previous LR video frame associated with the previous position in the time sequence, and a current LR video frame associated with a current position in the time sequence, using the machine-learned VSR model to generate the forward predicted HR video frame for the current position in the time sequence; 2. The computer-implemented method of claim 1, wherein performing the backward temporal prediction includes processing, by the computing system, a subsequent HR video frame associated with a subsequent position in the time sequence, a subsequent LR video frame associated with the subsequent position in the time sequence, and a current LR associated with a current position in the time sequence, to generate the backward predicted HR video frame for the current position in the time sequence using the machine-learned VSR model.

3. the previous HR video frame comprises a previous predicted HR video frame; The computer-implemented method of claim 2 , wherein the subsequent HR video frame comprises a subsequent predicted HR video frame.

4. the previous HR video frame comprises a previous ground truth HR video frame; The computer-implemented method of claim 2 , wherein the subsequent HR video frame comprises a subsequent ground truth HR video frame.

5. performing, by the computing system, the forward temporal prediction, processing, by the computing system, the previous LR video frame and the current LR video frame to generate an LR forward flow estimate and an HR forward flow estimate using a flow estimation portion of the machine-learned VSR model; warping, by the computing system, the previous LR video frame according to the LR forward flow estimation to generate a forward predicted LR video frame for the current position in the time sequence; warping, by the computing system, the previous HR video frame according to the HR forward flow estimate to generate a forward-intermediate HR video frame for the current position in the time sequence; Including, performing, by the computing system, the backward temporal prediction, processing, by the computing system, the subsequent LR video frame and the current LR video frame to generate an LR reverse flow estimate and an HR reverse flow estimate using the flow estimation portion of the machine-learned VSR model; warping, by the computing system, the subsequent LR video frame according to the LR backward flow estimation to generate a backward predicted LR video frame for the current position in the time sequence; warping, by the computing system, the subsequent HR video frame according to the HR backward flow estimate to generate a backward-intermediate HR video frame for the current position in the time sequence; 5. The computer-implemented method of claim 2, comprising:

6. The loss function further comprises: comparing the forward predicted LR video frame for the current position to the current LR video frame associated with the current position in the time sequence; The computer-implemented method of claim 5 , further comprising comparing the backward predicted LR video frame for the current position to the current LR video frame associated with the current position in the time sequence.

7. performing, by the computing system, the forward temporal prediction, applying, by the computing system, a Laplacian enhancement filter to the forward-intermediate HR video frame; after applying the Laplacian enhancement filter, processing, by the computing system, the forward-intermediate HR video frame and the current LR video frame to generate, using a frame generation portion of the machine-learned VSR model, the forward predicted HR video frame for the current position in the time sequence; Further comprising: performing, by the computing system, the backward temporal prediction, applying, by the computing system, the Laplacian enhancement filter to the backward-intermediate HR video frame; after applying the Laplacian enhancement filter, processing, by the computing system, the backward-intermediate HR video frame and the current LR video frame to generate, using the frame generation portion of the machine-learned VSR model, the backward predicted HR video frame for the current position in the time sequence; 7. The computer-implemented method of claim 5 or 6, comprising:

8. 8. The computer-implemented method of claim 2, wherein the previous position in the time sequence comprises an immediately preceding position in the time sequence and the subsequent position in the time sequence comprises an immediately succeeding position in the time sequence.

9. The computer-implemented method of claim 1 , wherein the compressed video comprises a compressed video conferencing video stream.

10. 10. The computer-implemented method of claim 1, further comprising: using, by the computing system, the machine-learned VSR model to super-resolve an additional compressed video, wherein using, by the computing system, the machine-learned VSR model to super-resolve the additional compressed video comprises performing only forward temporal prediction on video frames of the additional compressed video.

11. 11. The computer-implemented method of claim 1, further comprising the step of: performing the method of claim 1 on a plurality of compressed training videos for a plurality of training iterations each, wherein the plurality of compressed training videos are compressed using the same compression algorithm.

12. The computer-implemented method of claim 11 , wherein the compression algorithm comprises an H.264 codec.

13. 11. The computer-implemented method of claim 1, further comprising the step of: performing the method of claim 1 on a plurality of compressed training videos for a plurality of training iterations each, the plurality of compressed training videos being compressed using two or more different compression algorithms.

14. 14. The computer-implemented method of claim 1, further comprising using, by the computing system, the machine-learned VSR model to super-resolve additional compressed video.

15. 15. The computer-implemented method of claim 14, wherein using the machine-learned VSR model to super-resolve the additional compressed video, by the computing system, comprises performing only forward temporal prediction on video frames of the additional compressed video.

16. one or more processors; processing a previous or subsequent LR video frame and a current LR video frame to generate a low resolution (LR) flow estimate and a high resolution (HR) flow estimate; warping the previous or subsequent LR video frame according to the LR flow estimation to generate a predicted LR video frame for a current position in a time sequence; warping a previous or subsequent HR video frame according to the HR flow estimate to generate an intermediate HR video frame for the current position in the time sequence; a flow estimation portion configured to a Laplacian enhancement portion configured to enhance the intermediate HR video frame; and a frame generation portion configured to process the intermediate HR video frame and the current LR video frame to generate a predicted HR video frame for the current position in the time sequence; A machine-learned video super-resolution (VSR) model comprising: instructions that, when executed by the one or more processors, cause a computing system to use the machine-learned VSR model to super-resolution compressed video. and one or more computer-readable storage media collectively storing: A computing system comprising:

17. 17. The computing system of claim 16, wherein the machine-learned VSR model is trained according to a method according to any one of claims 1 to 15.

18. A computer program arranged to cause a computer to carry out the method according to any one of claims 1 to 15.

Images