Video codec assisted real-time video enhancement using deep learning

Abstract

This application discloses video codec assisted real-time video enhancement using deep learning. Techniques related to accelerated video enhancement with deep learning selectively applied using video codec information are discussed. Such techniques include selectively applying a deep learning video enhancement network to decoded non-skip blocks in low quantization parameter frames, bypassing the deep learning network for decoded skip blocks in low quantization parameter frames, and applying non-deep learning video enhancement to high quantization parameter frames.

Description

Technical Field

[0001] This application relates to the field of video, and more specifically to video codecs using deep learning to assist in real-time video enhancement. Background Technology

[0002] In deep learning super-resolution, videos are upscaled using deep learning networks (e.g., convolutional neural networks trained on training videos and base real-world upscaled videos). Current deep learning-based video super-resolution (i.e., video upscaling) requires significant computational resources and memory bandwidth to achieve real-time performance (e.g., upscaling from 1080p to 4k at 60 fps). This requirement limits its widespread deployment on many resource-constrained or power-budget-constrained hardware platforms, such as laptops and tablets that only include integrated graphics cards.

[0003] Techniques to accelerate deep learning-based video super-resolution include simplifying the network topology of the deep learning networks employed. This involves reducing the number of layers, channels, connections between two consecutive layers, and the bit precision used to represent network weights and activations. Other techniques use low-rank approximations to reduce the complexity of the most computationally intensive layers, such as convolutional and fully connected layers. Finally, some networks reduce complexity by seeking temporal relevance via another neural network that predicts pixel-by-pixel motion vectors.

[0004] Complexity reduction techniques decrease the number of layers, channels, and / or bit precision to improve speed, but at the expense of video super-resolution quality. It's worth noting that super-resolution networks need to be sufficiently "deep" (i.e., maintaining at least a minimum number of network layers / channels / bit precision) to exhibit a significant quality improvement relative to traditional methods such as bicubic or Lanczos interpolation. The requirement for improved upsampling quality performance in deep networks limits the complexity that can be reduced. The same problem exists for low-rank approximation techniques. Finally, for time-based neural networks, the computational savings are very limited because the motion vectors between two frames are computed by another computationally and memory-intensive network.

[0005] Providing deep learning-based super-resolution for video can be advantageous, improving super-resolution quality and / or offering acceleration, reduced computational complexity, and lower memory costs. It is for these and other considerations that this improvement is necessary. As the desire to upgrade video becomes more widespread, this improvement may become critical. Summary of the Invention

[0006] A first aspect of this application discloses a system for providing adaptive enhanced video processing, comprising: a memory for storing a bitstream; and one or more processors coupled to the memory, the one or more processors being configured to: decode the bitstream to generate a first block and a second block of video frames, wherein the first block and the second block respectively include a first video coding mode and a second video coding mode; in response to the first video coding mode, apply a deep learning network to the first block to generate a first output block; in response to the second video coding mode, bypass the application of the deep learning network to the second block and extract a previous second output block for the second block; and generate an output video frame corresponding to the video frame, at least based on merging the first output block and the second output block into an output video frame.

[0007] A second aspect of this application discloses a method for providing adaptive enhanced video processing, comprising: decoding a bitstream to generate a first block and a second block of a video frame, wherein the first block and the second block respectively include a first video coding mode and a second video coding mode; in response to the first video coding mode, applying a deep learning network to the first block to generate a first output block; in response to the second video coding mode, bypassing the application of the deep learning network to the second block and extracting a previous second output block for the second block; and generating an output video frame corresponding to the video frame based at least on merging the first output block and the second output block into an output video frame. Attached Figure Description

[0008] The materials described herein are illustrated in the accompanying drawings by way of example, not limitation. For simplicity and clarity, the elements illustrated in the drawings are not necessarily drawn to scale. For example, the dimensions of some elements may be exaggerated relative to others for clarity. Additionally, reference numerals are repeated between drawings where deemed appropriate to indicate corresponding or similar elements. In the accompanying drawings: Figure 1 This is a schematic diagram of an example system for processing via selective application of deep learning networks; Figure 2 The illustration shows an example of block-by-block selective application of a deep learning network to frames with low quantization parameters; Figure 3 The illustration shows an example pixel value transfer process in a super-resolution scenario; Figure 4 The illustration shows a frame-by-frame selective application of a deep learning network to an I-frame. Figure 5 The illustration shows an example of frame-by-frame selective application of a deep learning network to frames with low quantization parameters; Figure 6 This is a flowchart illustrating an example process 600 for providing adaptive video enhancement processing based on frame-level quantization parameters and block-level video coding modes; Figure 7 This is a flowchart illustrating an example process for providing adaptive enhanced video processing; Figure 8 This is a schematic diagram of an example system for providing adaptive enhanced video processing; Figure 9 This is a schematic diagram of the example system; and Figure 10 The illustrations show example apparatuses arranged according to at least some implementations of this disclosure. Detailed Implementation

[0009] One or more embodiments or implementations will now be described with reference to the accompanying drawings. While specific configurations and arrangements are discussed, it should be understood that this is for illustrative purposes only. Those skilled in the art will recognize that other configurations and arrangements may be employed without departing from the spirit and scope of the description. It will be apparent to those skilled in the art that the techniques and / or arrangements described herein can also be used in a variety of other systems and applications different from those described herein.

[0010] While the following description illustrates various implementations that can manifest in architectures such as system-on-a-chip (SoC) architectures, the implementations of the technologies and / or arrangements described herein are not limited to specific architectures and / or computing systems, but can be implemented by any architecture and / or computing system for similar purposes. For example, the technologies and / or arrangements described herein can be implemented using various architectures employing, for example, multiple integrated circuit (IC) chips and / or packages, and / or various computing devices and / or consumer electronic (CE) devices such as set-top boxes, smartphones, etc. Furthermore, while the following description may elaborate on many specific details, such as logical implementations, types and interrelationships of system components, logical partitioning / integration choices, etc., the claimed subject matter can be implemented without such specific details. In other cases, some materials, such as control structures and complete software instruction sequences, may not be shown in detail to avoid obscuring the materials disclosed herein.

[0011] The materials disclosed herein can be implemented in hardware, firmware, software, or any combination thereof. The materials disclosed herein can also be implemented as instructions stored on a machine-readable medium that can be read and executed by one or more processors. A machine-readable medium can include any medium and / or mechanism for storing or transmitting information in a machine-readable form (e.g., a computing device). For example, a machine-readable medium can include read-only memory (ROM); random access memory (RAM); disk storage media; optical storage media; flash memory devices; and electrical, optical, acoustic, or other forms of propagation signals (e.g., carrier waves, infrared signals, digital signals, etc.).

[0012] The use of phrases such as "an implementation," "an implementation," or "an example implementation" in the specification indicates that the described implementation may include specific features, structures, or characteristics, but not every embodiment may necessarily include that specific feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same implementation. Additionally, when a specific feature, structure, or characteristic is described in conjunction with an embodiment, it is assumed that implementing that feature, structure, or characteristic in combination with other implementations (whether explicitly described herein or not) is within the knowledge of those skilled in the art.

[0013] This article describes methods, apparatuses, devices, computing platforms, and items related to real-time video super-resolution, and in particular to accelerating deep learning-based video super-resolution by utilizing metadata obtainable from video decoding.

[0014] In many situations, it may be desirable to apply super-resolution processing to video to upgrade and optionally improve the detail of lower-resolution video, thereby providing higher-resolution video or other enhancements. For the purposes of this document, the term super-resolution refers to an upgrade from a lower resolution to a higher resolution. The interchangeable terms resolution or size refer to the width multiplied by the height (or other suitable metric) of a video frame in pixels, where a higher resolution includes a larger number of pixels in the horizontal, vertical, or both dimensions. The resulting upgraded or super-resolution video, or other enhanced video, can be used in any application, such as displaying it to a user, artificial intelligence applications, etc. For example, the resulting upgraded or super-resolution video may be presented to a user, provided to another application, or stored in memory for future use at the device performing the super-resolution or at a remote device.

[0015] The techniques discussed in this paper provide a framework for accelerating any deep learning-based video enhancement. For example, deep learning video enhancement networks with any topology can be employed here. Deep learning-based video enhancement networks can be simplified (e.g., with a reduced number of layers, channels, number of connections between two consecutive layers, and / or bit precision) or not simplified. For the purposes of this paper, the term deep learning network or deep learning-based video enhancement refers to a network that utilizes an artificial neural network to perform machine learning, having several layers between the network's input and output layers, including deep learning networks, convolutional neural networks, etc. In some embodiments, a deep learning network includes multiple convolutional neural network layers. In some embodiments, a deep learning network is a convolutional neural network (CNN). For the purposes of this paper, the term CNN includes any network that employs one or more convolutional layers as part of the network. The term convolutional layer refers to a layer that employs multiple convolutional filters and optionally includes ReLU layers or pooling layers. This paper focuses on super-resolution applications for clarity of presentation, but any video enhancement including super-resolution, anti-aliasing, or noise reduction can be employed.

[0016] In some embodiments, the timing and other information from the decoding of the input video (i.e., information embedded in the video bitstream) are used to guide super-resolution or other enhancement processes (including the application of deep networks (video super-resolution networks)) to the input video. In some embodiments, for video frames with low quantization parameters (QP), when the current block being processed is encoded as an inter-frame mode residual without residuals in the bitstream, instead of applying deep learning super-resolution or other enhancement networks, the corresponding region in the previous high-resolution or enhanced output frame is used as a reference for that block to generate the final super-resolution or enhanced output. In some embodiments, when a frame has a high QP, the frame (i.e., all blocks of the frame) is upsampled or otherwise enhanced using interpolation techniques, thereby bypassing the deep learning network. In some embodiments, the interpolation or other enhancements can be advantageously implemented in hardware. In some embodiments, when a frame has a low QP, blocks other than those with inter-frame modes without residuals are upsampled or otherwise enhanced using deep learning networks. As described above, for those blocks with a residual-free inter-frame pattern (e.g., skipped blocks), the deep learning network is bypassed, and previously generated super-resolution pixel samples or otherwise enhanced pixel samples are extracted and used for that block. In such frames, those super-resolution blocks or enhanced blocks generated using the deep learning network and those generated by extracting from previously generated super-resolution or enhanced frames are then merged into a resulting super-resolution or enhanced frame.

[0017] The techniques discussed herein use metadata determined in the video decoding of an input video (e.g., via a video codec) to reduce the computational complexity of a deep learning network or an artificial intelligence-based workload, with little or no quality impact. Such techniques connect video compression and machine learning / deep learning-based video processing to achieve system performance optimization. Although mainly illustrated and discussed herein for super-resolution video processing, the techniques discussed can be applied to any suitable deep learning network-based video processing environment, such as a deep learning-based anti-aliasing network, a deep learning-based noise reduction network, etc. For example, the techniques discussed can be applied in a wide range of AI-based workloads to address computational / memory bandwidth bottlenecks, which is advantageous in network deployments on especially constrained computing platforms (e.g., low-end / mid-range GPU platforms).

[0018] Figure 1 FIG. 1 is a schematic diagram of an example system 100 arranged according to at least some implementations of the present disclosure for processing via selective application of a deep learning network. As Figure 1 shown, system 100 includes a video decoder 101, a QP comparison module 102 (labeled QP < TH?), a residual value comparison module 103 (labeled Residual = 0?), a hardware upsampler 104, a pixel value transfer module 105, a deep learning super-resolution network 106 (labeled AI-based super-resolution), and a frame merging module 107. For example, system 100 receives an input video bitstream 111 (representing a lower-resolution video) for super-resolution processing, and system 100 provides upsampling to a higher-resolution output video 118. Although illustrated for super-resolution processing, system 100 can be used in any suitable video deep learning application, such as anti-aliasing, noise reduction, etc., as further discussed below for Figure 6 FIG. 2.

[0019] System 100 can be implemented via any suitable device, such as a personal computer, laptop computer, tablet device, phablet, smartphone, digital camera, game console, wearable device, all-in-one device, two-in-one device, etc. or a platform such as a mobile platform, etc. For example, as used herein, a system, device, computer, or computing device can include any such device or platform. System 100 can include other modules or components not shown for clarity of presentation. The techniques presented for system 100 use existing metadata (i.e., QP, block motion vectors, and residual information) embedded in the input video bitstream 111 to adaptively select different upgrade (i.e., super-resolution) schemes for the application of the deep learning super-resolution network to all blocks of all frames of the decoded video to reduce computational complexity, with little or no noticeable quality loss.

[0020] The input video bitstream 111 can be any suitable bitstream, such as a standard bitstream. For example, bitstream 115 can conform to any of Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), VP9, ​​or AV1. Video decoder 101 receives the input video bitstream 111 and decodes it (by implementing the appropriate codec) to generate decoded video as video frames 113. Video frames 113 can include any suitable video frame, video frame, video frame sequence, frame group, multiple frame groups, video data, etc., at any suitable resolution. For example, the video can be a video graphics array (VGA), high definition (HD), full HD (e.g., 1080p), 4K resolution video, etc., and the video can include any number of video frames, video frame sequences, frames, frame groups, etc. For clarity of presentation, the techniques discussed herein are discussed in terms of video frames and their blocks. However, such video frames can be characterized as pictures, etc., and blocks can be characterized as coding units, maximum coding units, macroblocks, coding blocks, etc. For example, a picture or frame of color video data may include a luma plane or component and two chroma planes or components at the same or different resolutions relative to the luma plane. Video frame 113 (during encoding) can be divided into blocks of any size, which contain data corresponding to blocks of pixels. Such blocks may include data from one or more planes or color channels of pixel data.

[0021] In addition, video decoder 101 provides metadata including quantization parameters 112 (QP) and motion vectors and coding modes 114. Coding modes 114 may include frame-level coding information (e.g., frame types including I-frames, P-frames, etc.) and block-level coding information, including the block's coding mode, and, where applicable, block motion vectors and residual information, for use as described herein. For example, the metadata may include all such information, or only information relevant to selective deep learning network applications as described herein. For example, compared to a typical video decoder that only outputs video frames 113, video decoder 101 generates metadata via requests for target information, requests for metadata dumps of all such information, etc.

[0022] Video frame 113, or blocks thereof, is selectively provided to hardware upsampler 104, pixel value transfer module 105, and deep learning super-resolution network 106 for selective processing as described herein. For example, pixel samples of the (lower resolution) video frame 113 may be stored in memory and accessed for processing.

[0023] QP comparison module 102 receives QP 112, which can be a frame, slice, or block-level quantization parameter. In some embodiments, QP 112 is a frame-level quantization parameter, such that the selection made via QP comparison module 102 is applied at the frame level. QP comparison module 102 receives QP 112 and compares each quantization parameter with a threshold (TH). The threshold can be any appropriate value indicating high compression (e.g., low QP) versus low compression (e.g., high QP). For example, in some coding scenarios, the range of available quantization parameters can be from 1 (or 0) to 51, and the threshold can be in the range of approximately 26 to 34. In some embodiments, the threshold is established as a percentage of the range of quantization parameters of the relevant codec used for decoding the input video bitstream 111. In some embodiments, the threshold is in the range of 50% to 70% from the lowest available QP value to the highest QP value (e.g., for an example of 65%). In some embodiments, the threshold ranges from 55% to 65% of the lowest available QP value to the highest QP value. In some embodiments, the threshold ranges from 55% to 60% of the lowest available QP value to the highest QP value. It is worth noting that when the quantization parameter is greater than the threshold, the corresponding video frames are highly compressed and many texture details are lost. In this context, the implementation of the deep learning super-resolution network 106 does not justify the increased computational cost compared to the quality improvement of the hardware upsampler 104.

[0024] As shown in the figure, when the quantization parameter exceeds the threshold at QP comparison module 102, processing continues at hardware upsampler 104. As described above, hardware upsampler 104 implements interpolation-based upsampling to generate super-resolution block 115 at a resolution greater than that of video frame 113 and equal to the resolution of the resulting output video 118. Output video 118 can be at any suitable resolution greater than that of video frame 113, such as VGA resolution, HD resolution, full HD resolution, 4K resolution, 8K resolution, etc. Upsampling can be performed by any suitable factor, such as 2x in both the horizontal and vertical directions, or 4x in both the horizontal and vertical directions.

[0025] Although the discussion focuses on a hardware implementation, the interpolation described can be performed in software; however, providing interpolation-based upsampling in hardware is common in many systems and platforms. This interpolation-based upsampling can include any one or more suitable techniques, such as bilinear interpolation, bicubic interpolation, Lanzos interpolation, or others. Note that this technique can be distinguished from the upsampling provided by the deep learning super-resolution network 106, as no deep learning layers, convolutional layers, connection layers, etc., are applied. While super-resolution block 115 is shown as output from hardware upsampler 104 to frame merging module 107, in some embodiments, hardware upsampler 104 outputs a full super-resolution frame that does not need to be merged with other blocks. It is noteworthy that hardware upsampler 104 can be applied frame-by-frame based on frame-level QP, where interpolation-based upsampling is performed without considering the encoding mode of the frame's blocks.

[0026] When the quantization parameter does not exceed the threshold at QP comparison module 102, processing continues at residual value comparison module 103. Residual value comparison module 103 receives encoding mode 114. As described above, encoding mode 114 includes at least block-based motion vectors with residual information. Encoding mode 114 may also include other encoding mode information, such as indications of whether a block is an inter-frame or intra-frame block, the mode of an intra-frame block, etc., but such information is not strictly necessary for the processing discussed herein. The term residual information refers to whether any residuals have been encoded for the block (e.g., included in the input video bitstream 111), the number of encoded residuals, a skip flag, or similar information indicating whether residuals have been encoded for the block. The term residual refers to the value of a block determined based on subtracting, transforming, and quantizing the block with a reference block. It is noteworthy that some blocks are encoded with motion vectors (which may be zero MV) and have no residual information, such that during decoding by video decoder 101, the block is reconstructed only by accessing pixel samples of previously reconstructed frames (without adding residuals to the pixel samples). In some codecs, such blocks are characterized as skip blocks, skip mode blocks, or similar terms.

[0027] As shown in the figure, when a block has an inter-frame video coding mode (e.g., it is an inter-frame block) and the residual information is zero (e.g., there is no coding residual for the block), processing continues at the pixel value transfer module 105, where the deep learning super-resolution network 106 is bypassed again, and the corresponding super-resolution block 116 is generated by extracting previously generated pixel samples using the block's motion vectors. This pixel sample extraction technique is discussed further in this paper. In short, the block's motion vectors (as indicated by coding mode 114) are used to access pixel samples of previously generated super-resolution frames of the output video 118, and these pixel samples are provided to the frame merging module 107 as super-resolution block 116. It is worth noting that super-resolution block 116 is at the desired resolution of the output video 118. Furthermore, since both the video decoder 101 and the deep learning super-resolution network 106 are block-based (e.g., processed in a block-by-block manner), processing can be selected block-by-block. This processing advantageously bypasses the deep learning super-resolution network 106 at the block level, and instead uses a computationally lightweight pixel transfer process to generate super-resolution blocks 116. This technique has been found to provide sufficient image quality with little or no quality loss, while advantageously reducing processing time.

[0028] Returning to the residual comparison module 103, processing continues at the deep learning super-resolution network 106 when the block has a residual corresponding to a residual value (e.g., in the input video bitstream 111) or when the block is intra-coded (which typically also includes residual values). Here, the deep learning super-resolution network is applied to the block to generate a super-resolution block 117 of the resolution of the output video 118. In some embodiments, processing continues at the deep learning super-resolution network 106 when the block is not in skip mode. In some embodiments, processing continues at the deep learning super-resolution network 106 when the block is intra-frame processed. In some embodiments, processing continues at the deep learning super-resolution network 106 when an MV is provided and one or more residual values ​​are provided for the block.

[0029] The deep learning super-resolution network 106 can apply any suitable AI-based or deep learning-based super-resolution processing. In some embodiments, the deep learning super-resolution network 106 applies one or more convolutional layers. Notably, the deep learning super-resolution network 106 applies deep learning layers, convolutional layers, connection layers, or similar deep learning architecture components. Furthermore, the deep learning super-resolution network 106 is only applied to those blocks with low QP and residual values. This technique restricts the application of the deep learning super-resolution network 106 to those blocks that would most benefit from the expensive computations advantageously applied to it.

[0030] As shown in the figure, frame merging module 107 receives super-resolution blocks 116, 117, and, if applicable, 115, to generate frames for output video 118 by merging the super-resolution blocks into a full super-resolution frame of output video 118. In some embodiments, frame merging module 107 also applies filtering, such as deblocking filtering or other image enhancement techniques. As described above, output video 118 can be used in various scenarios, such as displaying to a user or for use in other applications. Output video 118 can be transmitted via a communication link to a display of system 100, to the memory of system 100, to another device, etc.

[0031] The techniques discussed in this paper offer several advantages, including low overhead (since all the information used to control super-resolution processing is required by the video decoder), acceleration of super-resolution processing by leveraging the characteristics of video content (acceleration depends on content type: without skip blocks, performance is roughly the same as selectively applying the network, where 45%–90% performance gains can be achieved for game and screen content), no conflict with other performance enhancements made via the network, and support for a wide range of codecs (including AVC, HEVC, VVC, and others).

[0032] Furthermore, the techniques discussed can be used in conjunction with region of interest (ROI) video processing techniques. Notably, ROI-based processing can be used in video compression or processing to save bitrate and / or reduce computational resource usage. In some embodiments, video frames are divided into more important ROI regions and less important non-ROI regions, with more bits utilized and / or special processing performed in the ROI regions. The techniques discussed provide joint system-level performance optimization between video decoders and AI-based (e.g., deep learning-based) video processing workloads. It is noteworthy that all regions are equally important in the techniques discussed, and the same processing choices are applied to both ROI and non-ROI regions. As a result, the techniques discussed can be used with ROI-based techniques such as ROI-based codecs. For example, it can be utilized for… Figure 1 The techniques discussed elsewhere in this paper are used to process all blocks of a video frame, so that regions of interest and non-regions of interest are processed in the same way.

[0033] Figure 2 The illustration shows an example block-by-block selective application of a deep learning network arranged according to at least some implementations of this disclosure to frames with low quantization parameters. Figure 2In the example, video frame 113 includes a lower resolution video frame 201 with a frame-level quantization parameter 211. Frame 201 is indicated as a P-frame 251, but may have any suitable frame type. As mentioned above, the frame-level quantization parameter 211 is less than a threshold TH, as determined by the QP comparison module 102. Furthermore, video frame 201 is divided into any number of blocks, as indicated by the partitions provided by the input video bitstream 111. That is, the partition information provided by the input video bitstream 111 corresponds to video frame 201. The blocks of video frame 201 include block 202 with a specific video coding mode 203 and block 212 with different video coding modes 213. Notably, video coding mode 203 indicates that block 202 has a motion vector 204 and zero residual coefficients 205, such that block 202 is a skipped block. In some embodiments, video coding mode 203 indicates a skip flag and a motion vector (which may be a zero motion vector). Similarly, video coding mode 213 indicates that block 212 has motion vectors 214 and residual coefficients 215 (21 residual coefficients in this example). For the purposes of this document, video coding may include information required for block decoding, such as intra-frame mode or inter-frame mode, motion vectors, skip flag or no skip flag, intra-frame mode (based on the available intra-frame modes of the codec), and so on.

[0034] In response to video coding mode 203, pixel value transfer module 105 is implemented for block 202. For example, in response to video coding mode 203 indicating a skip mode, pixel value transfer module 105 is implemented for block 202, bypassing deep learning super-resolution network 106. As shown, pixel value transfer module 105 extracts pixel samples to generate a corresponding super-resolution block 242 (or output block) for block 202, such that super-resolution block 242 is at a resolution 231 that is larger than resolution 221 and equal to the resolution of output frame 208. Super-resolution block 242 can be generated using any one or more suitable techniques, such as those described herein. Figure 3 Those that are discussed.

[0035] Furthermore, in response to video coding mode 213, deep learning super-resolution network 106 is applied to block 212 to generate super-resolution block 232 (or output block) for block 212, such that super-resolution block 232 is also at resolution 231. For example, in response to video coding mode 213 indicating a non-skip mode (any type: intra-frame mode, inter-frame mode with residuals, etc.), deep learning super-resolution network 106 is implemented without pixel value shifting. As described above, deep learning super-resolution network 106 can implement any suitable deep learning network, at the cost of increased computation and memory usage, to improve the upsampling quality of block 212. Also, for... Figure 2As shown, blocks 202 and 212 can have any suitable size. Although illustrated with respect to block 202 being larger than block 212, they can have the same size, or block 212 can be larger than block 213. Furthermore, although illustrated with respect to square blocks, blocks 202 and 212 can have any suitable shape, such as rectangles.

[0036] For example, for Figure 1 The super-resolution blocks 232 and 242, along with other super-resolution blocks generated by the pixel value transfer module 105, the deep learning super-resolution network 106, and optionally the hardware upsampler 104, are received by frame merging, which combines the super-resolution blocks into a super-resolution output frame 208 of resolution 231. For example, the super-resolution output frame 208 could be a frame of the output video 118. Note that video frame 201 and super-resolution output frame 208 are temporally related, meaning the two frames have the same temporal instance in the video stream. As further discussed below, the pixel value transfer module 105 can access previously upsampled frames to generate super-resolution block 242.

[0037] Figure 3 An example pixel value transfer process 300 is illustrated in a super-resolution scenario arranged according to at least some implementations of the present disclosure. For example, the pixel value transfer process 300 may be performed by a pixel value transfer module 105 as described herein. Notably, the pixel value transfer process 300 provides a low-complexity pixel value transfer technique to generate super-resolution (high-resolution) blocks for the current frame based on pixel samples extracted from previously processed frames.

[0038] As shown in the figure, the lower-resolution video frame 201 is in a video sequence 340 containing lower-resolution frames 221, including video frame 201, video frame 311, etc. For example, video sequence 340 could be a video sequence of video frame 113. As described above, based on video frame 113, a corresponding higher-resolution output video 118 is generated, which includes super-resolution frame 208 (time-corresponding to video frame 201), super-resolution frame 301 (time-corresponding to video frame 311), etc., such that each video frame 113 has a corresponding super-resolution video frame in the super-resolution sequence 350 with the same time instance, timestamp, and time position. Figure 3 For clarity, only the left portion of super-resolution frames 208 and 301 is shown. Note that the first video frame of video frame 113 is encoded as an intra-frame (as shown for...). Figure 4 As mentioned above, since the intraframe has a QP less than TH and no skipped blocks, all blocks (and all pixel samples) of the intraframe are processed by the deep learning super-resolution network 106 to generate the corresponding super-resolution frame.

[0039] As described above, block 202 has a corresponding motion vector 204, which has x and y components (mv x , mv y Furthermore, block 202 has a width of 308, w2 and a height of 307, h2, and is located at a horizontal position of 305, x2 and a vertical position of 306, y2 within video frame 201. It is worth noting that a block (or other rectangular partition) of pixel samples within video frame 201 can be represented by a 4-tuple (x, y, w, h), where (x, y) provides the horizontal and vertical components of the top-left corner of the block, respectively, and (w, h) provides the block width and height, respectively.

[0040] Therefore, the pixel samples of block 202 are provided by the 4-tuple (x2, y2, w2, h2). As mentioned above, block 202 is a skip mode with motion vector 204 but no residual coding (e.g., making the residual information equal to 0 before the deblocking filter is applied during decoding). In decoding, based on equation (1), block 202 is copied directly from block 341 of video frame 311 (video frame 311 may be before or after video frame 201 in terms of display order and time, but must be before video frame 201 in terms of encoding order and time) via decoding operation 334, as shown: (1) In this context, block 341 in video frame 311 has a width of 308, w2 and a height of h2307, and is located at a horizontal position of 323, x1 and a vertical position of 333, y1 within video frame 201. Note that equation (1) can also be used to locate blocks for pixel transfer in an enhanced frame when super-resolution processing is not performed as described herein.

[0041] Moving on to the generation of super-resolution block 242 in super-resolution frame 208, pixel value transfer module 105 locates the previous super-resolution block 342 in the previous super-resolution frame 301 via positioning operation 302. In this context, "previous" means that super-resolution block 342 and super-resolution frame 301 were generated before the generation of super-resolution block 242, but it does not necessarily mean that super-resolution frame 301 is displayed before super-resolution frame 208.

[0042] As shown in the figure, in order to provide pixel samples of super-resolution block 242 within super-resolution video frame 208, the pixel samples of super-resolution block 242 are copied via copy operation 335 and merged into super-resolution video frame 208. The pixel samples of super-resolution block 242 are represented by a 4-tuple (X2, Y2, W2, H2), such that super-resolution block 242 has a width of 354, W2, and a height of 353, H2, and is located at a horizontal position of 355, X2, and a vertical position of 356, Y2 within super-resolution video frame 208. For example, the 4-tuple (X2, Y2, W2, H2) of super-resolution block 342 can be a scaled version of the 4-tuple (x2, y2, w2, h2) of block 202, such that (X2, Y2, W2, H2) = (sx2, sy2, sw2, sh2).

[0043] It is worth noting that the pixel value transfer module 105 locates the super-resolution block 342 based on the size and position of block 202, motion vector 204, and a scaling factor s within the super-resolution frame 301 at a certain size and position. This scaling factor s matches the upsampling scaling from video frame 201 to super-resolution frame 208. For example, for a 2x scaling in both the horizontal and vertical directions, s=2; for a 4x scaling in both the horizontal and vertical directions, s=4, and so on. In the illustrated example, the upsampling is the same in both the horizontal and vertical directions (s=s). h =s v In some embodiments, this upsampling factor can be different (s h ≠s v In some embodiments, upsampling is not provided in either the horizontal or vertical direction. h =1 or s=1 v In this context, a vertical scaler is applied in the vertical direction. v And apply horizontal scaling subs in the vertical direction. h .

[0044] As described above, the pixel samples of super-resolution block 342 are copied via copy operation 335 and merged into super-resolution video frame 208 as super-resolution block 242. The pixel samples of super-resolution block 342 in super-resolution video frame 301 are provided by 4-tuples (X1, Y1, W1, H1), such that super-resolution block 342 has a width 314, W1, and a height 313, H1, and is located at a horizontal position 311, X1 and a vertical position 312, Y1 within super-resolution video frame 301, where the 4-tuple values ​​(X1, Y1, W1, H1) are provided using equation (2): (2) Where horizontal position 311 X1 is the horizontal position x2 of block 202, which is the horizontal component mv of motion vector 204. x Adjusted and scaled by scaling factor s, vertical position 312 Y1 is the vertical position y2 of block 202, which is the vertical component mv of motion vector 204. y The values ​​are adjusted and scaled by a scaling factor s, where the width 314 W1 is the width W2 of block 202 scaled by the scaling factor s, and the height 313 H1 is the height h2 of block 202 scaled by the scaling factor s. Note that if the horizontal position 311 X1 or the vertical position 312 Y1 is determined to be fractional coordinates, image interpolation is applied to generate pixel sample values ​​using any one or more appropriate techniques. In some embodiments, bilinear interpolation is used.

[0045] Figure 4 The illustration shows an example frame-by-frame selective application of a deep learning network arranged according to at least some implementations of this disclosure to I-frame 401. For example... Figure 4 As shown, video frame 113 may include any number of I-frames (inner frames), including I-frame 401 at resolution 221. It is worth noting that I-frame 401 is compressed without referencing any other frame of video frame 113, and therefore does not include blocks with motion vectors and zero residual coefficients (or no residual information). Since I-frame 401 is typically encoded with low quantization parameters (to provide a high-quality frame, both for presentation to the user and for reference when encoding subsequent frames), all blocks 421 of I-frame 401, including block 402, are upsampled via deep learning super-resolution network 106 to generate super-resolution output frame 408 at resolution 231.

[0046] As shown in the figure, block 402 includes a video coding mode 403, which can be any suitable intra-frame coding mode, such as corner mode, planar mode, or DC mode. However, regardless of the video coding mode 403, block 402 and all blocks 421 of I-frame 401 are upsampled (i.e., super-resolution processing) using a deep learning super-resolution network 106. This processing provides high-quality upsampling for I-frame 401. In some embodiments, the super-resolution output frame 408 corresponding to I-frame 401 is a reference frame (e.g., super-resolution frame 301) for the pixel value transfer module 105 to extract super-resolution pixel samples. By providing high-quality upsampling for all blocks 421 of I-frame 401, high-quality reference pixels are provided in some scenarios to improve performance.

[0047] Figure 5 The illustration shows an example frame-by-frame selective application of a deep learning network arranged according to at least some implementations of this disclosure to a low-quantization parameter frame 501. For example... Figure 5 As shown, video frame 113 may include any number of high QP 511 frames, such as frame 501 of resolution 221, such that the high QP 411 is greater than the threshold TH applied to QP comparison module 102. It is worth noting that frame 501 is compressed with high QP 511, making it highly likely that details and textures will be lost in the reconstruction of frame 501. As described above, since the high QP 511 frame 501 has a QP exceeding the threshold TH, all blocks 521 of frame 501, including block 502, are upsampled via interpolation techniques, for example, applied by hardware upsampler 104, to generate a super-resolution output frame 508 of resolution 231.

[0048] Block 502 includes video coding mode 503, which can be any suitable coding mode, such as an intra mode, an inter mode with or without coding residuals, etc. However, regardless of the video coding mode 503, all blocks 521 of block 502 and frame 501 are upsampled (i.e., interpolated) using hardware upsampler 104. This process provides frame 501 with fast and hardware-implemented upsampling. It is worth noting that the output frame 508 is unlikely to be a reference frame for the pixel value transfer module 105 to extract super-resolution pixel samples, thus the impact of lower-quality upsampling is relatively small. Furthermore, due to the lack of detail and texture, the difference in quality between the upsampling provided by the deep learning super-resolution network 106 and interpolation techniques may be small (and in some cases, interpolation may outperform the deep learning super-resolution network 106). Therefore, the high computational cost of applying the deep learning super-resolution network 106 to frame 501 at a very low loss of quality is mitigated.

[0049] Figure 6 This is a flowchart illustrating an example process 600 arranged according to at least some implementations of the present disclosure for providing adaptive video enhancement processing based on frame-level quantization parameters and block-level video coding modes. Process 600 may include, for example... Figure 6 One or more operations 601-612 are shown. Process 600 can be performed by any apparatus or system discussed herein to process the decoded video.

[0050] Processing begins at operation 601, where the bitstream is decoded to generate a video sequence comprising any number of video frames. Furthermore, this decoding generates metadata including quantization parameters (frame level, slice level, and / or block level) and the video coding mode of the block of video frames. It is noteworthy that the video coding mode of a block may include any or all the information required for the block's decoding. For example, the video coding mode of a block may include whether the block is a skipped block (i.e., an inter-frame block with motion vectors and no residuals), a non-skipped inter-frame block (i.e., an inter-frame block with motion vectors and residuals), an intra-frame block, etc., or information that can be used to determine the video coding mode. For the purposes of this document, terms such as block includes, contains, or has a coding mode or video coding mode, or similar terms, refer to the video coding mode that corresponds to the block and is used to encode the block (remotely) and decode the block (locally, at the video decoder).

[0051] It is worth noting that the video frames to be decoded will be enhanced in some way, such as super-resolution processing (e.g., upsampling), anti-aliasing (e.g., to smooth edges), noise reduction (e.g., to remove unwanted noise), or other image processing. The techniques discussed in process 600 adaptively apply this processing by applying a deep learning network to samples of the decoded video frames, by extracting previously processed samples, or by applying a scheme based on a non-deep learning network (optionally applied in hardware) to samples of the decoded video frames. In any case, the output block of the processed block is provided to be merged into the output frame. This anti-aliasing, noise reduction, or other video frame enhancement processing is performed similarly to the techniques discussed above for super-resolution processing.

[0052] Processing continues at operation 602, where video frames from the video sequence decoded at operation 601 are selected for processing. The video frames can be selected in any order, such as the display order of the video sequence.

[0053] Processing continues at decision operation 603, where it is determined whether the frame-level QP of the selected frame is less than a threshold. This threshold can be any suitable value and may depend on the codec used in operation 601. The threshold can be a percentage of the range of quantization parameters of the relevant codec used for decoding the bitstream 111, such as 50% to 70% from the lowest available QP value to the highest QP value, 55% to 65% from the lowest available QP value to the highest QP value, or 55% to 60% from the lowest available QP value to the highest QP value. Furthermore, this threshold may vary depending on any enhancement processing performed in process 600.

[0054] If the QP reaches or exceeds the threshold, processing continues at operation 604, where a first video enhancement is applied to the current frame (i.e., all blocks of the current frame), optionally applied in hardware. The first video enhancement is based on a non-deep learning network and is less computationally demanding than the processing discussed below for operation 607. In some embodiments, the first video enhancement processing is performed in hardware. For example, the first video enhancement processing can be a conventional process accelerated in hardware on a single integrated circuit. In some embodiments, the first video enhancement processing does not employ any deep learning layers, convolutional layers, connection layers, etc. The first video enhancement processing can be linear or non-linear, but is characterized by not applying pre-trained parameters trained using deep learning. The resulting output frame (i.e., a super-resolution frame, an anti-aliasing frame, a noise-reduced frame, etc.) is output and processing continues at decision operation 610 as described below. For example, the resulting output frame can be output to present to a user, output to memory for future use, or output to another device.

[0055] If the QP is less than the threshold at decision operation 603, processing continues at operation 605, where blocks of the current video frame are selected for processing. The blocks of the current frame can be processed in any order, such as raster scan order. While process 600 illustrates frame-level QP and block-level video coding mode processing, in other embodiments, the QP-based decision discussed may be based on slice-level QP, maximum coding unit QP, or block-level QP.

[0056] Processing continues at decision operation 606, where it is determined whether the selected block is a skipped block (i.e., includes a skipped mode or has a corresponding skipped mode). As mentioned above, a skipped block is characterized by being encoded and decoded as an inter-frame block, having motion vectors (which can be zero motion vectors), and having no residuals. Note that not all blocks selected at operation 605 and processed as described herein are of the same size. Instead, this block size can correspond to the coded blocks, such that each coded block is partitioned into a specific size based on the optimal partition of the larger block (e.g., the maximum coding unit) determined during encoding.

[0057] If the current block does not have a skip mode (i.e., it has an intra-frame mode or an inter-frame mode with residuals), processing continues at operation 607, where a second video enhancement process is applied, employing a deep learning network. As described above, a deep learning network is characterized by utilizing an artificial neural network to perform machine learning, which has several layers between its input and output layers, such that these layers may include convolutional layers, fully connected layers, and so on. In some embodiments, the deep learning network employs pre-trained network parameters, such as convolutional filter weights, activation values, and so on. A deep learning network can be characterized by the task it performs, including deep learning super-resolution networks, deep learning anti-aliasing networks, deep learning denoising networks, and so on. A deep learning network can have any suitable architecture or topology. Furthermore, processing blocks of the same size in operation 607 requires more computational resources (e.g., in terms of processor time required for the same processor, the number of basic operations, the amount of memory used, etc.). The processing performed in operation 607 requires more computational resources than the processing performed in operation 604 and is expected to provide higher quality (both objectively and subjectively).

[0058] Returning to decision operation 606, if the current block has a skip mode, processing continues at operation 608, where the second video enhancement process is bypassed (as is the first video enhancement process), and for the current block, the previously processed block is extracted and copied. In some embodiments, the previous block was processed using the second video enhancement process. In some embodiments, the previous block is part of a previously processed frame (e.g., a super-resolution frame, an anti-aliasing frame, a noise-reduced frame, etc.), and the pixel samples of the previous block are located and copied based on the motion vector corresponding to the current block. This location and copying of the super-resolution block is discussed here with respect to equation (2). For blocks of the same size and the previous block, this location and copying is performed as described with respect to equation (1).

[0059] As described herein, in video frames using Region of Interest (ROI) coding, ROI blocks and non-ROI blocks are processed in the same manner. It is noteworthy that the techniques discussed in process 600 are applied equivalently to both ROI blocks and non-ROI blocks, and depending on the video coding mode of the block, skip-mode ROI blocks and skip-mode non-ROI blocks are processed in the same manner (e.g., performing previous block extraction), and similarly, non-skip-mode ROI blocks and non-skip-mode non-ROI blocks are processed in the same manner (e.g., performing deep learning network processing).

[0060] Processing continues from operation 607 or 608 for the current block at decision operation 609, where it is determined whether the last block of the current frame is being processed. If not, processing continues as described in operations 605-609 until all blocks have been processed.

[0061] If so, processing continues at operation 610, where blocks generated by applying a second video enhancement process using a deep learning network and blocks that bypass the second video enhancement process and are instead extracted from previously enhanced video frames are merged into the current enhanced video frame. This merger may also include deblocking filtering or other processing. The resulting video frame is output as described for operation 604.

[0062] Processing continues at decision operation 611, where it is determined whether the last frame of the video sequence is being processed. If not, processing continues as described in operations 602-610 until all frames have been processed. If yes, processing ends at end operation 612.

[0063] Figure 7 This is a flowchart illustrating an example process 700 for providing adaptive enhanced video processing, arranged according to at least some implementations of this disclosure. Process 700 may include, for example... Figure 7 One or more operations 701-704 are shown. Process 700 may form at least a portion of a video enhancement process. As a non-limiting example, process 700 may form at least a portion of a video process performed by any device or system as described herein, such as system 100. Furthermore, reference will be made herein to... Figure 8 The system 800 is used to describe process 700.

[0064] Figure 8 This is a schematic diagram illustrating an example system 800 for providing adaptive enhanced video processing, arranged according to at least some implementations of this disclosure. Figure 8As shown, system 800 includes a central processing unit 801, a video processor 802, a memory 803, and a video enhancement processor 804. Also as shown, video processor 802 may include or implement a video decoder 101, a QP comparison module 102, a residual value comparison module 103, a pixel value transfer module 105, a deep learning network 806, and a frame merging module 107. For example, deep learning network 806 may implement any deep learning network discussed herein, such as a deep learning super-resolution network, a deep learning anti-aliasing network, a deep learning denoising network, etc. Furthermore, video enhancement processor 804 may provide the same processing using only different non-deep learning-based, non-AI-based techniques as described herein. As shown, video enhancement processor 804 may be implemented separately from video processor 802. In some embodiments, video enhancement processor 804 is implemented as part of video processor 802 (e.g., as a hardware component of video processor 802). Furthermore, in the example of system 800, memory 803 may store video data or related content, such as video frame data, video block data, video encoding modes, deep learning network parameters, frame level QP, and / or any other data as described herein.

[0065] As shown in the figures, in some embodiments, the video decoder 101, QP comparison module 102, residual value comparison module 103, pixel value transfer module 105, deep learning network 806, and frame merging module 107 are implemented via the video processor 802. In other embodiments, one or more of the video decoder 101, QP comparison module 102, residual value comparison module 103, pixel value transfer module 105, deep learning network 806, and frame merging module 107 are implemented via the central processing unit 801 or another processing unit such as an image processor, graphics processor, etc.

[0066] Video processor 802 may include any number and type of video, image, or graphics processing units that provide the operations described herein. Such operations may be implemented via software or hardware, or a combination thereof. For example, video processor 802 may include circuitry dedicated to manipulating video frames, video frame data, etc., obtained from memory 803. Central processing unit 801 may include any number and type of processing units or modules that can provide control and other high-level functions and / or provide any of the operations described herein for system 800. Memory 803 may be any type of memory, such as volatile memory (e.g., Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), etc.) or non-volatile memory (e.g., flash memory, etc.), etc. In a non-limiting example, memory 803 may be implemented as cache memory.

[0067] In this embodiment, one or more of the video decoder 101, QP comparison module 102, residual value comparison module 103, pixel value transfer module 105, deep learning network 806, and frame merging module 107 are implemented via an execution unit (EU). The EU may include, for example, programmable logic or circuitry, such as one or more logic cores providing a wide variety of programmable logic functions. In this embodiment, one or more of the video decoder 101, QP comparison module 102, residual value comparison module 103, pixel value transfer module 105, deep learning network 806, and frame merging module 107 are implemented via dedicated hardware such as fixed-function circuitry. Fixed-function circuitry may include dedicated logic or circuitry and may provide a set of fixed-function entry points that can be mapped to dedicated logic for a fixed purpose or function.

[0068] Return to Figure 7 The discussion continues, with process 700 beginning with operation 701, where the bitstream is decoded to generate first and second blocks of video frames having first and second video encoding modes. This decoding can be based on any suitable video codec, and the first and second encoding modes can be obtained via a decoding parameter memory request, or similar method.

[0069] Processing continues at operation 702, where a deep learning network is applied to a first block in response to a first coding mode to generate a first output block. In some embodiments, the first coding mode is any coding mode other than a skipped mode (i.e., inter-frame coding without residual information). The deep learning network is applied in response to the first block not being a skipped block. In some embodiments, the first block is intra-frame coded or inter-frame coded and has encoded residual information. The deep learning network can be applied to provide any suitable video enhancement processing, including super-resolution processing, anti-aliasing processing, or noise reduction processing. In some embodiments, the deep learning network is a deep learning super-resolution network and the first output block is at a second resolution greater than the first resolution of the video frame decoded in operation 701. In some embodiments, the deep learning network is a deep learning anti-aliasing network. In some embodiments, the deep learning network is a deep learning noise reduction network. The deep learning network can have any suitable architecture or topology.

[0070] Processing continues at operation 703, where, in response to a second encoding mode, the deep learning network is applied to the second block and a previous second output block is extracted for the second block. In the context of super-resolution processing, the second output block is at a second resolution larger than the first resolution of the video frame. Furthermore, in these super-resolution processing scenarios, extracting the second output block includes extracting the second output block using motion vectors based on previously generated second super-resolution video frames that are temporally adjacent to the video frame. For example, a temporally adjacent super-resolution video is for a time instance different from the time instance of the video frame. In some embodiments, extracting the second output block includes accessing pixel samples of the second super-resolution video frame via a position at the horizontal location of the second block adjusted based on motion vectors and scaled using a scaling factor, and having a size based on the width of the second block scaled using a scaling factor.

[0071] Processing continues at operation 704, where an output video frame corresponding to the video frame is generated, based at least on merging the first and second output blocks into the output video frame. For example, the output video frame has the same temporal instance as the video frame but is enhanced in terms of upsampling, anti-aliasing, or noise reduction. In the context of super-resolution processing, the video frame is at a first resolution, and the output video frame is at a second resolution greater than the first resolution.

[0072] In some embodiments, process 700 further includes decoding the bitstream to generate a third block of the second video frame, such that the video frame has a first quantization parameter and the second video frame has a second quantization parameter less than the first quantization parameter, and applying non-deep learning-based video processing to the third block in response to the second quantization parameter being greater than a threshold to generate a third output block enhanced relative to the third block. In the context of super-resolution processing, the third block is at a first resolution, and the third output block is at a second resolution greater than the first resolution. In some embodiments, the non-deep learning-based video processing includes video upscaling via interpolation of the third block to generate the third output block. In some embodiments, the interpolation includes one of bilinear interpolation, bicubic interpolation, or Lanzos interpolation, and wherein the deep learning network includes at least one convolutional layer. In some embodiments, applying the deep learning network to the first block, bypassing the application of the deep learning network to the second block, and extracting the previous second output block are in response to the first quantization parameter being less than a threshold. In some embodiments, the bitstream is decoded to generate metadata including first and second video coding modes and first and second quantization parameters.

[0073] In some embodiments, process 700 further includes decoding the bitstream to generate third and fourth blocks, respectively comprising first and second video coding modes, such that the first and second blocks are regions of interest (ROI) blocks and the third and fourth blocks are regions of non-ROI blocks; applying a deep learning network to the third block in response to the first coding mode to generate a third output block at a second resolution; and bypassing the application of a deep learning super-resolution network to the fourth block in response to the second coding mode and extracting a previous fourth output block at a second resolution for the fourth block, such that generating an output video frame includes merging the third and fourth output blocks into the output video frame. For example, the processes described can be performed in the same manner in the ROIs and non-ROIs of the video frame.

[0074] Process 700 can be repeated serially or in parallel an arbitrary number of times for any number of video blocks, video frames, video segments, etc. As described above, process 700 can provide adaptive enhanced video processing based on frame-level quantization parameters and block coding modes.

[0075] The various components of the systems described herein can be implemented in software, firmware, and / or hardware and / or any combination thereof. For example, the various components of the systems or apparatus discussed herein may be provided at least in part by the hardware of a computer system-on-a-chip (SoC), such as that present in a computing system (e.g., a smartphone). Those skilled in the art will recognize that the systems described herein may include additional components not depicted in the corresponding figures. For example, the systems discussed herein may include additional components not depicted for clarity, such as bitstream multiplexer or demultiplexer modules, etc.

[0076] While the implementation of the example procedures discussed herein may include performing all the operations shown in the illustrated order, this disclosure is not limited thereto, and in various examples, the implementation of the example procedures herein may include only a subset of the operations shown, including operations performed in a different order than illustrated, or include additional operations.

[0077] Furthermore, any one or more operations discussed herein may be performed in response to instructions provided by one or more computer program products. Such program products may include signal-bearing media providing instructions that, when executed by, for example, a processor, provide the functionality described herein. Computer program products may be provided in any form of one or more machine-readable media. Thus, for example, a processor including one or more graphics processing units or processor cores may perform one or more blocks of the example processes herein in response to program code and / or instructions or instruction sets delivered to the processor by one or more machine-readable media. Generally, machine-readable media may deliver software in the form of program code and / or instructions or instruction sets that cause any apparatus and / or system described herein to implement at least some portions of the operations discussed herein and / or any portion of the apparatus, system, or any module or component as described herein.

[0078] As used in any implementation described herein, the term "module" refers to any combination of software logic, firmware logic, hardware logic, and / or circuitry configured to provide the functionality described herein. Software may be implemented as a software package, code, and / or instruction set or instructions, and "hardware" as used in any implementation described herein may, for example, individually or in combination, include hardwired circuitry, programmable circuitry, state machine circuitry, fixed-function circuitry, execution unit circuitry, and / or firmware storing instructions executed by programmable circuitry. Modules may be implemented collectively or individually as circuitry forming part of a larger system, such as integrated circuits (ICs), system-on-a-chip (SoCs), etc.

[0079] Figure 9This is a schematic diagram of an example system 900 arranged according to at least some implementations of this disclosure. In various implementations, system 900 may be a mobile system, although system 900 is not limited to this scenario. For example, system 900 may be included in the following: personal computer (PC), laptop computer, ultra-portable laptop computer, tablet device, touchpad, portable computer, handheld computer, PDA, personal digital assistant (PDA), cellular phone, combined cellular phone / PDA, television, smart device (e.g., smartphone, smart tablet, or smart TV), mobile internet device (MID), messaging device, data communication device, camera (e.g., point-and-shoot camera, super zoom camera, digital single-lens reflex (DSLR) camera), etc.

[0080] In various implementations, system 900 includes a platform 920 coupled to display 902. Platform 902 may receive content from content devices, such as one or more content serving devices 930 or one or more content delivery devices 940 or other similar content sources. A navigation controller 950, including one or more navigation features, may be used to interact with, for example, platform 902 and / or display 920. Each of these components is described in more detail below.

[0081] In various implementations, platform 902 may include any combination of chipset 905, processor 910, memory 912, antenna 913, storage device 914, graphics subsystem 915, application 916, and / or radio device 918. Chipset 905 provides communication between processor 910, memory 912, storage device 914, graphics subsystem 915, application 916, and / or radio device 918. For example, chipset 905 may include a storage adapter (not shown) capable of providing communication with storage device 914.

[0082] The processor 910 can be implemented as a Complex Instruction Set Computer (CISC) or Reduced Instruction Set Computer (RISC) processor, an x86 instruction set compatible processor, a multi-core processor, or any other microprocessor or central processing unit (CPU). In various implementations, the processor 910 can be one or more dual-core processors, one or more dual-core mobile processors, and so on.

[0083] The memory 912 may be implemented as a volatile memory device, such as, but not limited to, random access memory (RAM), dynamic random access memory (DRAM), or static RAM (SRAM).

[0084] Storage device 914 can be implemented as a non-volatile storage device, such as, but not limited to, disk drives, optical disc drives, tape drives, internal storage devices, attached storage devices, flash memory, battery-backed SDRAM (synchronous DRAM), and / or network-accessible storage devices. In various implementations, such as when including multiple hard disk drives, storage device 914 may include techniques to add storage performance enhancement protection for valuable digital media.

[0085] The graphics subsystem 915 can perform image processing, such as still images or video, for display. The graphics subsystem 915 may be, for example, a graphics processing unit (GPU) or a visual processing unit (VPU). An analog or digital interface may be used to communicatively couple the graphics subsystem 915 and the display 920. For example, this interface may be any of a high-definition multimedia interface, a display port, wireless HDMI, and / or wireless HD compliant technologies. The graphics subsystem 915 may be integrated into a processor 910 or a chipset 905. In some implementations, the graphics subsystem 915 may be a standalone device communicatively coupled to the chipset 905.

[0086] The graphics and / or video processing techniques described herein can be implemented in various hardware architectures. For example, graphics and / or video functions can be integrated within a chipset. Alternatively, discrete graphics and / or video processors can be used. As another implementation, graphics and / or video functions can be provided by general-purpose processors, including multi-core processors. In another embodiment, these functions can be implemented in consumer electronic devices.

[0087] Radio device 918 may include one or more radio devices capable of transmitting and receiving signals using a variety of suitable wireless communication technologies. Such technologies may involve communication across one or more wireless networks. Example wireless networks include (but are not limited to) wireless local area networks (WLANs), wireless personal area networks (WPANs), wireless metropolitan area networks (WMANs), cellular networks, and satellite networks. When communicating across such networks, radio device 918 may operate according to any version of one or more applicable standards.

[0088] In various implementations, display 920 may include any television-type monitor or display. Display 920 may include, for example, a computer screen, a touchscreen display, a video monitor, a television-like device, and / or a television set. Display 920 may be digital and / or analog. In various implementations, display 920 may be a holographic display. Additionally, display 920 may be a transparent surface capable of receiving visual projections. Such projections may convey various forms of information, images, and / or objects. For example, such projections may be visual overlays for mobile augmented reality (MAR) applications. Under the control of one or more software applications 916, platform 902 may display user interface 920 on display 922.

[0089] In various implementations, one or more content service devices 930 may be hosted by any national, international, and / or independent service and thus accessible to platform 902, for example, via the Internet. One or more content service devices 930 may be coupled to platform 902 and / or display 920. Platform 902 and / or one or more content service devices 930 may be coupled to network 960 to transmit (e.g., send and / or receive) media information to and from network 960. One or more content delivery devices 940 may also be coupled to platform 902 and / or display 920.

[0090] In various implementations, one or more content service devices 930 may include cable TV boxes, personal computers, networks, telephones, internet-enabled devices or appliances capable of delivering digital information and / or content, and any other similar devices capable of transmitting content unidirectionally or bidirectionally between the content provider and platform 902 and / or display 920 via network 960 or directly. It will be understood that content may be transmitted unidirectionally and / or bidirectionally to and from any component in system 900 and the content provider via network 960. Examples of content may include any media information, such as video, music, medical and gaming information, etc.

[0091] One or more content service devices 930 may receive content, such as cable television programs, including media information, digital information, and / or other content. Examples of content providers may include any cable or satellite television, or radio or internet content provider. The examples provided are not intended to limit in any way the implementations according to this disclosure.

[0092] In various implementations, platform 902 may receive control signals from navigation controller 950 having one or more navigation features. Navigation features may be used, for example, to interact with user interface 922. In various embodiments, navigation may be a pointing device, which may be a computer hardware component (specifically, a human-machine interface device) that allows users to input spatial (e.g., continuous and multidimensional) data into the computer. Many systems, such as graphical user interfaces (GUIs) and televisions and monitors, allow users to control a computer or television and provide data to it using physical gestures.

[0093] Movement of navigation features can be replicated on the display (e.g., display 920) by movement of pointers, cursors, focus rings, or other visual indicators displayed on the display. For example, under the control of software application 916, navigation features located on the navigation path can be mapped to virtual navigation features displayed on user interface 922. In various embodiments, these may not be separate components but may be integrated into platform 902 and / or display 920. However, this disclosure is not limited to the elements or scenarios shown or described herein.

[0094] In various implementations, for example, when enabled, the driver (not shown) may include technology that allows a user to instantly turn platform 902 on and off like a television by touching a button after initial startup. Program logic may allow platform 902 to stream content to a media adapter or(s) other content service device 930 or(s) content delivery device 940 even when the platform is “off”. Furthermore, chipset 905 may include hardware and / or software support for, for example, 5.1 surround stereo audio and / or high-definition 7.1 surround stereo audio. The driver may include a graphics driver for an integrated graphics platform. In various embodiments, the graphics driver may include a high-speed peripheral component interconnect (PCI) Express graphics card.

[0095] In various implementations, any one or more components shown in system 900 may be integrated. For example, platform 902 and one or more content service devices 930 may be integrated, or platform 902 and one or more content delivery devices 940 may be integrated, or platform 902, one or more content service devices 930, and one or more content delivery devices 940 may be integrated. In various embodiments, platform 902 and display 920 may be integrated units. For example, display 920 and one or more content service devices 930 may be integrated, or display 920 and one or more content delivery devices 940 may be integrated. These examples are not intended to limit this disclosure.

[0096] In various embodiments, system 900 may be implemented as a wireless system, a wired system, or a combination of both. When implemented as a wireless system, system 900 may include components and interfaces suitable for communication via a wireless shared medium, such as one or more antennas, transmitters, receivers, transceivers, amplifiers, filters, control logic, etc. Examples of a wireless shared medium may include portions of the wireless spectrum, such as the RF spectrum, etc. When implemented as a wired system, system 900 may include components and interfaces suitable for communication via a wired communication medium, such as input / output (I / O) adapters, physical connectors connecting I / O adapters to a corresponding wired communication medium, network interface cards (NICs), disk controllers, video controllers, audio controllers, etc. Examples of a wired communication medium may include wires, cables, metal leads, printed circuit boards (PCBs), backplanes, switching structures, semiconductor materials, twisted pairs, coaxial cables, optical fibers, etc.

[0097] Platform 902 can establish one or more logical or physical channels to transmit information. This information may include media information and control information. Media information can refer to any data representing content intended for a user. Examples of content may include data from a voice conversation, video conferencing, streaming video, email (“email”) messages, voicemail messages, alphanumeric symbols, graphics, images, video, text, and so on. Data from a voice conversation may be, for example, voice information, silence periods, background noise, comfort noise, tone, and so on. Control information can refer to any data representing commands, instructions, or control words intended for an automated system. For example, control information can be used to route media information through the system or to instruct nodes to process media information in a predetermined manner. However, the embodiments are not limited to this. Figure 9 The elements or situations shown or described in the text.

[0098] As described above, system 900 can be implemented in different physical styles or shape factors. Figure 10 An example small form factor device 1000 arranged according to at least some implementations of the present disclosure is illustrated. In some examples, system 900 may be implemented via device 1000. In other examples, system 100 or some portion thereof may be implemented via device 1000. In various embodiments, for example, device 1000 may be implemented as a mobile computing device with wireless capabilities. A mobile computing device may, for example, refer to any device having a processing system and a mobile power source or power supply (e.g., one or more batteries).

[0099] Examples of mobile computing devices may include: personal computers (PCs), laptop computers, ultra-portable laptop computers, tablet devices, touchpads, portable computers, handheld computers, PDAs, personal digital assistants (PDAs), cellular phones, combined cellular phones / PDAs, smart devices (e.g., smartphones, smart tablets, or smart mobile TVs), mobile internet devices (MIDs), messaging devices, data communication devices, cameras, and so on.

[0100] Examples of mobile computing devices may also include computers arranged for wearability, such as wrist computers, finger computers, ring computers, glasses computers, belt buckle computers, armband computers, shoe computers, clothing computers, and other wearable computers. In various embodiments, for example, the mobile computing device may be implemented as a smartphone capable of performing computer applications as well as voice and / or data communications. While some embodiments may be described using a mobile computing device implemented as a smartphone as examples, it is understood that other embodiments may also be implemented using other wireless mobile computing devices. The embodiments are not limited to this context.

[0101] like Figure 10 As shown, device 1000 may include a housing having a front side 1001 and a rear side 1002. Device 1000 includes a display 1004, an input / output (I / O) device 1006, and an integrated antenna 1008. Device 1000 may also include navigation features 1012. I / O device 1006 may include any suitable I / O device for inputting information into a mobile computing device. Examples of I / O device 1006 may include an alphanumeric keypad, numeric keypad, touchpad, input keys, buttons, switches, microphone, speaker, voice recognition device, and software, etc. Information may also be input into device 1000 via a microphone (not shown) or may be digitized by a voice recognition device. As shown, device 1000 may include a camera 1005 (e.g., including a lens, aperture, and imaging sensor) and a flash 1010 integrated into the rear side 1002 (or elsewhere) of device 1000. In other examples, camera 1005 and flash 1010 may be integrated into the front 1001 of device 1000, or both front and rear cameras may be provided. Camera 1005 and flash 1010 may be components of a camera module for generating image data processed into streaming video, which may be output to display 1004 and / or remotely communicated from device 1000 via antenna 1008, for example.

[0102] Various embodiments may be implemented using hardware elements, software elements, or a combination of both. Examples of hardware elements may include processors, microprocessors, circuits, circuit elements (e.g., transistors, resistors, capacitors, inductors, etc.), integrated circuits, application-specific integrated circuits (ASICs), programmable logic devices (PLDs), digital signal processors (DSPs), field-programmable gate arrays (FPGAs), logic gates, registers, semiconductor devices, chips, microchips, chipsets, etc. Examples of software may include software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, procedures, software interfaces, application program interfaces (APIs), instruction sets, computational code, computer code, code segments, computer code segments, words, values, symbols, or any combination thereof. Determining whether an embodiment utilizes hardware and / or software elements can vary depending on any number of factors, such as desired computing speed, power level, thermal tolerance, processing cycle budget, input data rate, output data rate, memory resources, data bus speed, and other design or performance constraints.

[0103] One or more aspects of at least one embodiment may be implemented by representative instructions stored on a machine-readable medium representing various logic within a processor, which, when read by a machine, cause the machine-manufacturing logic to perform the techniques described herein. This representation, referred to as an IP core, may be stored on a tangible machine-readable medium and provided to various customers or manufacturing facilities for loading into the manufacturing machine that actually produces the logic or processor.

[0104] In one or more first embodiments, a method for providing adaptive enhanced video processing includes: decoding a bitstream to generate a first block and a second block of a video frame, wherein the first block and the second block respectively include a first video coding mode and a second video coding mode; applying a deep learning network to the first block in response to the first video coding mode to generate a first output block; bypassing the application of the deep learning network to the second block and extracting a previous second output block for the second block in response to the second video coding mode; and generating an output video frame corresponding to the video frame, at least based on merging the first output block and the second output block into an output video frame.

[0105] In one or more second embodiments, with respect to the first embodiment, the deep learning network includes a deep learning super-resolution network, wherein the video frame is at a first resolution, and the first output block, the second output block, and the output video frame are at a second resolution greater than the first resolution.

[0106] In one or more third embodiments, with respect to the first or second embodiment, the second video coding mode is inter-frame coded and has no coded residual information, and the second block has a corresponding motion vector.

[0107] In one or more fourth embodiments, with respect to any of the first to third embodiments, extracting the second output block includes: extracting the second output block based on a second super-resolution video frame that is temporally adjacent to the video frame using motion vectors.

[0108] In one or more fifth embodiments, with respect to any of the first to fourth embodiments, extracting the second output block includes: accessing pixel samples of a second super-resolution video frame, which includes a position at the horizontal position of the second block adjusted based on motion vectors and scaled using a scaling factor, and has a size based on the width of the second block scaled using a scaling factor.

[0109] In one or more sixth embodiments, with respect to any of the first to fifth embodiments, the first block is intra-frame coded or inter-frame coded and has coded residual information.

[0110] In one or more seventh embodiments, with respect to any of the first to sixth embodiments, the method further includes: decoding the bitstream to generate a third block of a second video frame, wherein the video frame includes a first quantization parameter and the second video frame includes a second quantization parameter less than the first quantization parameter, and applying interpolation to the third block in response to the second quantization parameter being greater than a threshold to generate a third output block of a second resolution.

[0111] In one or more eighth embodiments, with respect to any of the first to seventh embodiments, applying the deep learning network to the first block, bypassing the application of the deep learning network to the second block, and extracting the previous second output block at the second resolution are performed in response to a first quantization parameter being less than a threshold.

[0112] In one or more ninth embodiments, with respect to any of the first to eighth embodiments, the interpolation includes one of bilinear interpolation, bicubic interpolation, or Lanzos interpolation, and wherein the deep learning network includes at least one convolutional layer.

[0113] In one or more tenth embodiments, and with respect to any of the first to ninth embodiments, the method further includes decoding the bitstream to generate metadata including a first video coding mode and a second video coding mode, as well as a first quantization parameter and a second quantization parameter.

[0114] In one or more eleventh embodiments, with respect to any of the first to tenth embodiments, the method further includes decoding the bitstream to generate a third block and a fourth block of a video frame, wherein the third block and the fourth block respectively include a first video coding mode and a second video coding mode, and wherein the first block and the second block include regions of interest blocks, and the third block and the fourth block include regions of non-interest blocks, applying a deep learning network to the third block in response to the first coding mode to generate a third output block of a second resolution, and bypassing the application of a deep learning super-resolution network to the fourth block in response to the second coding mode, and extracting a previous fourth output block of the second resolution for the fourth block, wherein generating an output video frame includes merging the third output block and the fourth output block into the output video frame.

[0115] In one or more twelfth embodiments, with respect to any of the first to eleventh embodiments, the deep learning network includes a deep learning-based anti-aliasing network or a deep learning-based denoising network.

[0116] In one or more thirteenth embodiments, an apparatus or system includes a memory and a processor for performing the method according to any of the above embodiments.

[0117] In one or more of the fourteenth embodiments, at least one machine-readable medium includes a plurality of instructions that, in response to being executed on a computing device, cause the computing device to perform the method according to any of the above embodiments.

[0118] In one or more fifteenth embodiments, an apparatus includes means for performing a method according to any of the above embodiments.

[0119] While certain features described herein have been referenced to various implementations, this description is not intended to be construed as limiting. Therefore, various modifications and other implementations of the implementations described herein that will be apparent to those skilled in the art to which this disclosure pertains are considered to be within the spirit and scope of this disclosure.

[0120] It will be appreciated that the embodiments are not limited to those described as such, but can be implemented through modifications and variations without departing from the scope of the appended claims. For example, the above embodiments may include specific combinations of features. However, the above embodiments are not limited thereto, and in various implementations, the above embodiments may include only a subset of such features, different orders of such features, different combinations of such features, and / or additional features different from those expressly listed. Therefore, the scope of the embodiments should be determined by reference to the appended claims and the full scope of their equivalents.

Claims

1. One or more non-transitory computer-readable media storing instructions executable to perform operations, the operations including: Obtain the video and its metadata; A first output is generated by a deep learning network based on at least a portion of the video and the video's metadata; Based on the video, a second output is generated by an upsampler, wherein the second output is generated by bypassing the deep learning network; and The first output and the second output are combined to generate a new video, wherein the new video has a higher resolution than the original video.

2. The one or more non-transitory computer-readable media according to claim 1, wherein, The video and its metadata are generated by a video decoder, wherein the metadata includes decoding information.

3. One or more non-transitory computer-readable media according to claim 2, wherein, The metadata of the video includes quantization parameters or parameters indicating intra-frame or inter-frame modes.

4. One or more non-transitory computer-readable media according to claim 1, wherein, The deep learning network is a super-resolution network.

5. One or more non-transitory computer-readable media according to claim 1, wherein, The deep learning network includes one or more convolutional layers.

6. One or more non-transitory computer-readable media according to claim 5, wherein, The deep learning network also includes a rectified linear unit layer.

7. One or more non-transitory computer-readable media according to claim 1, wherein, The video was generated by reducing the resolution of another video.

8. A calculation method, comprising: Obtain the video and its metadata; A first output is generated by a deep learning network based on at least a portion of the video and the video's metadata; Based on the video, a second output is generated by an upsampler, wherein the second output is generated by bypassing the deep learning network; and The first output and the second output are combined to generate a new video, wherein the new video has a higher resolution than the original video.

9. The calculation method according to claim 8, wherein, The video and its metadata are generated by a video decoder, wherein the metadata includes decoding information.

10. The calculation method according to claim 9, wherein, The metadata of the video includes quantization parameters or parameters indicating intra-frame or inter-frame modes.

11. The calculation method according to claim 8, wherein, The deep learning network is a super-resolution network.

12. The calculation method according to claim 8, wherein, The deep learning network includes one or more convolutional layers.

13. The calculation method according to claim 12, wherein, The deep learning network also includes a rectified linear unit layer.

14. The calculation method according to claim 8, wherein, The video was generated by reducing the resolution of another video.

15. A computing system, comprising: The memory stores instructions; as well as One or more processors, The instruction is responsive to execution by the one or more processors, causing the one or more processors to perform the computation method according to any one of claims 8 to 14.

16. A computing device comprising means for performing the computing method according to any one of claims 8 to 14.

17. A computer program product comprising instructions that, in response to execution by one or more processors, cause the one or more processors to perform the computation method according to any one of claims 8 to 14.

Images