Enhanced resolution generation at decoder

Abstract

An apparatus includes a decoder including a spatial prediction engine, a temporal prediction engine, a reconstruction engine, and a decoded picture buffer. The apparatus also includes a controller configured to cause the decoder to reconstruct a base resolution version of the block of the frame in a base resolution mode using at least one of the spatial prediction engine or the temporal prediction engine and the reconstruction engine. The controller is further configured to cause the decoder to generate an enhanced resolution version of the block in an enhanced resolution mode using at least one of the spatial prediction engine or the temporal prediction engine and the reconstruction engine.

Description

[0001] I. CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Nonprovisional Patent Application No. 18 / 177,942, filed March 3, 2023, the contents of which are expressly incorporated by reference in their entirety. TECHNICAL FIELD

[0003] The present disclosure generally relates to decoding data and generating an enhanced resolution version of the decoded data.

[0004] III. DESCRIPTION OF RELATED ART

[0005] Advances in technology have resulted in an increasing popularity of high resolution display devices, such as for televisions and portable electronic devices. However, high resolution video content can not be available at the high resolution display device, such as when the video resolution is limited by the available transmission bandwidth. In such cases, upsampling can be performed at the display device for playback at the display at a higher resolution.

[0006] The quality of the upsampled video can be limited by the amount of computational resources available for the upsampling. For devices with a relatively low amount of available computational resources, low complexity upsampling techniques, such as simple interpolation and filtering with increased sharpening, can be performed; however, the resulting visual quality can not be satisfactory. Higher complexity techniques, such as super resolution, can provide higher visual quality by exploiting non-local similarities of patches or learning a mapping that relates pixels from a low resolution video to pixels from a high resolution video from an external dataset. However, super resolution algorithms are computationally more expensive and slower compared to simple interpolation / filtering, and the amount of power and computational resources required to perform real-time super resolution computations can prohibit use on portable consumer devices. SUMMARY

[0007] According to one implementation of the present disclosure, a device includes a decoder comprising a spatial prediction engine, a temporal prediction engine, a reconstruction engine, and a decoded picture buffer. The device also includes a controller configured to cause the decoder to reconstruct, in a base resolution mode, a base resolution version of a block of a frame using at least one of the spatial prediction engine or the temporal prediction engine and the reconstruction engine. The controller is further configured to cause the decoder to generate, in an enhanced resolution mode, an enhanced resolution version of the block using the at least one of the spatial prediction engine or the temporal prediction engine and the reconstruction engine.

[0008] According to another implementation of the disclosure, a method includes reconstructing, at a decoder, a base resolution version of a block of a frame using at least one of a spatial prediction engine or a temporal prediction engine of the decoder and a reconstruction engine. The method also includes changing an operational mode of the decoder from a base resolution mode to an enhanced resolution mode after reconstructing the base resolution version of the block, and generating, at the decoder, an enhanced resolution version of the block using the at least one of the spatial prediction engine or the temporal prediction engine and the reconstruction engine.

[0009] According to another implementation of the disclosure, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause the one or more processors to reconstruct, at a decoder, a base resolution version of a block of a frame using at least one of a spatial prediction engine or a temporal prediction engine of the decoder and a reconstruction engine. The instructions, when executed by one or more processors, also cause the one or more processors to change an operational mode of the decoder from a base resolution mode to an enhanced resolution mode after reconstructing the base resolution version of the block, and generate, at the decoder, an enhanced resolution version of the block using the at least one of the spatial prediction engine or the temporal prediction engine and the reconstruction engine.

[0010] According to another implementation of the disclosure, an apparatus includes means for decoding including reconstructing, in a base resolution mode, a base resolution version of a block of a frame using at least one of a spatial prediction engine or a temporal prediction engine and a reconstruction engine, and generating, in an enhanced resolution mode, an enhanced resolution version of the block using the at least one of the spatial prediction engine or the temporal prediction engine and the reconstruction engine. The apparatus also includes means for controlling an operational mode of the means for decoding.

[0011] Other aspects, advantages, and features of the disclosure will become apparent after review of the entire application, including the following sections: BRIEF DESCRIPTION OF DRAWINGS, DETAILED DESCRIPTION, and CLAIMS. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 is a block diagram of particular illustrative aspects of a system operable to perform decoding and enhanced resolution generation at a decoder in accordance with some examples of the present disclosure.

[0013] Figure 2 is a block diagram of particular illustrative aspects of a system operable to perform decoding and enhanced resolution generation at a decoder in accordance with some examples of the present disclosure. Figure 1 is a block diagram of particular illustrative aspects of a system operable to perform decoding and enhanced resolution generation at a decoder in accordance with some examples of the present disclosure.

[0014] Figure 3 is a block diagram of particular illustrative aspects of a system operable to perform decoding and enhanced resolution generation at a decoder in accordance with some examples of the present disclosure. Figure 1 is a block diagram of particular illustrative aspects of a system operable to perform decoding and enhanced resolution generation at a decoder in accordance with some examples of the present disclosure.

[0015] Figure 4 is a diagram of an example of a decoder associated with a Figure 1 basic resolution decoding and enhanced resolution generation at a decoder of a

[0016] Figure 5 is a diagram of an example of a decoder associated with a Figure 1 basic resolution decoding and enhanced resolution generation at a decoder of a

[0017] Figure 6 is a diagram of an example of a decoder associated with a Figure 1 basic resolution decoding and enhanced resolution generation at a decoder of a

[0018] Figure 7 An example of an integrated circuit operable to perform enhanced resolution generation at a decoder according to some examples of the disclosure is illustrated.

[0019] Figure 8 is a diagram of a mobile device operable to perform enhanced resolution generation at a decoder according to some examples of the disclosure.

[0020] Figure 9 is a diagram of a headset operable to perform enhanced resolution generation at a decoder according to some examples of the disclosure.

[0021] Figure 10 is a diagram of a wearable electronic device operable to perform enhanced resolution generation at a decoder according to some examples of the disclosure.

[0022] Figure 11 is a diagram of a sound-controlled speaker system operable to perform enhanced resolution generation at a decoder according to some examples of the disclosure.

[0023] Figure 12 is a diagram of a camera operable to perform enhanced resolution generation at a decoder according to some examples of the disclosure.

[0024] Figure 13 is a diagram of a head-mounted device (such as a virtual reality, mixed reality, or augmented reality head-mounted device) operable to perform enhanced resolution generation at a decoder according to some examples of the disclosure.

[0025] Figure 14 is a diagram of a first example of a vehicle operable to perform enhanced resolution generation at a decoder according to some examples of the disclosure.

[0026] Figure 15 is a diagram of a second example of a vehicle operable to perform enhanced resolution generation at a decoder according to some examples of the disclosure.

[0027] Figure 16 is a system that performs a method of enhanced resolution generation in accordance with some examples of the present disclosure. Figure 1 is a diagram of a particular implementation of a method of enhanced resolution generation performed by a system in accordance with some examples of the present disclosure.

[0028] Figure 17 is a block diagram of a particular illustrative example of a device that is operable to perform enhanced resolution generation at a decoder in accordance with some examples of the present disclosure. DETAILED DESCRIPTION

[0029] While high resolution displays are increasingly common for both televisions and portable devices, typically available video content has a lower resolution than the capacity of the display. While high quality up-sampling techniques such as super-resolution exist, such techniques can not be available due to large processing speed and power requirements associated with real-time video processing. Thus, a user can only be able to view a video at a lower resolution or quality than supported by the user's display device, which can negatively impact the user's viewing experience.

[0030] Systems and methods of enhanced resolution generation at a decoder are disclosed. According to some aspects, adaptive transport techniques are used to significantly reduce the amount of processing speed and power required to perform high quality, real-time upscaling (e.g., super-resolution) to generate enhanced resolution video. A decoder is configured to decode a bitstream representing video frames and reconstruct video frames having a base resolution associated with video encoding that generated the bitstream. The decoder is further configured to apply adaptive transport that reuses motion compensation information received in the bitstream to generate up-scaled frames, such as super-resolution frames, having an enhanced resolution.

[0031] An upscaler device, such as a neural processing unit that includes a machine learning model, can process a first frame ("key frame") in the reconstructed video frames having the base resolution to generate an enhanced resolution version of the key frame. The decoder uses the enhanced resolution version of the key frame as a source of enhanced resolution pixel blocks to generate an enhanced resolution version of a next frame, taking advantage of temporal correlation between adjacent frames such that only a subset of the frames are offloaded to the upscaler device as key frames. Transmission of enhanced resolution pixels from a preceding frame during generation of a subsequent frame has a negligible computational cost because the transmission uses information already embedded in the compressed video (e.g., motion vectors and residuals).

[0032] According to an aspect, the component of the decoder for reconstructing the base resolution frames by applying motion compensation for blocks of the base resolution key frames is also for generating an enhanced resolution version of the next frame by applying motion compensation for blocks of the enhanced resolution key frames. The technology thus includes video decoding using a common core and performing enhanced resolution with re-use of components (e.g., hardware blocks) of the decoder. In illustrative examples, re-use of hardware blocks can reduce video engine area by 15-20% and also result in approximately 16% average bandwidth savings compared to implementations where video decoding and enhanced resolution using adaptive transmission are performed at different hardware blocks.

[0033] According to another aspect, a determination is made as to whether to apply motion compensation or, instead, upscaling (e.g., bi-cubic interpolation) for each block within the enhanced resolution frames. For example, because using motion compensation to generate a sequence of enhanced resolution frames can result in accumulated errors with each enhanced resolution frame generated in the sequence, a comparison is made for each block that determines whether an energy metric of the block (e.g., an energy of residuals of the block) is less than a dynamic threshold. In some implementations, the dynamic threshold is computed for each frame based on a decay factor to model error propagation and also based on a transmission distance from the most recent key frame (e.g., a count of how many frames have been generated since the last enhanced resolution key frame). The determination based on the decay factor and the transmission distance reduces an amount of cache memory usage compared to implementations where a decision to skip motion transmission is made by storing a cumulative residual error for each coding unit of each reference frame used for motion compensation and comparing the cumulative residual error to a threshold.

[0034] By using components of the decoder as a common core for decoding and for enhanced resolution generation, the disclosed systems and methods provide technical advantages of reduced power consumption, reduced video engine area, reduced bandwidth usage, and reduced cache usage in addition to determining whether to apply motion compensation for each block based on a decay factor and a transmission distance.

[0035] Particular aspects of the present disclosure are described below with reference to the accompanying drawings. In this description, common features are designated by common reference numbers throughout the description. As used herein, various terminology is used for the purpose of describing particular implementations only and is not intended to be limiting of implementations. For example, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. Further, some features described herein are singular in some implementations and plural in other implementations. For the sake of example, Figure 17 A block diagram of an example system 100 is depicted in FIG. 1. The system 100 includes one or more processors 102 and memory 104 coupled to the one or more processors 102. Figure 17“one or more” feature and subsequently referred to in the singular, unless the context indicates otherwise.

[0036] As used herein, the term “comprise” can be used interchangeably with “include,” “comprise” or “have.” Additionally, the term “wherein” can be used interchangeably with “where.” As used herein, “exemplary” indicates an example, implementation, and / or aspect, and should not be construed as indicating a preference or a preferred implementation. As used herein, ordinal terms (e.g., “first,” “second,” “third,” etc.) are used primarily for purposes of narration and / or description. Such terms are not intended to limit the number, order, and / or identity of the elements being described.

[0037] In some drawings, multiple instances of a particular type of feature are used. Although these features are physically and / or logically distinct, the same reference label is used for each feature, and the different instances are distinguished by the addition of a letter. When reference is made herein to a feature as a group or of a type (e.g., when a particular one of the features is not referenced), the reference label is used without the distinguishing letter. However, when a particular one of multiple features of the same type is referenced herein, the reference label is used with the distinguishing letter. For example, with reference to Figure 2 , multiple amplifiers are illustrated and associated with reference labels 220A, 220B, and 220C. When a particular one of these amplifiers is referenced (such as amplifier 220A), the distinguishing letter “A” is used. However, when any of these amplifiers or the amplifiers as a group are referenced, the reference label 220 is used without the distinguishing letter.

[0038] As used herein, “coupled” can include “communicatively coupled,” “electrically coupled,” or “physically coupled,” and can also (or alternatively) include any combination thereof. Two devices (or components) can be coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) directly or indirectly via one or more other devices, components, wires, buses, networks (e.g., wired, wireless, or a combination thereof), etc. As an illustrative, non-limiting example, two devices (or components) that are electrically coupled can be included in the same device, can be included in different devices, and can be connected via electronics, one or more connectors, or inductive coupling. In some implementations, two devices (or components) that are communicatively coupled (such as in electrical communication) can transmit and receive signals (e.g., digital signals or analog signals) directly or indirectly via one or more wires, buses, networks, etc. As used herein, “directly coupled” can include two devices that are coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) without an intervening component.

[0039] In this disclosure, terms such as “determine,” “calculate,” “estimate,” “shift,” “adjust,” and the like can be used to describe how one or more operations are performed. It should be noted that such terms are not to be construed as limiting and that other technologies can be utilized to perform similar operations. Additionally, as referred to herein, “generate,” “calculate,” “estimate,” “use,” “select,” “access,” and “determine” are used interchangeably. For example, “generating,” “calculating,” “estimating,” or “determining” a parameter (or signal) can refer to either actively generating, estimating, calculating, or determining the parameter (or signal), or can refer to using, selecting, or accessing a parameter (or signal) that has been generated (such as by another component or device).

[0040] Figure 1 is a block diagram of a particular illustrative aspect of a system 100 that is operable to perform decoding and enhanced resolution generation at a decoder 102. The system 100 includes a device 101 that includes the decoder 102, an upscaling engine 104, and a controller 160. The device 101 is configured to receive a bitstream 106 that includes a representation of a sequence of frames 190, and to process the bitstream 106 to generate a decoder output 108. According to an aspect, the device 101 processes the bitstream 106 to reconstruct a base resolution version of the frames 190 (e.g., 480p or 720p) from information received in the bitstream 106, and also to generate an enhanced resolution version of the frames 190. The base resolution version of the frames 190, the enhanced resolution version of the frames 190, or both can be provided in the decoder output 108, which can be communicated to a display device for playback.

[0041] The decoder 102 includes a syntax engine 110, a transform engine 112, a prediction engine 114, a filter engine 116, and a decoded picture buffer 118. The syntax engine 110 is configured to perform decoding on the bitstream 106 to extract quantized coefficients, motion vectors, and other syntax elements, according to a particular implementation. The motion vectors and other syntax elements can be forwarded to the prediction engine 114 as data 111.

[0042] The transform engine 112 is configured to perform inverse transforms on portions of the bitstream 106. In an illustrative example, the transform engine 112 performs inverse transforms (e.g., inverse discrete cosine transform (DCT), inverse integer transform, inverse Karhunen-Loeve transform (KLT), etc.) on coefficient blocks from the bitstream 106 to produce residual blocks in the pixel domain, which are provided as data 113 for use by the prediction engine 114.

[0043] The prediction engine 114 is configured to generate predicted blocks for frames based on information from the bitstream 106. For example, the prediction engine 114 can perform intra-prediction processing that generates prediction data (e.g., predicted blocks) for blocks based on signaled intra-prediction modes and data from previously decoded blocks of the current frame or picture. As another example, the prediction engine 114 can perform inter-prediction processing that produces predicted blocks for video blocks based on motion vectors and other syntax elements of the syntax engine 110. The predicted blocks can be produced from one or more reference frames stored in the decoded picture buffer 118, as explained further below.

[0044] The filter engine 116 is configured to perform filtering to reduce blocking artifacts associated with coded blocks, and resulting pixels of reconstructed frames can be stored in the decoded picture buffer 118 and made available to the prediction engine 114 for use when generating predicted blocks for subsequent frames.

[0045] The controller 160 is configured to cause the decoder 102 to operate in a base resolution mode 162 or in an enhanced resolution mode 164. To illustrate, in the base resolution mode 162, the prediction engine 114 is configured to perform base resolution prediction 130 to generate a reconstructed pixel block having a base resolution, which is illustrated as a base resolution pixel block 132. For example, for an inter-predicted block, the prediction engine 114 can copy a base resolution block 150 from a previously reconstructed base resolution frame 138 (e.g., one or more base resolution reference frames) stored in the decoded picture buffer 118. As another example, for an intra-predicted block, the prediction engine 114 can use a previously decoded base resolution block from the current frame. Also in the base resolution mode 162, the filter engine 116 is configured to perform base resolution deblocking 134 of the base resolution pixel block 132 to generate a base resolution filtered block 136 of a base resolution frame. The resulting reconstructed frame can be stored in the decoded picture buffer 118 as one of the base resolution frames 138.

[0046] According to an aspect, the operation of the decoder 102 in the base resolution mode 162 includes the prediction engine 114 performing enhanced resolution prediction 140 to generate a reconstructed pixel block having an enhanced resolution, which is illustrated as an enhanced resolution pixel block 142. For example, for an inter-predicted block, the prediction engine 114 can copy an enhanced resolution block 152 from a previously generated enhanced resolution frame 148 (e.g., one or more enhanced resolution reference frames) stored in the decoded picture buffer 118. As another example, for an intra-predicted block, the prediction engine 114 can use a previously decoded enhanced resolution block from the current frame. Also in the enhanced resolution mode 164, the filter engine 116 performs enhanced resolution deblocking 144 of the enhanced resolution pixel block 142 to generate an enhanced resolution filtered block 146 of an enhanced resolution frame. The resulting enhanced resolution frame can be stored in the decoded picture buffer 118 as one of the enhanced resolution frames 148.

[0047] According to an aspect, the controller 160 is further configured to offload the enhanced resolution processing of selected frames, referred to as key frames, to the upscaling engine 104. For example, since performing the enhanced resolution prediction 140 at the prediction engine 114 relies on the availability of initial enhanced resolution frames from which to predict the enhanced resolution blocks, such initial enhanced resolution frames are generated at the upscaling engine 104. According to some implementations, the controller 160 is configured to offload the enhanced resolution processing to the upscaling engine 104 based on the bitstream 106, such as when the current frame corresponds to an I-frame (e.g., coded without inter-frame prediction of blocks from previous frames), or based on determining that the prediction error in an inter-frame predicted block exceeds a threshold (as an illustrative, non-limiting example).

[0048] According to an aspect, the upscaling engine 104 is configured to receive the base resolution key frames 120 from the decoder 102 and perform upscaling processing of the base resolution key frames 120 to generate enhanced resolution key frames 122, which can be stored in the decoded picture buffer 118 as enhanced resolution frames 148 to be available as sources of enhanced resolution blocks for inter-frame prediction of blocks of subsequent frames. For example, the upscaling engine 104 can perform computationally intensive processing, such as super-resolution upscaling. In a particular example, the upscaling engine 104 is implemented via execution of a machine learning model using one or more high-performance hardware components, such as a neural processing unit (NPU). To illustrate, the upscaling engine 104 can implement a deep neural network to perform super-resolution (e.g., a super-resolution convolutional neural network (SRCNN)), which can apply several convolutional layers and non-linear functions to map the base resolution key frames 120 to the enhanced resolution key frames 122.

[0049] During operation, the decoder 102 receives representations of a sequence of frames 190 in the bitstream 106, and for each particular frame in the sequence and during a single frame decoding time, reconstructs both a base resolution version of the particular frame and an enhanced resolution version of the particular frame, such as described with reference to Figure 5 and Figure 6 Further described. The decoder 102 stores the base resolution version of the particular frame and the enhanced resolution version of the particular frame in the decoded picture buffer 118 to be available for use as source blocks for motion transfer during generation of base resolution versions of subsequent frames in the base resolution mode 162 and enhanced resolution versions of subsequent frames in the enhanced resolution mode 164.

[0050] In an example, the controller 160 causes the decoder 102 to operate the prediction engine 114 according to the base resolution mode 162 to generate a base resolution version of one of the frames in the sequence, which the decoder 102 stores as one of the base resolution frames 138 in the decoded picture buffer 118. After generating the base resolution version, the controller 160 causes the decoder 102 to operate the prediction engine 114 according to the enhanced resolution mode 164 to generate an enhanced resolution version of the frame, which is then stored as one of the enhanced resolution frames 148 in the decoded picture buffer 118.

[0051] After generating the base resolution version and the enhanced resolution version of the frame, the controller 160 causes the decoder 102 to operate the prediction engine 114 according to the base resolution mode 162 to generate a base resolution version of a next sequence frame in the sequence, which the decoder 102 stores as one of the base resolution frames 138 in the decoded picture buffer 118. After generating the base resolution version, the controller 160 causes the decoder 102 to operate the prediction engine 114 according to the enhanced resolution mode 164 to operate the prediction engine 114 according to the enhanced resolution mode 164 to generate an enhanced resolution version of the second particular frame, which is also stored as one of the enhanced resolution frames 148 in the decoded picture buffer 118.

[0052] When one of the frames 190 is encoded without using inter-prediction (without using a block of a previously decoded frame to predict a block of the current frame), the controller 160 causes the decoder 102 to not generate an enhanced resolution version of the frame. Instead, the controller 160 causes the base resolution version of the frame to be provided to the upscaling engine 104 as a base resolution key frame 120, and processed by the upscaling engine 104 to generate an enhanced resolution key frame 122, which is stored as an enhanced resolution frame 148 in the decoded picture buffer 118. After generating the enhanced resolution version of the key frame, the enhanced resolution versions of one or more subsequent frames are generated by the decoder 102 in the enhanced resolution mode 164 based on blocks of the enhanced resolution version of the key frame 122.

[0053] In some implementations, the device 101 corresponds to or is included in one of various types of devices. In illustrative examples, the device 101 is integrated in an integrated circuit, as described with reference to FIGS. 1A and IB. In other examples, the device 101 is integrated in at least one of a mobile phone or tablet computer device, as described with reference to FIGS. 1C and ID, a television, as described with reference to FIG. 1E, a set-top box, as described with reference to FIG. 1F, a computer, as described with reference to FIG. 1G, or a gaming console, as described with reference to FIG. 1H. Figure 7 In some implementations, the device 101 corresponds to or is included in one of various types of devices. In illustrative examples, the device 101 is integrated in an integrated circuit, as described with reference to FIGS. 1A and IB. In other examples, the device 101 is integrated in at least one of a mobile phone or tablet computer device, as described with reference to FIGS. 1C and ID, a television, as described with reference to FIG. 1E, a set-top box, as described with reference to FIG. 1F, a computer, as described with reference to FIG. 1G, or a gaming console, as described with reference to FIG. 1H. Figure 8 In some implementations, the device 101 corresponds to or is included in one of various types of devices. In illustrative examples, the device 101 is integrated in an integrated circuit, as described with reference to FIGS. 1A and IB. In other examples, the device 101 is integrated in at least one of a mobile phone or tablet computer device, as described with reference to FIGS. 1C and ID, a television, as described with reference to FIG. 1E, a set-top box, as described with reference to FIG. 1F, a computer, as described with reference to FIG. 1G, or a gaming console, as described with reference to FIG. 1H. Figure 9The described headphone device, as referenced Figure 10 The described wearable electronic device, as referenced Figure 11 The described acoustic control loudspeaker system, as referenced Figure 12 The described camera device, or as referenced Figure 13 The described virtual reality, mixed reality, or augmented reality headset device. In another illustrative example, the device 101 is integrated into a vehicle, such as further referenced Figure 14 and Figure 15 The described.

[0054] The device 101 thus enables the use of information from the bitstream 106, such as residual information and motion vectors, and using the same components of the decoder 102 used to generate the base resolution reconstruction of the frames to generate the enhanced resolution frames. Using information from the bitstream to generate the enhanced resolution frames provides a technical advantage of reduced power consumption and a reduction in the amount of processing resources needed to generate the enhanced resolution frames compared to using the upscaling engine 104 to upscale each frame. Using the same components of the decoder 102 to decode the base resolution frames and generate the enhanced resolution frames provides a technical benefit of reducing the number, size, and data transfer bandwidth of components compared to another implementation in which the bitstream data and the base resolution frames 138 are transferred from the decoder 102 to another set of components for generating the enhanced resolution frames.

[0055] Although the controller 160 is illustrated as being distinct from and coupled to the decoder 102, in other implementations the controller 160 can be included in the decoder 102, such as via one or more state machines or other circuitry integrated in the decoder 102 to schedule the switching between operation in the base resolution mode 162 and operation in the enhanced resolution mode 164, to detect (or select) particular frames 190 as key frames for offloading to the upscaling engine 104, etc.

[0056] Although the decoder 102 is illustrated as including the syntax engine 110, the transform engine 112, the prediction engine 114, the filter engine 116, and the decoded picture buffer 118, in other implementations, the functionality described with respect to two or more of the syntax engine 110, the transform engine 112, the prediction engine 114, the filter engine 116, or the decoded picture buffer 118 can be combined into a single component. Although in some implementations, one or more (or all) of the syntax engine 110, the transform engine 112, the prediction engine 114, the filter engine 116, and the decoded picture buffer 118, the controller 160, and the upsampling engine 104 can be implemented in hardware (e.g., special-purpose circuitry), in other implementations, one or more (or all) of the syntax engine 110, the transform engine 112, the prediction engine 114, the filter engine 116, the decoded picture buffer 118, the controller 160, and the upsampling engine 104 are implemented as one or more processors executing instructions.

[0057] Figure 2 is Figure 1 FIG. 2 is a diagram of an illustrative aspect of components of a decoder of

[0058] The temporal prediction engine 202 is responsive to an inter-prediction indicator (“inter mode”) 258 that indicates to use inter-prediction based on a block from a previous frame. The temporal prediction engine 202 is configured to transfer pixels 250 from the decoded picture buffer 118 based on a motion vector 252 received from the bitstream 106 (such as from the transform engine (TE) 112), and output resulting inter-predicted pixels 256.

[0059] According to an aspect, when operating in the base resolution mode 162, the upsampler 220A receives a motion vector 252 for a block of a current decoded frame via the bitstream 106, and passes the motion vector 252 to the temporal prediction engine 202 without upsampling. The temporal prediction engine 202 transfers base resolution (BR) pixels 250A from the decoded picture buffer (DPB) 118 based on the motion vector 252, and outputs inter-predicted pixels 256 as a base resolution prediction 254A for the block.

[0060] According to an aspect, when operating in the enhanced resolution mode 164, the amplifier 220A amplifies the motion vector 252 to generate an amplified motion vector 253, and the decoder 102 is configured to transmit enhanced resolution pixels based on the amplified motion vector 253. For example, the temporal prediction engine 202 transmits enhanced resolution (ER) pixels 250B from the decoded picture buffer 118 based on the amplified motion vector 253, and outputs inter-predicted pixels 256 as an enhanced resolution prediction 254B for the block.

[0061] The spatial prediction engine 204 is responsive to an intra-prediction indicator ("intra mode") 260 that indicates to use intra-prediction based on previously reconstructed blocks from the current frame. The spatial prediction engine 204 is configured to transmit reconstructed pixels 262 from the preceding blocks to generate resulting intra-predicted pixels 266. According to an aspect, in the base resolution mode 162, the reconstructed pixels 262 correspond to base resolution pixels of previously decoded blocks of the current frame, and the spatial prediction engine 204 generates a base resolution prediction 264A for the block. Alternatively, in the enhanced resolution mode 164, the reconstructed pixels 262 correspond to enhanced resolution pixels of the previously decoded blocks, and the spatial prediction engine 204 generates an enhanced resolution prediction 264B for the block.

[0062] The selector 206 is configured to select either the first input from the temporal prediction engine 202 or the second input from the spatial prediction engine 204 as a predicted pixel 270 output to the reconstruction engine 208. In an example, the selection is determined based on whether to use inter-prediction or intra-prediction for the current block, such as indicated by one or both of the inter-prediction indicator 258 or the intra-prediction indicator 260. When using inter-prediction, the predicted pixel 270 corresponds to the base resolution prediction 254A in the base resolution mode 162, and to the enhanced resolution prediction 254B in the enhanced resolution mode 164. When using intra-prediction, the predicted pixel 270 corresponds to the base resolution prediction 264A in the base resolution mode 162, and to the enhanced resolution prediction 264B in the enhanced resolution mode 164.

[0063] The reconstruction engine 208 is configured to receive the predicted pixels 270 and generate a reconstructed block 274 based on the residual pixels 240 (also referred to as “residuals”). According to an aspect, under the base resolution mode 162, the residual pixels 240 are received at the reconstruction engine 208 and used during generation of the reconstructed base resolution block 274A. In some implementations, the residual pixels 240 are received at the amplifier 220B configured to pass the residual pixels 240 to the reconstruction engine 208 without amplification, and the reconstruction engine 208 adds the residual pixels 240 to the predicted pixels 270 to generate the reconstructed base resolution block 274A. Alternatively, under the enhanced resolution mode 164, the amplifier 220B is configured to amplify the residual pixels 240 to generate amplified residuals 272 for the block, and the reconstruction engine 208 is configured to use the amplified residuals 272 (e.g., by adding the amplified residuals 272 to the predicted pixels 270) during generation of the enhanced resolution version of the block 290B.

[0064] In some implementations, the reconstructed block 274 (e.g., the reconstructed base resolution block 274A or the reconstructed enhanced resolution block 274B) is provided as an output of the prediction engine 114 to the filter engine 116.

[0065] Optionally, the prediction engine 114 includes adaptive transmission logic 230 configured to generate a bypass control signal based on a comparison of an energy metric 248 of the residuals of the block to a dynamic threshold 232, as exemplified by the flag 246. For example, generating enhanced resolution frames using motion transmission can result in one or more artifacts, such as ringing artifacts due to loss of high frequency components when the residual pixels are upsampled by the amplifier 220B (e.g., using bi-cubic upsampling), and errors due to such artifacts can accumulate for each successive frame. In particular implementations, the dynamic threshold 232 is based on a decay factor and a transmission distance 244 from a most recently generated enhanced resolution key frame. According to an aspect, the adaptive transmission logic 230 sets the flag 246 to a value indicating that motion transmission is to be bypassed (e.g., “0” to bypass motion transmission or “1” to perform motion transmission) based on whether the following expression is true:

[0066] PU residual energy > THR bypass *β i ,

[0067] where PU residual energy represents an energy (e.g., Laplacian) of the residual pixels 240 for a current prediction unit (e.g., a current block), THR bypassis a threshold for bypassing motion transmission, β is a decay factor having a value less than one, and i indicates a transmission distance 244 from a most recently generated key frame (e.g., a count based on how many frames have been generated since the last enhanced resolution key frame 122). In particular implementations, the PU residual energy corresponds to the energy metric 248, and THR bypass *β i corresponds to the dynamic threshold 232 and has a value that decreases with each frame that is continuously generated using motion transmission.

[0068] In some implementations, the adaptive transmission logic 230 is further configured to set the value of the flag 246 based on the block type 242. For example, the adaptive transmission logic 230 can set the flag 246 to bypass motion compensation based on the block type 242 indicating that the current frame is a key frame.

[0069] According to some aspects, the prediction engine 114 is configured to bypass operation of one or more of the amplifier 220, the temporal prediction engine 202, the spatial prediction engine 204, or the reconstruction engine 208 based on the bypass control signal (e.g., the flag 246). For example, when the flag 246 has a value indicating that motion compensation is to be bypassed due to the energy metric 248 exceeding the dynamic threshold 232 (e.g., “0”), the output of the prediction engine 114 can be generated as an upscaled version of the base resolution version of the block.

[0070] To illustrate, the prediction engine 114 optionally includes a selector 210 and an amplifier 220C. When the flag 246 indicates that motion compensation is not to be performed for the current block, the amplifier 220C generates an upscaled version 282 of the base resolution block 280. The selector 210 is configured to select one of the output of the reconstruction engine 208 or the upscaled version 282 of the base resolution block 280 as the output of the prediction engine 114 based on the flag 246. To illustrate, the selector 210 selects the output of the reconstruction engine 208 when motion compensation is to be performed, and selects the upscaled version 282 when motion compensation is to be bypassed. In such cases where the upscaled version 282 is selected instead of the output of the reconstruction engine 208, power savings can be achieved by deactivating one or more of the amplifier 220A, the amplifier 220B, the temporal prediction engine 202, the spatial prediction engine 204, the selector 206, or the reconstruction engine 208 (or otherwise transitioning one or more of these to a reduced power usage state). Similarly, when the flag 246 indicates that motion compensation is to be bypassed due to the current frame being a key frame, one or more (or all) of the components of the prediction engine 114 can be deactivated or otherwise transitioned to a low power consumption state.

[0071] During operation, in the base resolution mode 162, the prediction engine 114 functions to reconstruct a base resolution version of a block 290A of a frame using at least one of the spatial prediction engine 204 or the temporal prediction engine 202 and the reconstruction engine 208. To illustrate, based on receiving an inter-prediction indicator 258 for the block in the base resolution mode 162, the prediction engine 114 generates a base resolution prediction 254A of the block using the motion vector 252 at the temporal prediction engine 202 to copy pixels from the decoded picture buffer 118, such as base resolution pixels 250A of a base resolution version of a reference frame. The prediction engine 114 also generates a base resolution reconstruction of the block at the reconstruction engine 208 based on the base resolution prediction 254A and a residual (e.g., residual pixels 240) of the block (e.g., a reconstructed base resolution block 274A).

[0072] In the enhanced resolution mode 164, the prediction engine 114 generates an enhanced resolution version of the block 290B using at least one of the spatial prediction engine 204 or the temporal prediction engine 202 and the reconstruction engine 208 used in generating the base resolution version of the block 290A. For example, in the enhanced resolution mode 164, the upscaler 220A upscales the motion vector 252 to generate an upscaled motion vector 253, the upscaler 220B upscales the residual pixels 240 to generate upscaled residuals 272, the temporal prediction engine 202 generates an enhanced resolution prediction 254B of the block using the upscaled motion vector 253 at the temporal prediction engine 202 to copy enhanced resolution pixels 250B of an enhanced resolution version of a reference frame from the decoded picture buffer 118. Additionally, the reconstruction engine 208 generates an enhanced resolution reconstruction of the block (e.g., a reconstructed enhanced resolution block 274B) based on the enhanced resolution prediction 254B and the upscaled residuals 272.

[0073] By including an upscaler 220A to generate upscaled motion vectors 253 and an upscaler 220B to generate upscaled residuals 272, the prediction engine 114 can perform the enhanced resolution generation using the same components as used to perform the base level decoding (e.g., temporal prediction engine 202, spatial prediction engine 204, selector 206, and reconstruction engine 208). This provides the technical advantage of reduced chip area and reduced transmission bandwidth compared to performing the enhanced resolution generation using different components than used to perform the base level decoding. In implementations in which the optional adaptive transport logic 230 is included in the prediction engine 114, using the flag 246 to control adaptive transport based on the energy metric 248 provides the technical benefit of improving the overall quality of the video output by replacing the output of the reconstruction engine 208 that can be expected to have perceptible artifacts due to residual error accumulation with an upscaled version 282 of the base resolution block 280. Implementations in which the flag 246 is optionally used to deactivate one or more components of the prediction engine 114 provide the technical benefit of reducing the power consumption of components that no longer contribute to the generation of the enhanced resolution version of the block 290B.

[0074] Figure 3 is a diagram of an illustrative aspect of operations 300 associated with performing adaptive transport at the decoder 102. The operations 300 include the decoder 102 receiving the bitstream 106 and performing base resolution mode decoding 310 (e.g., the operation of the decoder 102 in the base resolution mode 162 as described above) to generate a sequence 302 of base resolution video frames (e.g., YUV frames) via the operation in the base resolution mode 162.

[0075] Additionally, an enhanced resolution version of each frame in the sequence 302 of base resolution video frames is generated using adaptive transport. Specifically, the first base resolution video frame 304 in the sequence is designated as a key frame and is upsampled to generate a first enhanced resolution frame 314. For example, the first base resolution video frame 304 can be provided to the upscaling engine 104 as the base resolution key frame 120, and the upscaling engine 104 generates the first enhanced resolution frame 314 (e.g., using a deep learning super-resolution model) as the enhanced resolution key frame 122. Figure 1

[0076] ​For a number of video frames in the sequence 302 following the first base resolution video frame 304, the decoder 102 performs the enhanced resolution mode decoding 320 (e.g., adaptive transport corresponding to the operation of the decoder 102 in the enhanced resolution mode 164 as described above) to generate corresponding enhanced resolution frames. In the illustrated example, the third sequential video frame 306 in the sequence 302 is processed by the decoder 102 using the enhanced resolution mode decoding 320 to generate an enhanced resolution frame 316 corresponding to an upscaled version of the third sequential video frame 306.

[0077] The decoder 102 can continue to alternate between generating the next base resolution video frame in the sequence 302 using the base resolution mode decoding 310 and generating a corresponding enhanced resolution version of the base resolution video frame using the enhanced resolution mode decoding 320 until a determination is made that the particular base resolution video frame 308 corresponds to a key frame. For example, the base resolution video frame 308 can correspond to an I-frame, or the controller 160 (or the decoder 102) can determine that the number of blocks for which motion compensation is bypassed (e.g., due to the energy metric 248 exceeding the dynamic threshold 232) exceeds a threshold. Determining that the number of blocks for which motion compensation is bypassed exceeds a threshold can trigger the controller 160 (or the decoder 102) to designate the frame as a key frame, resetting the transmission distance 244 to zero and the dynamic threshold 232 for the key frame. The base resolution video frame 308 is upscaled in a similar manner as described for the first base resolution video frame 304 to generate a corresponding enhanced resolution frame 318.

[0078] Figure 4 is a diagram of an illustrative aspect of operations 400 performed at the decoder 102, including temporal prediction of enhanced resolution frames in the enhanced resolution mode 164. A first base resolution frame 402 is generated at the decoder 102, and upsampling 410 is performed to generate a first enhanced resolution frame 412. For example, the first enhanced resolution frame 412 can be an enhanced resolution key frame 122 generated by the upsampling engine 104, or can be an enhanced resolution frame generated by operation of the decoder 102 in the enhanced resolution mode 164.

[0079] The second base resolution frame 405 is generated at the decoder 102 using inter- frame prediction in the base resolution mode 162. For example, the bitstream 106 can include a motion vector 408 that maps a block 406 of the second base resolution frame 405 to a block 404 of the first base resolution frame 402. The temporal prediction engine 202 copies the block 404 from the first base resolution frame 402 in the decoded picture buffer 118 and outputs the block 404 (e.g., as a base resolution prediction 254A) to the reconstruction engine 208. The reconstruction engine 208 obtains a residual 430 from the bitstream 106 (e.g., residual pixels 240) and adds the residual 430 to the block 404 (e.g., base resolution prediction 254A) to generate a reconstructed base resolution block 406 for the second base resolution frame 405.

[0080] Following the generation of the second base resolution frame 405, the decoder 102 switches to operation in the enhanced resolution mode 164 to generate a second enhanced resolution frame 450. The decoder 102 (e.g., the upscaler 220A of the prediction engine 114) upscales the motion vector 408 and uses the upscaled motion vector to retrieve 416 a corresponding block 414 of the first enhanced resolution frame 412. For example, the temporal prediction engine 202 copies the block 414 from the first enhanced resolution frame 412 in the decoded picture buffer 118 and outputs the block 414 (e.g., as an enhanced resolution prediction 254B) to the reconstruction engine 208, which can include fractional interpolation due to upsampling. The reconstruction engine 208 obtains an upscaled residual 432 by upsampling the residual 430 (e.g., at the upsampler 220B of the prediction engine 114) and adds the upscaled residual 432 to the block 414 (e.g., enhanced resolution prediction 254B) to generate a reconstructed enhanced resolution block 442 for the second enhanced resolution frame 450.

[0081] Figure 5 FIG. 1 is a diagram of an example of a decoder 102 in accordance with some examples of the present disclosure. Figure 1 FIG. 2 is a diagram of example aspects of operation of components 500 of the decoder 102 of FIG. 1 in accordance with some examples of the present disclosure. The prediction engine 114 is configured to receive a bitstream representation (e.g., the bitstream 106) of a sequence 502 of frames. The sequence 502 of frames is illustrated as a first frame (Fl), a second frame (F2), and one or more additional frames including an Nth frame (FN), where N is an integer greater than two. The prediction engine 114 (and / or the filter engine 116, which is not illustrated for purposes of clarity) is configured to output a sequence 510 of base resolution versions of the frames Fl-FN, which are illustrated as base resolution frames Bl-BN.

[0082] After generating each base resolution frame B1, B2,..., BN in the sequence 510, the each base resolution frame is stored in the decoded picture buffer 118, where it can be used as a source of base resolution blocks 150 for decoding subsequent frames in the sequence 502 of frames, and the decoder 102 switches to the enhanced resolution mode 164 to generate corresponding enhanced resolution versions of the frames, resulting in a sequence 520 of enhanced resolution frames. The sequence 520 of enhanced resolution frames includes a frame E1 corresponding to an enhanced resolution version of the frame B1, a frame E2 corresponding to an enhanced resolution version of the frame B2, and a frame EN corresponding to an enhanced resolution version of the frame BN.

[0083] After generating each frame E1-EN in the sequence 520, the each frame is stored in the decoded picture buffer 118, where it can be used as a source of enhanced resolution blocks 152 for decoding subsequent frames in the sequence 502 of frames, and the decoder 102 switches back to the base resolution mode 162 to generate the next frame in the sequence 510. The decoder 102 generates an output sequence 530 of frames corresponding to the sequence 520 of enhanced resolution frames.

[0084] Figure 6 is a timing diagram 600 of base resolution decoding and enhanced resolution generation at the decoder 102 according to a particular implementation. For example, the decoder 102 is configured to receive a bitstream representation of the sequence 502 of frames Figure 5 and to alternate between the base resolution mode 162 and the enhanced resolution mode 164 when generating the decoder output 108.

[0085] In particular implementations, the decoder 102 is configured to, for each particular frame in the sequence 520 and during a single frame decoding time 620, reconstruct a base resolution version of the particular frame (e.g., a base resolution frame 138) and generate an enhanced resolution version of the particular frame (e.g., an enhanced resolution frame 148). As described above with reference to Figure 5 the decoder 102 is further configured to store the base resolution version of the particular frame and the enhanced resolution version of the particular frame in the decoded picture buffer 118, and subsequently use the base resolution version of the particular frame from the decoded picture buffer 118 as a source block for motion transfer under the base resolution mode 162, and use the enhanced resolution version of the particular frame from the decoded picture buffer 118 as a source block for motion transfer under the enhanced resolution mode 164.

[0086] As illustrated by the timing diagram 600, the decoder 102 is controlled (e.g., via the controller 160) such that, for each particular frame in the sequence 520, the base resolution version of the particular frame is reconstructed and stored in the decoded picture buffer 118 during a first time period 622, and the enhanced resolution version of the particular frame is generated and stored in the decoded picture buffer 118 during a second time period 624. Figure 5During the first decoding time 620A associated with F1 in the image, prediction engine 114 operates according to base resolution mode 162 to generate a base resolution version of a first specific frame, which is exemplified as base resolution frame 138A (e.g., ...). Figure 5 (B1). Also during the first decoding time 620A, prediction engine 114 operates according to enhanced resolution mode 164 to generate an enhanced resolution version of a first specific frame, which is exemplified as enhanced resolution frame 148A (e.g., B1). Figure 5 E1).

[0087] Following the first decoding time 620A in sequence and the second specific frame following the first specific frame in sequence (e.g., Figure 5 During the second decoding time 620B associated with F2, prediction engine 114 operates according to base resolution mode 162 to generate a base resolution version of a second specific frame, which is exemplified as base resolution frame 138B (e.g., F2). Figure 5 (B2). During the second decoding time 620B associated with the second frame, prediction engine 114 operates according to enhanced resolution mode 164 to generate an enhanced resolution version of the second specific frame, exemplified as enhanced resolution frame 148B (e.g., B2). Figure 5 E2).

[0088] In the Nth specific frame (e.g., Figure 5 During the Nth decoding time 620N associated with the FN), prediction engine 114 operates according to base resolution mode 162 to generate a base resolution version of the Nth specific frame, which is exemplified as base resolution frame 138N (e.g., Figure 5 During the Nth decoding time 620N associated with the Nth frame, prediction engine 114 operates according to enhanced resolution mode 164 to generate an enhanced resolution version of the Nth specific frame, which is exemplified as enhanced resolution frame 148N (e.g., BN). Figure 5 (EN).

[0089] Figure 7 A specific embodiment 700 of device 101 is depicted as an integrated circuit 702 including one or more processors 790. The one or more processors 790 include a decoding unit 740. The decoding unit 740 includes... Figure 1decoder 102, controller 160, and amplification engine 104. The integrated circuit 702 also includes an input 704 (such as one or more bus interfaces) to enable receiving the bitstream 106 for processing. The integrated circuit 702 also includes an output 706 (such as a bus interface) to enable transmitting an output signal, such as the decoder output 108. The integrated circuit 702 enables the device 101 to be implemented as a component in a system that includes a display and / or other components, such as: Figure 8 a mobile phone or tablet as depicted in Figure 9 a headset as depicted in Figure 10 a wearable electronic device as depicted in Figure 11 a sound-controlled speaker system as depicted in Figure 12 a camera as depicted in Figure 13 a virtual reality, mixed reality, or augmented reality headset as depicted in Figure 14 or Figure 15 a vehicle.

[0090] As illustrative, non-limiting examples, Figure 8 a particular implementation 800 is depicted in which the device 101 includes a mobile device 802, such as a phone or tablet. The mobile device 802 includes a plurality of speakers 820 and a display screen 804. In addition, the decoding unit 740 is integrated in the mobile device 802, and is illustrated using dashed lines to indicate internal components that are generally not visible to a user of the mobile device 802. In a particular example, the decoding unit 740 operates to process a video bitstream that can be received wirelessly from another device (e.g., a streaming video source) or that can be retrieved from local storage (e.g., a video recording captured and saved at the mobile device 802), which is then processed at the decoding unit 740 to generate enhanced resolution video content (e.g., enhanced resolution frames generated by the decoder 102) that can be played out at the display screen 804. Audio associated with the enhanced resolution video content can be played out via the speakers 820.

[0091] Figure 9A particular implementation 900 is depicted in which the device 101 comprises a headset device 902. The headset device 902 comprises a microphone 910 and a speaker 920. In addition, the decoding unit 740 is integrated in the headset device 902. In a particular example, the decoding unit 740 operates to process a video bitstream that is wirelessly receivable from another device (e.g., a streaming video source) or is retrievable from local or network storage (e.g., a video recording captured and saved at the headset device 902), which is then processed at the decoding unit 740 to generate enhanced resolution video content (e.g., enhanced resolution frames generated by the decoder 102). The enhanced resolution video content can be played out at a display screen (not shown) integrated with or coupled to the headset device 902. Additionally, audio associated with the enhanced resolution video content can be played out via the speaker 920.

[0092] Figure 10 A particular implementation 1000 is depicted in which the device 101 comprises a wearable electronic device 1002, illustrated as a "smart watch." The decoding unit 740, a speaker 1020, and a display screen 1004 are integrated into the wearable electronic device 1002. In a particular example, the decoding unit 740 operates to process a video bitstream that is wirelessly receivable from another device (e.g., a streaming video source) or is retrievable from local or network storage, such as a video recording captured and saved at the wearable electronic device 1002. The video bitstream can be processed at the decoding unit 740 to generate enhanced resolution video content (e.g., enhanced resolution frames generated by the decoder 102) that can be played out at the display screen 1004. Additionally, audio associated with the enhanced resolution video content can be played out via the speaker 1020.

[0093] In a particular example, the wearable electronic device 1002 comprises a haptic device that provides a haptic notification (e.g., a vibration) in response to detection of an activity with respect to video decoding and playback. For example, the haptic notification can cause the user to look at the wearable electronic device 1002 to see a display notification indicating a change in resolution suggested for video playback, such as a change from enhanced resolution to base resolution in response to detecting a low battery condition at the wearable electronic device 1002, or a change from base resolution to enhanced resolution in response to determining that a battery of the wearable electronic device 1002 is charging. The wearable electronic device 1002 can thus alert such notifications to a user with a hearing impairment or a user wearing a headset.

[0094] Figure 11is a particular implementation 1100 in which the device 101 includes a wireless speaker and voice-activated device 1102. The wireless speaker and voice-activated device 1102 can have wireless network connectivity and be configured to perform auxiliary operations. One or more processors 1190 including the decoding unit 740, the first microphone 1110, the second microphone 1120, the speaker 1104, or a combination thereof are included in the wireless speaker and voice-activated device 1102. In a particular example, the decoding unit 740 operates to process a video bitstream that can be wirelessly received from another device (e.g., a streaming video source) or that can be retrieved from local or network storage, such as a video recording captured and saved at the wireless speaker and voice-activated device 1102. The video bitstream can be processed at the decoding unit 740 to generate enhanced resolution video content (e.g., enhanced resolution frames generated by the decoder 102) that can be played out at a display screen (not shown) integrated with or coupled to the wireless speaker and voice-activated device 1102. Additionally, audio associated with the enhanced resolution video content can be played out via the speaker 1104.

[0095] During operation, in response to receiving a verbal command from a user via the microphones 1110, 1120, the wireless speaker and voice-activated device 1102 can perform an assistant operation, such as via execution of a voice-activated system (e.g., an integrated assistant application). The auxiliary operation can include adjusting a temperature, playing music, turning on a light, and the like. For example, the auxiliary operation can include initiating video playback at the decoding unit 740.

[0096] Figure 12 is depicted in which the device 101 includes a portable electronic device corresponding to a camera device 1202. The decoding unit 740, the speaker 1220, or a combination thereof are included in the camera device 1202. In a particular example, the decoding unit 740 operates to process a video bitstream that can be wirelessly received from another device (e.g., a streaming video source) or that can be retrieved from local or network storage, such as a video recording captured and saved at the camera device 1202. The video bitstream can be processed at the decoding unit 740 to generate enhanced resolution video content (e.g., enhanced resolution frames generated by the decoder 102) that can be played out at a display screen (not shown) integrated with or coupled to the camera device 1202. Additionally, audio associated with the enhanced resolution video content can be played out via the speaker 1220.

[0097] Figure 13A particular implementation 1300 is depicted in which the device 101 comprises a portable electronic device corresponding to a virtual reality, mixed reality, or augmented reality headset 1302. The decoding unit 740 and the speaker 1320 are integrated into the headset 1302. A visual interface device, such as a display screen, is positioned in front of the user's eyes to enable augmented reality, mixed reality, or virtual reality images or scenes to be displayed to the user when the headset 1302 is worn. In a particular example, the decoding unit 740 operates to process a video bitstream that can be received wirelessly from another device (e.g., a streaming video source) or that can be retrieved from local or network storage, such as a video recording. The video bitstream can be processed at the decoding unit 740 to generate enhanced resolution video content (e.g., enhanced resolution frames generated by the decoder 102) that can be played out at the visual interface device. Additionally, audio associated with the enhanced resolution video content can be played out via the speaker 1320, such as a transducer of one or more earphones coupled to or integrated with the headset 1302.

[0098] Figure 14 A particular implementation 1400 is depicted in which the device 101 corresponds to or is integrated within a vehicle 1402, illustrated as a manned or unmanned aerial device (e.g., a package delivery drone). The decoding unit 740, the speaker 1420, the display device 1404, or a combination thereof are integrated into the vehicle 1402. In a particular example, the decoding unit 740 operates to process a video bitstream that can be received wirelessly from another device (e.g., a streaming video source) or that can be retrieved from local or network storage, such as a video recording. The video bitstream can be processed at the decoding unit 740 to generate enhanced resolution video content (e.g., enhanced resolution frames generated by the decoder 102) that can be played out at the display device 1404. Additionally, audio associated with the enhanced resolution video content can be played out via the speaker 1420. For example, the video content can include unboxing or other instructions that can be played out to a recipient of a package being delivered by the vehicle 1402.

[0099] Figure 15Another implementation 1500 is depicted in which the device 101 corresponds to or is integrated within a vehicle 1502, illustrated as an automobile. The decoding unit 740, one or more speakers 1510, a display device 1520, or a combination thereof are integrated into the vehicle 1502. In a particular example, the decoding unit 740 operates to process a video bitstream that can be received wirelessly from another device (e.g., a streaming video source) or that can be retrieved from local or network storage, such as a video recording. The video bitstream can be processed at the decoding unit 740 to generate enhanced resolution video content (e.g., enhanced resolution frames generated by the decoder 102) that can be played out at the display device 1520. Additionally, audio associated with the enhanced resolution video content can be played out via the one or more speakers 1510. In some implementations, the decoding unit 740 is included in a vehicle entertainment system and can be configured to provide video content for various occupants of the vehicle 1502 at one or more display screens.

[0100] Figure 16 is a diagram of a particular implementation of a method of performing enhanced resolution generation. In a particular aspect, one or more operations of the method 1600 are performed by Figure 1 the decoder 102, the controller 160, the upscaling engine 104, the device 101, the system 100, or a combination thereof.

[0101] The method 1600 includes reconstructing, at 1602 at a decoder, a base resolution version of a block of a frame using at least one of a spatial prediction engine or a temporal prediction engine of the decoder and a reconstruction engine. For example, the prediction engine 114 of the decoder 102 generates the reconstructed base resolution block 274A using at least one of the temporal prediction engine 202 or the spatial prediction engine 204 and the reconstruction engine 208.

[0102] The method 1600 includes changing, at 1604 after reconstructing the base resolution version of the block, an operational mode of the decoder from a base resolution mode to an enhanced resolution mode. For example, the controller 160 causes the decoder 102 to switch from operating in the base resolution mode 162 to operating in the enhanced resolution mode 164. According to an aspect, the switch between operating in the base resolution mode 162 and operating in the enhanced resolution mode 164 is performed after completion of reconstruction of all blocks of a current frame, such as described with reference to Figure 5 and Figure 6 .

[0103] The method 1600 also includes generating, at the decoder, an enhanced resolution version of the block after reconstructing the base resolution version of the block using the at least one of the spatial prediction engine or the temporal prediction engine and the reconstruction engine, at 1606. For example, the prediction engine 114 of the decoder 102 generates the reconstructed enhanced resolution block 274B using the same one of the temporal prediction engine 202 or the spatial prediction engine 204 and the reconstruction engine 208 that was used to generate the reconstructed base resolution block 274A.

[0104] In some implementations, the method 1600 includes generating a bypass control signal based on a comparison of an energy metric of a block’s residual to a dynamic threshold. The dynamic threshold is based on a decay factor and a transmission distance from a most recently generated enhanced resolution key frame. For example, the adaptive transmission logic 230 generates the flag 246 based on a comparison of the energy metric 248 to the dynamic threshold 232. As explained earlier, the dynamic threshold 232 can be determined as THR bypass *β i where β represents a decay factor and i represents a transmission distance from a most recent key frame. The method 1600 can also include bypassing operation of one or more of an upsampler, a temporal prediction engine, a spatial prediction engine, or a reconstruction engine based on the bypass control signal. For example, one or more of the upsamplers 220A, 220B, the temporal prediction engine 202, the spatial prediction engine 204, or the reconstruction engine 208 can be bypassed or deactivated based on the value of the flag 246.

[0105] In some implementations, the method 1600 includes offloading upsampling of a key frame to an upsampling engine to generate an enhanced resolution version of the key frame. For example, the base resolution key frame 120 is offloaded to the upsampling engine 104 to generate the enhanced resolution key frame 122. In some implementations, after generating the enhanced resolution version of the key frame, enhanced resolution versions of one or more subsequent frames can be generated by a decoder operating in an enhanced resolution mode based on blocks of the enhanced resolution version of the key frame. To illustrate, the enhanced resolution key frame 122 can be stored in the decoded picture buffer 118 as an enhanced resolution frame 148 to be available as a reference frame to generate subsequent enhanced resolution frames.

[0106] Therefore, method 1600 enables the generation of enhanced resolution frames using the same components of decoder 102 used for basic resolution reconstruction of the frames, as well as information from the bitstream. This provides the technical advantage of reduced power consumption and a smaller amount of processing resources required to generate enhanced resolution frames compared to using amplification engine 104 to amplify each frame. Using the same components of decoder 102 to decode the basic resolution frames and generate enhanced resolution frames also provides the technical benefits of reduced component number, size, and data transmission bandwidth compared to a specific implementation where bitstream data and basic resolution frames are transmitted from decoder 102 to another set of components used to generate enhanced resolution frames.

[0107] Figure 16 Method 1600 can be implemented by a field-programmable gate array (FPGA) device, an application-specific integrated circuit (ASIC), a processing unit (such as a central processing unit (CPU)), a DSP, a controller, another hardware device, a firmware device, or any combination thereof. As an example, Figure 16 Method 1600 can be executed by a processor that executes instructions, such as reference Figure 16 As described.

[0108] refer to Figure 17 A block diagram depicting a specific, exemplary embodiment of the device is provided, and is generally designated as 1700. In various embodiments, device 1700 may have... Figure 17 More or fewer components are illustrated. In an illustrative embodiment, device 1700 may correspond to device 101. In an illustrative embodiment, device 1700 may perform reference... Figures 1 to 16 One or more operations as described.

[0109] In a particular implementation, device 1700 includes a processor 1706 (e.g., a central processing unit (CPU)). Device 1700 may include one or more additional processors 1710 (e.g., one or more DSPs). Processor 1710 may include a speech and music decoder-decoder (codec) 1708, which includes a speech decoder (“phonecoder”) encoder 1736, a phonecoder decoder 1738, a decoding unit 740, or a combination thereof.

[0110] Device 1700 can include memory 1786 and codec 1734. Memory 1786 can include instructions 1756 that are executable by one or more additional processors 1710 (or processor 1706) to implement the functionality described with reference to decoding unit 740, or both. Device 1700 can include modem 1748 coupled to antenna 1752 via transceiver 1750. Encoded video data (such as bitstream 106) can be received from remote video source device 1790 via wireless transmissions 1796 received at antenna 1752, which can be further processed by decoding unit 740 for playout of enhanced resolution video at display 1728.

[0111] Device 1700 can include display 1728 coupled to display controller 1726. Speaker 1792 and microphone 1794 can be coupled to codec 1734. Codec 1734 can include digital-to-analog converter (DAC) 1702, analog-to-digital converter (ADC) 1704, or both. In particular implementations, codec 1734 can receive analog signals from microphone 1794, use analog-to-digital converter 1704 to convert the analog signals to digital signals, and provide the digital signals to speech and music codec 1708. Speech and music codec 1708 can process the digital signals. In particular implementations, speech and music codec 1708 can provide the digital signals to codec 1734. Codec 1734 can use digital-to-analog converter 1702 to convert the digital signals to analog signals, and can provide the analog signals to speaker 1792.

[0112] In particular implementations, device 1700 can be included in a system-in- package or system-on-chip device 1722. In particular implementations, memory 1786, processor 1706, processor 1710, display controller 1726, codec 1734, and modem 1748 are included in system-in-package or system-on-chip device 1722. In particular implementations, input device 1730 and power source 1744 are coupled to system-in-package or system-on-chip device 1722. Also, in particular implementations, as illustrated in FIG. 17B, display 1728, input device 1730, speaker 1792, microphone 1794, antenna 1752, and power source 1744 are external to system-in-package or system-on-chip device 1722. In particular implementations, each of display 1728, input device 1730, speaker 1792, microphone 1794, antenna 1752, and power source 1744 can be coupled to a component of system-in-package or system-on-chip device 1722, such as an interface or a controller. Figure 17

[0113] ​Device 1700 can include a smart speaker, a speaker bar, a mobile communication device, a smartphone, a cellular telephone, a laptop computer, a computer, a tablet computer, a personal digital assistant, a display device, a television, a gaming console, a music player, a radio, a digital video player, a digital video disc (DVD) player, a tuner, a camera, a navigation device, a vehicle, a head-worn device, an augmented reality head-worn device, a mixed reality head-worn device, a virtual reality head-worn device, an aerial vehicle, a home automation system, a voice-activated device, a wireless speaker and voice-activated device, a portable electronic device, an automobile, a computing device, a communication device, an Internet of Things (IoT) device, a virtual reality (VR) device, a base station, a mobile device, or any combination thereof.

[0114] In connection with the described particular implementations, an apparatus includes means for decoding that includes reconstructing, in a base resolution mode, a base resolution version of a block of a frame using at least one of a spatial prediction engine or a temporal prediction engine and a reconstruction engine, and generating, in an enhanced resolution mode, an enhanced resolution version of the block using the at least one of the spatial prediction engine or the temporal prediction engine and the reconstruction engine. For example, the means for reconstructing a base resolution version of a block and for generating an enhanced resolution version of a block can correspond to syntax engine 110, transform engine 112, prediction engine 114, filter engine 116, decoded picture buffer 118, decoder 102, amplifier 220, temporal prediction engine 202, spatial prediction engine 204, selector 206, reconstruction engine 208, selector 210, decoding unit 740, one or more processors 790, processor 1706, one or more processors 1710, one or more other circuits or components configured to perform decoding, including reconstructing, in a base resolution mode, a base resolution version of a block of a frame using at least one of a spatial prediction engine or a temporal prediction engine and a reconstruction engine, and generating, in an enhanced resolution mode, an enhanced resolution version of the block using the at least one of the spatial prediction engine or the temporal prediction engine and the reconstruction engine, or any combination thereof.

[0115] The apparatus also includes means for controlling a mode of operation of the means for decoding. For example, the means for controlling a mode of operation of the means for decoding can correspond to controller 160, device 101, decoding unit 740, one or more processors 790, processor 1706, one or more processors 1710, one or more other circuits or components configured to control a mode of operation of the means for decoding, or any combination thereof.

[0116] In some implementations, a non-transitory computer-readable medium (e.g., a computer-readable storage device such as memory 1786) includes instructions (e.g., instructions 1756) that, when executed by one or more processors (e.g., one or more processors 1710 or processor 1706), cause the one or more processors to, at a decoder (e.g., decoder 102), reconstruct a base resolution version of a block of a frame using at least one of a spatial prediction engine (e.g., spatial prediction engine 204 or temporal prediction engine 202) and a reconstruction engine (e.g., reconstruction engine 208) of the decoder; and after reconstructing the base resolution version of the block: change an operational mode of the decoder from a base resolution mode (e.g., base resolution mode 162) to an enhanced resolution mode (e.g., enhanced resolution mode 164); and generate, at the decoder, an enhanced resolution version of the block using the at least one of the spatial prediction engine or the temporal prediction engine and the reconstruction engine.

[0117] Certain aspects of the disclosure are described below with respect to a set of related embodiments:

[0118] According to embodiment 1, an apparatus comprises: a decoder comprising a spatial prediction engine, a temporal prediction engine, a reconstruction engine, and a decoded picture buffer; and a controller configured to cause the decoder to: in a base resolution mode, reconstruct a base resolution version of a block of a frame using at least one of the spatial prediction engine or the temporal prediction engine and the reconstruction engine; and in an enhanced resolution mode, generate an enhanced resolution version of the block using the at least one of the spatial prediction engine or the temporal prediction engine and the reconstruction engine.

[0119] Embodiment 2 includes the apparatus of embodiment 1, wherein based on receiving an inter prediction indicator for the block, the decoder is configured to: in the base resolution mode: generate, at the temporal prediction engine, a base resolution prediction of the block using a motion vector to copy pixels of a base resolution version of a reference frame from the decoded picture buffer; and generate, at the reconstruction engine, a base resolution reconstruction of the block based on the base resolution prediction and a residual of the block; and in the enhanced resolution mode: upscale the motion vector to generate an upscaled motion vector; upscale the residual to generate an upscaled residual; generate, at the temporal prediction engine, an enhanced resolution prediction of the block using the upscaled motion vector to copy pixels of an enhanced resolution version of the reference frame from the decoded picture buffer; and generate, at the reconstruction engine, an enhanced resolution reconstruction of the block based on the enhanced resolution prediction and the upscaled residual.

[0120] Example 3 includes the device of Example 1 or Example 2, wherein the decoder is configured to generate a bypass control signal based on a comparison of an energy metric of a residual of the block to a dynamic threshold, wherein the dynamic threshold is based on an attenuation factor and a transmission distance from a most recently generated enhanced resolution key frame.

[0121] Example 4 includes the device of Example 3, wherein the spatial prediction engine, the temporal prediction engine, and the reconstruction engine are included in a prediction engine of the decoder, and wherein in the enhanced resolution mode, the decoder is further configured to select, based on the bypass control signal, one of an output of the reconstruction engine or an up-scaled version of the base resolution version of the block as an output of the prediction engine.

[0122] Example 5 includes the device of Example 3 or Example 4, wherein the decoder is configured to bypass operation of one or more of an up-sampler, the temporal prediction engine, the spatial prediction engine, or the reconstruction engine based on the bypass control signal.

[0123] Example 6 includes the device of any of Examples 1-5 and further comprising an upscaling engine coupled to the decoder, wherein the decoder is configured to offload upscaling of a key frame to the upscaling engine to generate an enhanced resolution version of the key frame.

[0124] Example 7 includes the device of any of Example 6, wherein the upscaling engine comprises a machine learning model.

[0125] Example 8 includes the device of Example 6 or Example 7, wherein, after generation of an enhanced resolution version of a key frame, enhanced resolution versions of one or more subsequent frames are generated by the decoder in the enhanced resolution mode based on blocks of the enhanced resolution version of the key frame.

[0126] Example 9 includes the device of any of Examples 1-8, wherein the decoder further comprises an up-sampler configured to generate an up-scaled residual of the block, and wherein the reconstruction engine is configured to use the up-scaled residual during generation of the enhanced resolution version of the block.

[0127] Example 10 includes the device of any of Examples 1-9, wherein the decoder is configured to receive a motion vector for the block via a bitstream, and to upscale the motion vector in the enhanced resolution mode.

[0128] Example 11 includes the device of Example 10, wherein the decoder is configured to transmit an enhanced resolution pixel based on the up-scaled motion vector.

[0129] Example 12 includes the device of any of Examples 1-11, wherein the decoder is configured to: receive a bitstream representation of a sequence of frames; and for each particular frame in the sequence and during a single frame decode time, reconstruct a base resolution version of the particular frame and generate an enhanced resolution version of the particular frame.

[0130] Example 13 includes the device of Example 12, wherein the decoder is further configured to store the base resolution version of the particular frame and the enhanced resolution version of the particular frame in the decoded picture buffer.

[0131] Example 14 includes the device of Example 13, wherein the decoder is configured to use the enhanced resolution version of the particular frame from the decoded picture buffer as a source block for motion transfer in the enhanced resolution mode.

[0132] Example 15 includes the device of any of Examples 12-14, wherein the spatial prediction engine, the temporal prediction engine, and the reconstruction engine are included in a prediction engine of the decoder, and wherein the controller is configured to cause the decoder to: during a first decode time associated with a first particular frame in the sequence: operate the prediction engine according to the base resolution mode to generate a base resolution version of the first particular frame; and operate the prediction engine according to the enhanced resolution mode to generate an enhanced resolution version of the first particular frame; and during a second decode time that follows the first decode time in sequence and that is associated with a second particular frame that follows the first particular frame in sequence: operate the prediction engine according to the base resolution mode to generate a base resolution version of the second particular frame; and operate the prediction engine according to the enhanced resolution mode to generate an enhanced resolution version of the second particular frame.

[0133] Example 16 includes the device of any of Examples 1-15 and further comprising a display device configured to play out the enhanced resolution versions of the frames generated by the decoder.

[0134] Example 17 includes the device of Example 16 and further comprising one or more speakers configured to play out audio associated with the frames.

[0135] Example 18 includes the device of any of Examples 1-17 and further comprising a modem configured to receive the sequence of frames from an encoder device via a bitstream.

[0136] Example 19 includes the device of any of Examples 1-18, wherein the decoder and the controller are included in an integrated circuit.

[0137] Example 20 includes the device of any of Examples 1-19, wherein the decoder and the controller are integrated in a head-mounted device.

[0138] Example 21 includes the device of Example 20, wherein the head-mounted device corresponds to at least one of a virtual reality head-mounted device, a mixed reality head-mounted device, or an augmented reality head-mounted device.

[0139] Example 22 includes the device of any of Examples 1-19, wherein the decoder and the controller are integrated in at least one of a mobile phone, a tablet device, or a wearable electronic device.

[0140] Example 23 includes the device of any of Examples 1-19, wherein the decoder and the controller are integrated in a vehicle.

[0141] According to Example 24, a method includes reconstructing, at a decoder, a base resolution version of a block of a frame using at least one of a spatial prediction engine or a temporal prediction engine of the decoder and a reconstruction engine; and after reconstructing the base resolution version of the block: changing an operating mode of the decoder from a base resolution mode to an enhanced resolution mode; and generating, at the decoder, an enhanced resolution version of the block using the at least one of the spatial prediction engine or the temporal prediction engine and the reconstruction engine.

[0142] Example 25 includes the method of Example 24, the method further comprising: based on receiving an inter prediction indicator for the block: when the operating mode of the decoder is the base resolution mode: generating, at the temporal prediction engine, a base resolution prediction of the block using a motion vector to copy pixels of a base resolution version of a reference frame from the decoded picture buffer; and generating, at the reconstruction engine, a base resolution reconstruction of the block based on the base resolution prediction and a residual of the block; and when the operating mode of the decoder is the enhanced resolution mode: upsampling the motion vector to generate an upscaled motion vector; upsampling the residual to generate an upscaled residual; generating, at the temporal prediction engine, an enhanced resolution prediction of the block using the upscaled motion vector to copy pixels of an enhanced resolution version of the reference frame from the decoded picture buffer; and generating, at the reconstruction engine, an enhanced resolution reconstruction of the block based on the enhanced resolution prediction and the upscaled residual.

[0143] Example 26 includes the method of Example 24 or Example 25, the method further comprising generating a bypass control signal based on a comparison of an energy metric of a residual of the block to a dynamic threshold, wherein the dynamic threshold is based on an attenuation factor and a transmission distance from a most recently generated enhanced resolution key frame.

[0144] Example 27 includes the method of Example 26, wherein the spatial prediction engine, the temporal prediction engine, and the reconstruction engine are included in a prediction engine of the decoder, the method further comprising selecting, based on the bypass control signal, one of an output of the reconstruction engine or an upscaled version of the base resolution version of the block as an output of the prediction engine.

[0145] Example 28 includes the method of Example 26 or Example 27 and further comprising bypassing, based on the bypass control signal, operation of one or more of an upsampler, the temporal prediction engine, the spatial prediction engine, or the reconstruction engine.

[0146] Example 29 includes the method of any of Examples 24-28 and further comprising offloading upsampling of a key frame to an upsampling engine to generate an enhanced resolution version of the key frame.

[0147] Example 30 includes the method of Example 29, wherein the upsampling engine comprises a machine learning model.

[0148] Example 31 includes the method of Example 28 or Example 29, wherein, after generating an enhanced resolution version of a key frame, enhanced resolution versions of one or more subsequent frames are generated by the decoder operating in the enhanced resolution mode based on blocks of the enhanced resolution version of the key frame.

[0149] Example 32 includes the method of any of Examples 24-31, further comprising generating an upscaled residual of the block; and using the upscaled residual at the reconstruction engine during generation of the enhanced resolution version of the block.

[0150] Example 33 includes the method of any of Examples 24-32, further comprising receiving, at the decoder via a bitstream, a motion vector for the block; and upsampling, by the decoder operating in the enhanced resolution mode, the motion vector.

[0151] Example 34 includes the method of Example 33 and further comprising transmitting, based on the upscaled motion vector, an enhanced resolution pixel.

[0152] Example 35 includes the method of any one of Examples 24-34, further comprising: receiving a bitstream representation of a sequence of frames; and for each particular frame in the sequence and during a single frame decoding time: reconstructing a base resolution version of the particular frame; and generating an enhanced resolution version of the particular frame.

[0153] Example 36 includes the method of Example 35 and further comprising storing the base resolution version of the particular frame and the enhanced resolution version of the particular frame in the decoded picture buffer.

[0154] Example 37 includes the method of Example 36 and further comprising using, by the decoder operating in the enhanced resolution mode, the enhanced resolution version of the particular frame from the decoded picture buffer as a motion-transferred source block.

[0155] Example 38 includes the method of any one of Examples 35-37, wherein the spatial prediction engine, the temporal prediction engine, and the reconstruction engine are included in a prediction engine of the decoder, and the method further comprises: during a first decoding time associated with a first particular frame in the sequence: operating the prediction engine according to the base resolution mode to generate a base resolution version of the first particular frame; and operating the prediction engine according to the enhanced resolution mode to generate an enhanced resolution version of the first particular frame; and during a second decoding time following the first decoding time in sequence and associated with a second particular frame following the first particular frame in sequence: operating the prediction engine according to the base resolution mode to generate a base resolution version of the second particular frame; and operating the prediction engine according to the enhanced resolution mode to generate an enhanced resolution version of the second particular frame.

[0156] Example 39 includes the method of any one of Examples 24-38 and further comprising playing out, at a display device, an enhanced resolution version of a frame generated by the decoder.

[0157] Example 40 includes the method of Example 39 and further comprising playing out, at one or more speakers, audio associated with the frame.

[0158] Example 41 includes the method of any one of Examples 24-40 and further comprising receiving, at a modem from an encoder device via a bitstream, a sequence of frames.

[0159] Example 42 includes the method of any one of Examples 24-41, wherein the decoder is included in an integrated circuit.

[0160] Example 43 includes the method of any of Examples 24-42, wherein the decoder is integrated in a head-mounted device apparatus.

[0161] Example 44 includes the method of Example 43, wherein the head-mounted device apparatus corresponds to at least one of a virtual reality head-mounted device, a mixed reality head-mounted device, or an augmented reality head-mounted device.

[0162] Example 45 includes the method of any of Examples 24-42, wherein the decoder is integrated in at least one of a mobile phone, a tablet device, or a wearable electronic device.

[0163] Example 46 includes the method of any of Examples 24-42, wherein the decoder is integrated in a vehicle.

[0164] According to Example 47, an apparatus includes a memory configured to store instructions and a processor configured to execute the instructions to perform the method of any of Examples 24-46.

[0165] According to Example 48, a non-transitory computer-readable medium includes instructions that, when executed by a processor, cause the processor to perform the method of any of Examples 24-46.

[0166] According to Example 49, an apparatus includes means for performing the method of any of Examples 24-46.

[0167] According to Example 50, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause the one or more processors to: reconstruct, at a decoder, a base resolution version of a block of a frame using at least one of a spatial prediction engine or a temporal prediction engine of the decoder and a reconstruction engine; and after reconstructing the base resolution version of the block: change an operational mode of the decoder from a base resolution mode to an enhanced resolution mode; and generate, at the decoder, an enhanced resolution version of the block using the at least one of the spatial prediction engine or the temporal prediction engine and the reconstruction engine.

[0168] According to an embodiment 51, an apparatus comprises: means for decoding including a spatial or temporal prediction engine and a reconstruction engine to reconstruct a base resolution version of a block of a frame in a base resolution mode and to generate an enhanced resolution version of the block in an enhanced resolution mode; and means for controlling an operational mode of the means for decoding.

[0169] Those of skill would further appreciate that the various illustrative logical blocks, configurations, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, computer software executed by a processor, or combinations of both. Various illustrative components, blocks, configurations, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or processor executable instructions depends upon the particular application and design constraints imposed on the overall system. Skilled persons can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0170] The steps of a method or algorithm described in connection with the implementations disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in Random Access Memory (RAM), flash memory, Read-only memory (ROM), Programmable Read-only memory (PROM), Erasable Programmable Read-only memory (EPROM), Electrically Erasable Programmable Read-only memory (EEPROM), registers, hard disk, a removable disk, a compact disc read-only memory (CD-ROM), or any other form of non-transitory storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium can be integral to the processor. The processor and the storage medium can reside in an Application-Specific Integrated Circuit (ASIC). The ASIC can reside in a computing device or a user terminal. In the alternative, the processor and the storage medium can reside as discrete components in a computing device or user terminal.

[0171] The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the disclosed aspects. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other aspects without departing from the scope of the disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A device comprising: a decoder comprising a spatial prediction engine, a temporal prediction engine, a reconstruction engine, and a decoded picture buffer; and a controller configured to cause the decoder to: in a base resolution mode, reconstruct a base resolution version of a block of a frame using at least one of the spatial prediction engine or the temporal prediction engine and the reconstruction engine; and in an enhanced resolution mode, generate an enhanced resolution version of the block using the at least one of the spatial prediction engine or the temporal prediction engine and the reconstruction engine.

2. The device of claim 1, wherein based on receiving an inter prediction indicator for the block, the decoder is configured to: in the base resolution mode: using a motion vector at the temporal prediction engine to generate a base resolution prediction of the block to copy pixels of a base resolution version of a reference frame from the decoded picture buffer; and generate a base resolution reconstruction of the block at the reconstruction engine based on the base resolution prediction and a residual of the block; and in the enhanced resolution mode: upscale the motion vector to generate an upscaled motion vector; upscale the residual to generate an upscaled residual; generate an enhanced resolution prediction of the block at the temporal prediction engine using the upscaled motion vector to copy pixels of an enhanced resolution version of the reference frame from the decoded picture buffer; and generate an enhanced resolution reconstruction of the block at the reconstruction engine based on the enhanced resolution prediction and the upscaled residual.

3. The device of claim 1, wherein the decoder is configured to generate a bypass control signal based on a comparison of an energy metric of a residual of the block to a dynamic threshold, wherein the dynamic threshold is based on a decay factor and a transmission distance from a most recently generated enhanced resolution key frame.

4. The device of claim 3, wherein the spatial prediction engine, the temporal prediction engine, and the reconstruction engine are included in a prediction engine of the decoder, and wherein in the enhanced resolution mode, the decoder is further configured to select, based on the bypass control signal, one of an output of the reconstruction engine or an upscaled version of the base resolution version of the block as an output of the prediction engine.

5. The device of claim 3, wherein the decoder is configured to bypass operation of one or more of an upsampler, the temporal prediction engine, the spatial prediction engine, or the reconstruction engine based on the bypass control signal.

6. The device of claim 1, further comprising an upsampling engine coupled to the decoder, wherein the decoder is configured to offload upsampling of key frames to the upsampling engine to generate enhanced resolution versions of the key frames.

7. The device of claim 6, wherein the upsampling engine comprises a machine learning model.

8. The device of claim 6, wherein after generating an enhanced resolution version of a key frame, enhanced resolution versions of one or more subsequent frames are generated by the decoder in the enhanced resolution mode based on blocks of the enhanced resolution version of the key frame.

9. The device of claim 1, wherein the decoder further comprises an upscaler configured to generate upscaled residuals for the block, and wherein the reconstruction engine is configured to use the upscaled residuals during generation of the enhanced resolution version of the block.

10. The device of claim 1, wherein the decoder is configured to receive motion vectors for the block via a bitstream, and to upscale the motion vectors in the enhanced resolution mode.

11. The device of claim 10, wherein the decoder is configured to transmit enhanced resolution pixels based on the upscaled motion vectors.

12. The device of claim 1, wherein the decoder is configured to: receive a bitstream representation of a sequence of frames; and for each particular frame in the sequence and during a single frame decoding time, reconstruct a base resolution version of the particular frame and generate an enhanced resolution version of the particular frame.

13. The device of claim 12, wherein the decoder is further configured to store the base resolution version of the particular frame and the enhanced resolution version of the particular frame in the decoded picture buffer.

14. The device of claim 13, wherein the decoder is configured to use the enhanced resolution version of the particular frame from the decoded picture buffer as a source block for motion transmission in the enhanced resolution mode.

15. The device of claim 12, wherein the spatial prediction engine, the temporal prediction engine, and the reconstruction engine are included in a prediction engine of the decoder, and wherein the controller is configured to cause the decoder to: during a first decoding time associated with a first particular frame in the sequence: operating the prediction engine according to the base resolution mode to generate a base resolution version of the first particular frame; and operate the prediction engine according to the enhanced resolution mode to generate an enhanced resolution version of the first particular frame; and during a second decoding time that follows the first decoding time in sequence and that is associated with a second particular frame that follows the first particular frame in sequence: operate the prediction engine according to the base resolution mode to generate a base resolution version of the second particular frame; and operate the prediction engine according to the enhanced resolution mode to generate an enhanced resolution version of the second particular frame.

16. The device of claim 1, further comprising a display device configured to play out enhanced resolution versions of frames generated by the decoder.

17. The device of claim 16, further comprising one or more speakers configured to play out audio associated with the frames.

18. The device of claim 1, further comprising a modem configured to receive a sequence of frames from an encoder device via a bitstream.

19. The device of claim 1, wherein the decoder and the controller are included in an integrated circuit.

20. The device of claim 1, wherein the decoder and the controller are integrated in a head-mounted device.

21. The device of claim 20, wherein the head-mounted device device corresponds to at least one of a virtual reality head-mounted device, a mixed reality head-mounted device, or an augmented reality head-mounted device.

22. The device of claim 1, wherein the decoder and the controller are integrated in at least one of a mobile phone, a tablet device, or a wearable electronic device.

23. The device of claim 1, wherein the decoder and the controller are integrated in a vehicle.

24. A method comprising: reconstructing, at a decoder, a base resolution version of a block of a frame using at least one of a spatial prediction engine or a temporal prediction engine of the decoder and a reconstruction engine; and after reconstructing the base resolution version of the block: changing an operating mode of the decoder from a base resolution mode to an enhanced resolution mode; and generating, at the decoder, an enhanced resolution version of the block using the at least one of the spatial prediction engine or the temporal prediction engine and the reconstruction engine.

25. The method of claim 24, further comprising: generating a bypass control signal based on a comparison of an energy metric of a residual of the block to a dynamic threshold, wherein the dynamic threshold is based on a decay factor and a transmission distance from a most recently generated enhanced resolution key frame.

26. The method of claim 25, further comprising: bypassing operation of one or more of an amplifier, the temporal prediction engine, the spatial prediction engine, or the reconstruction engine based on the bypass control signal.

27. The method of claim 24, further comprising: offloading magnification of a key frame to a magnification engine to generate an enhanced resolution version of the key frame.

28. The method of claim 27, wherein after generating an enhanced resolution version of a key frame, enhanced resolution versions of one or more subsequent frames are generated by the decoder operating in the enhanced resolution mode based on blocks of the enhanced resolution version of the key frame.

29. A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to: reconstructing, at the decoder, a base resolution version of the block of the frame using at least one of a spatial prediction engine or a temporal prediction engine of the decoder and a reconstruction engine; and after reconstructing the base resolution version of the block: change an operating mode of the decoder from a base resolution mode to an enhanced resolution mode; and generate, at the decoder, an enhanced resolution version of the block using the at least one of the spatial prediction engine or the temporal prediction engine and the reconstruction engine.

30. An apparatus comprising: means for decoding including reconstructing a base resolution version of a block of a frame using at least one of a spatial prediction engine or a temporal prediction engine and a reconstruction engine in a base resolution mode, and generating an enhanced resolution version of the block using the at least one of the spatial prediction engine or the temporal prediction engine and the reconstruction engine in an enhanced resolution mode; and means for controlling an operating mode of the means for decoding.

Images