Frame buffer in adaptive resolution management

Abstract

The present disclosure relates to frame buffering in adaptive resolution management. A method includes receiving a bitstream, decoding a first frame using the bitstream, determining a scaled first frame using the first frame and a scaling constant, storing the first frame at a first index location in a first picture buffer, and storing the scaled first frame at the first index location in a second picture buffer. Related devices, systems, techniques, and articles are also described.

Description

[0001] Information related to divisional application

[0002] This case is a divisional application. The parent application of this divisional application is the invention patent application filed on August 6, 2020, with application number 202080068756.3 and invention title "Frame Buffer in Adaptive Resolution Management".

[0003] Cross-references to related applications

[0004] This application claims priority to U.S. Provisional Patent Application No. 62 / 883,503, filed August 6, 2019, entitled "Frame Buffering in Adaptive Resolution Management," the entire contents of which are incorporated herein by reference. Technical Field

[0005] This invention generally relates to the field of video compression. Specifically, this invention relates to frame buffering in adaptive resolution management. Background Technology

[0006] Video codecs can include electronic circuitry or software that compresses or decompresses digital video. They can convert uncompressed video to a compressed format and vice versa. In the context of video compression, devices that compress video (and / or perform some of its functions) are generally referred to as encoders, and devices that decompress video (and / or perform some of its functions) are referred to as decoders.

[0007] The compressed data can conform to standard video compression specifications. Compression may be lossy because the compressed video may lack some information present in the original video. This can result in the decompressed video having lower quality than the original uncompressed video because there isn't enough information to accurately reconstruct the original video.

[0008] There can be complex relationships between video quality, the amount of data used to represent the video (e.g., determined by bit rate), the complexity of encoding and decoding algorithms, sensitivity to data loss and errors, ease of editing, random access, end-to-end latency (e.g., delay time), and so on.

[0009] Motion compensation can include methods for predicting a portion of a video frame or a given reference frame (e.g., a previous frame and / or a future frame) by taking into account the motion of objects in the camera and / or video. The methods can encode and decode video data for video compression, for example, using the Moving Picture Experts Group (MPEG) High-Level Video Coding (AVC) standard (also known as H.264).

[0010] Motion compensation can describe an image based on the transformation from a reference image to the current image. When compared with the current image, the reference image can be temporally previous, and / or, when compared with the current image, the reference image can be from the future. Summary of the Invention

[0011] On one hand, a decoder includes circuitry configured to receive a bitstream, decode a first frame using the bitstream, determine a scaled first frame using the first frame and a scaling constant, store the first frame at a first index position in a first image buffer, and store the scaled first frame at a first index position in a second image buffer.

[0012] On the other hand, one method includes: receiving a bitstream, decoding a first frame using the bitstream, determining a scaled first frame using the first frame and a scaling constant, storing the first frame at a first index position in a first image buffer, and storing the scaled first frame at a first index position in a second image buffer.

[0013] Details of one or more variations of the subject matter described herein are set forth in the accompanying drawings and detailed descriptions. Further features and advantages of the subject matter described herein will become apparent from the detailed descriptions, the drawings, and the claims. Attached Figure Description

[0014] To illustrate the invention, the accompanying drawings show various aspects of one or more embodiments of the invention. However, it should be understood that the invention is not limited to the precise arrangements and means shown in the drawings, in which:

[0015] Figure 1 These are illustrations of example reference frames and example predicted frames for various resolution levels;

[0016] Figure 2 It is a diagram depicting the prediction process of the example reference frame, the example rescaled reference frame, and the example subsequent block;

[0017] Figure 3 Four different frames with different resolutions are shown;

[0018] Figure 4 Two example buffers are shown, one for a full-resolution frame (top) and one for a scaled-down frame (bottom), where the scaled-down frame and the full-resolution frame are stored in the same location within their respective buffers;

[0019] Figure 5 This is a process flowchart illustrating example processes according to some implementations of the current topic;

[0020] Figure 6This is a system block diagram illustrating an example decoder capable of decoding bitstreams according to some implementations of the present topic;

[0021] Figure 7 This is a flowchart illustrating an example process for encoding video according to some implementations of the current topic;

[0022] Figure 8 This is a system block diagram illustrating an example video encoder according to some implementations of the current topic; and

[0023] Figure 9 It is a block diagram of a computational system that can be used to implement any one or more methods disclosed herein and any one or more parts thereof.

[0024] The accompanying drawings are not necessarily drawn to scale and may use dashed lines, schematic diagrams, and partial views for illustration. In some cases, details that are unnecessary for understanding the embodiments or that make other details difficult to understand may have been omitted. In the drawings, the same reference numerals refer to the same elements. Detailed Implementation

[0025] In many of today's state-of-the-art encoders, resolution is managed by re-encoding and retransmitting entire portions of video known as Groups of Pictures (GOPs). This requires sending internal frames (I-frames), which can result in additional cost because these frames are responsible for most of the bits in the GOP.

[0026] The embodiments described in this disclosure relate to Adaptive Resolution Management (ARM), a technique that provides additional flexibility to video encoders / decoders, thereby allowing bitrate savings in a variety of use cases. Typically, ARM involves performing prediction using a reference frame with a different resolution than the current frame. In current coding standards, the reference frame has the same resolution as the predicted frame. In ARM, the resolution of the reference frame may be less than or greater than the resolution of the frame being predicted. This approach can be used to reduce video resolution, thereby reducing the bitrate, or to increase video resolution, thereby improving the display characteristics of video playback.

[0027] For the purposes of this disclosure, ARM may be alternatively or equivalently referred to as Reference Image Resampling (RPR); RPR and ARM are used interchangeably.

[0028] Some implementations of the current topic may include using ARM for any number of frames at any location within a GOP, thereby eliminating the need for I-frame re-encoding.

[0029] Figure 1This is a diagram illustrating reference and predicted frames at various resolution levels. Frame 1 is smaller than the reference frame (lower resolution), frame 2 is the same size as the reference frame (same resolution), and frame 3 is larger than the reference frame (higher resolution). As used in this disclosure, "resolution" refers to the number of pixels in an image, frame, subframe, and / or other display area or portion thereof used in video playback, compression, etc., with a higher number of pixels corresponding to a higher resolution and a lower number of pixels corresponding to a lower resolution. Resolution can be measured by area, for example, but not limited to, by using one or more length dimensions to define the area in pixels. For example, a circular subframe or other area may have a resolution defined by its radius. Alternatively or additionally, resolution may be defined by the total number of pixels.

[0030] As an example, continue to refer to Figure 1 The reference frame and / or subframe have a geometry whose area can be defined entirely by two length parameters, such as, but not limited to, triangles, parallelograms, and / or rectangles. The reference frame and / or subframe can have a resolution W×H, where W and H can indicate the number of pixels describing the width (or bottom) and height dimensions of the reference frame and / or subframe, respectively. Each predicted frame can also have a resolution that can be determined similarly to the resolution of the reference frame; for example, frame 1 can have a smaller resolution WS×HS, frame 2 can have the same resolution W×H as the reference frame, and frame 3 can have a larger resolution WL×HL. The width and height of the smaller and larger frames can be obtained by multiplying the reference width and height by an arbitrary rescaling constant (Rc), also known as a scaling factor and / or constant. In the case of smaller frames, Rc can have values ​​between 0 and 1. In the case of larger frames, Rc can have values ​​greater than 1; for example, Rc can have values ​​between 1 and 4. Other values ​​are also possible. The rescaling constant for one resolution dimension may differ from that for another resolution dimension; for example, the rescaling constant Rch can be used to rescale the height, while another rescaling constant Rcw can be used to rescale the width.

[0031] Still referencing Figure 1ARM can be implemented as a mode. If ARM mode is activated at some point during decoding, the decoder may have already received a reference frame with a resolution of W×H and can rescale the predicted frame using a rescaling constant. In some implementations, the encoder can identify to the decoder which rescaling constant to use. This identification can be performed in the Sequence Parameter Set (SPS) corresponding to the GOP containing the current image and / or in the Image Parameter Set (PPS) corresponding to the current image. For example, but not limited to, the encoder can use fields such as pps_pic_width_in_luma_samples, pps_pic_height_in_luma_samples, pps_scaling_win_left_offset, pps_scaling_win_right_offset, pps_scaling_win_top_offset, pps_scaling_win_bottom_offset, and / or sps_num_subpics_minus1 to identify the rescaled parameters.

[0032] Further reference Figure 1 The W and H parameters mentioned above can, but are not limited to, be represented by the variables CurrPicScalWinWidthL and CurrPicScalWinHeightL, respectively; these variables can be derived from the identified parameters as described above using one or more mathematical relationships representing the relationship between the parameters and the variables. For example, but not limited to, CurrPicScalWinWidthL can be derived from the following equation:

[0033] CurrPicScalWinWidthL=pps_pic_width_in_luma_samples–

[0034] SubWidthC*(pps_scaling_win_right_offset+pps_scaling_win_left_offset)

[0035] As another non-restrictive example, CurrPicScalWinHeightL can be derived from the following equation:

[0036] CurrPicScalWinWidthL=pps_pic_width_in_luma_samples–

[0037] SubWidthC*(pps_scaling_win_right_offset+pps_scaling_win_left_offset)

[0038] Those skilled in the art will understand, upon reading the full contents of this disclosure, that various alternative calculations can be used to derive the aforementioned variables. The encoder may, alternatively or additionally, directly, for example, but not limited to, identifying one or more such variables Rc, Rch, and / or Rcw in the PPS and / or SPS.

[0039] Alternatively or additionally, refer to Figure 1 The rescaling constants and / or a set of rescaling constants described above can be identified in the bitstream using references to the indices of one or more stored scaling constants and / or frames and / or blocks, which are identified using one or more previously identified and / or used scaling constants. The references to the indices of the stored scaling constants can be explicitly identified and / or determined from one or more additional parameters identified in the bitstream. For example, but not limited to, the decoder can identify a reference frame and / or image group containing the current frame; where rescaling constants have been previously identified and / or used in such image groups, the identified reference frame can be applied to the current frame and / or current image group, etc., and the decoder can identify the rescaling constant to use as the rescaling constant for the current frame.

[0040] In some implementations, reference continues. Figure 1 ARM operations can be performed at the block level of the encoded frame. For example, a reference frame can be rescaled first, followed by prediction, such as... Figure 2 As shown. Figure 2 This is an illustration depicting a reference frame, a rescaled reference frame, and the subsequent block prediction process. The block prediction process can be performed on the scaled reference frame (with scaled resolution) instead of the original reference frame. As described above, rescaling the reference frame can include rescaling according to any parameters identified by the encoder; for example, but not limited to, if a reference frame to be used with the current frame is identified, such as by referencing an index value associated with the reference frame, the identified reference frame can be rescaled according to any of the rescaling methods described above before prediction. The rescaled reference frame can be stored in memory and / or buffers, which can include, but are not limited to, buffers that identify the frames contained therein by index, according to which frame retrieval can be performed; the buffers can include a decoded picture buffer (DCB) and / or one or more additional buffers implemented by the decoder. The prediction process can include, for example, inter-picture prediction including motion compensation.

[0041] Some implementations of block-based ARM allow for the flexibility to apply optimal filters to each block, rather than applying the same filters to the entire frame. In some implementations, a skip-ARM mode is possible, allowing some blocks (e.g., based on pixel uniformity and bitrate cost) to be in skip-ARM mode (so that rescaling does not change the bitrate). The skip-ARM mode can be identified in the bitstream; for example, but not limited to, it can be identified in the PPS parameter. Alternatively or additionally, the decoder can determine whether the skip-ARM mode is active based on one or more parameters set by the decoder and / or identified in the bitstream. Spatial filters used in block-based ARM can include, but are not limited to, bicubic spatial filters applying bicubic interpolation, bilinear spatial filters applying bilinear interpolation, resampled Lanczos filters using Lanczos filtering and / or sinc filters, sinc function interpolation and / or signal reconstruction techniques, etc.; various filters consistent with this disclosure will be understood by those skilled in the art upon reading the full contents of this disclosure. As a non-limiting example, the interpolation filter may include any filter as described above, a low-pass filter, which may be used through, but is not limited to, an upsampling process, wherein pixels between pixels of a block and / or frame before scaling may be initialized to zero and then filled with the output of the low-pass filter. Alternatively or additionally, any luminance sample interpolation filtering process may be used. Luminance sample interpolation may include calculating an interpolated value at a half-sample interpolation filter index that falls between two consecutive sample values ​​in the unscaled sample array. The calculation of the interpolated value may be performed by retrieving coefficients and / or weights from a lookup table, but is not limited to this; the selection of the lookup table may be performed as a function of the motion model and / or scaling factor of the coding unit, for example, using the scaling constant as described above. The calculation may include, but is not limited to, performing a weighted summation of adjacent pixel values, wherein the weights are retrieved from a lookup table. Alternatively or additionally, the calculated value may be shifted; for example, but not limited to, the value may be shifted to Min(4, bit depth - 8), 6, Max(2, 14 - bit depth), etc. Various alternative or additional implementations that may be used for the interpolation filter will be apparent to those skilled in the art after reading the full contents of this disclosure.

[0042] In some implementations, reference is still made to Figure 2 The ARM can be used to rescale one or more frames at the encoder, which can then encode the one or more rescaled frames generated by the rescaling. At the decoder, the rescaled frames can be decoded and scaled back before the resulting frames are displayed at full resolution.

[0043] Now for reference Figure 3 When encoding rescaled frames, all frames that can be used as reference frames can be rescaled before motion estimation is performed. For example, and as... Figure 3 For illustrative purposes, frame i+1 can be encoded as a rescaled frame (reduced resolution), and frame i can be rescaled if frame i is available as a reference for decoding frame i. The encoder can perform the following operation: the encoder can first encode frame i at full resolution. The encoder can then rescale frame i+1 to the recall frame (i+1). r And rescale the reconstructed frame i in the form of reconstruction, which can be represented here as i * In this disclosure, it is represented as i *r In other words, the i used in this example *r This indicates that the corresponding i * The rescaled version of the rescaled reconstructed frame. The encoder can then use i *r Use it as a reference to encode (i+1) r .

[0044] Similarly, still refer to Figure 3 You can use the reconstructed frame (i+1). r* Use it as a reference to encode frame i+2.

[0045] Further reference Figure 3 Frame i+3 can be encoded at full resolution; therefore, frame i+3 can be decoded using one or more reference frames that have already been rescaled to full resolution. If referencing two previous frames is permitted, for example, for illustrative purposes, the two previous frames are frame i+1 and frame 1+2, then these two previous frames can be rescaled to full resolution before and / or after reconstruction.

[0046] Still referencing Figure 3 The rescaled frames can be complex. In one embodiment, the decoder and / or encoder can minimize the number of rescales required. As a non-limiting example, the decoder and / or encoder can reduce rescale complexity by maintaining frame buffers with both full-resolution and rescaled-resolution frames.

[0047] Continue to refer to Figure 3 In some embodiments, at the decoder, any or all reference frames can be available at the rescaled resolution. Any or all rescaled frames can be scaled back to full resolution for display and / or used as reference frames.

[0048] Still referencing Figure 3The decoder can maintain two decoded image buffers: one for full-resolution frames and one for rescaled frames. The rescaled frames and full-resolution frames can be stored in the decoded image buffer for the rescaled frames and the decoded image buffer for the full-resolution frames, respectively. When separate buffers are used for full-resolution and rescaled-resolution frames, the full-resolution and corresponding rescaled frames can be stored in the same buffer location, and / or in buffer locations indicated by the same index, where the buffers are implemented as indexed data structures, such as array-type structures. For example, a motion vector index can point to the same location in two or more different buffers, and the decoder and / or encoder can select the buffer retrieved from the index location based on whether the reference frame to be used has been rescaled.

[0049] Continue to refer to Figure 3 The corresponding full-resolution and rescaled resolution frames can be stored in the same location in different buffers and referenced using the same image index.

[0050] Figure 4Two non-limiting examples of buffers are shown, one for full-resolution frames (top) and one for scaled frames (bottom), where the scaled frames and full-resolution frames can be stored in the same location within their respective buffers as described above. In the non-limiting examples, a sublayer decoded image buffer (DPB) can be used to hold rescaled images, frames, subframes, etc.; in this case, the sublayer DBP can differ from the main DBP in one or more parameters and / or content elements, and the main DBP can contain other and / or unresampled and / or unscaled reference images or other image frames and / or subframes. The ability to resample and / or rescale the reference image, and / or the parameters used for such resampling and / or rescaling, can be identified in the bitstream, for example, in the SPS header. For example, but not limited to, the SPS parameter, which can be represented as sps_ref_pic_resampling_enabled_flag, can specify that reference image resampling is enabled when it is equal to 1; the current image referencing the SPS may have a slice that references a reference image in the active entry of the RPL, which has one or more of the following seven parameters that are different from the parameters of the current image: 1) pps_pic_width_in_luma_samples, 2) pps_pic_height_in_luma_samples, 3) pps_scaling_win_left_offset, 4) pps_scaling_win_right_offset, 5) pps_scaling_win_top_offset, 6) pps_scaling_win_bottom_offset, and 7) sps_num_subpics_minus1. A `sps_ref_pic_resampling_enabled_flag` equal to 0 can indicate that reference image resampling is disabled and / or the current image referencing the SPS does not have a block of the reference image in the active entry of the reference RPL, which has one or more of the aforementioned seven parameters that are different from the parameters of the current image. In one embodiment, when `sps_ref_pic_resampling_enabled_flag` equals 1, for the current image, a reference image having one or more of the aforementioned seven parameters that are different from the current image can belong to the same layer as the layer containing the current image or a different layer.Parameters such as vps_sublayer_dpb_params_present_flag can be used to control the existence of parameters that govern the behavior of DBP. For example, dpb_max_dec_pic_buffering_minus1[j] specifies the maximum required size of DBC, dpb_max_num_reorder_pics[j] specifies the maximum allowed number of images in the output layer set (OLS), which can be decoded before any image in the OLS and followed in the output order when Htid of the highest time sublayer to be decoded is equal to i, and / or dpb_max_latency_increase_plus1[j] can be used to specify the maximum number of images in the OLS, which can be decoded before any image in the OLS and followed in the decoding order when Htid is equal to i. Figure 5 This is a process flowchart illustrating an exemplary embodiment of an adaptive resolution management process 500, which can enable additional activity in the video encoder and / or decoder, thereby allowing bitrate savings in a variety of use cases.

[0051] At step 505, still refer to Figure 5 The decoder receives a bitstream. The current frame, including the current block, may be included in the bitstream received by the decoder. The bitstream may include, for example, data found in the bitstream that serves as input to the decoder when data compression is used. The bitstream may include information necessary for decoding the video. Receiving the bitstream may include extracting and / or parsing blocks from the bitstream and associated identification information. In some implementations, the current block may include a coding tree unit (CTU), a coding unit (CU), or a prediction unit (PU).

[0052] At step 510, still refer to Figure 5 The first frame is decoded using a bitstream.

[0053] Continue referencing at point 515. Figure 5 The scaled first frame can be determined using the first frame and a scaling constant. The scaling constant can be identified in the bitstream.

[0054] Still referencing Figure 5 In step 520, the first frame can be stored at the first index position in the first image buffer.

[0055] In step 525, further refer to Figure 5The scaled first frame can be stored at a first index position in the second image buffer. In some implementations, the second image buffer can store frames with a higher resolution than the first image buffer; alternatively, the second image buffer can store frames with a lower resolution than the first image buffer. The scaled first frame can then be displayed.

[0056] In some implementations, reference is still made to Figure 5 The second frame or a portion thereof can be decoded in adaptive resolution management mode using the scaled first frame as a reference frame. The first frame or a portion thereof can be decoded according to adaptive resolution management mode. The scaling constant can include a vertical scaling component and / or a horizontal scaling component. In one embodiment, the parameter pps_scaling_window_explicit_signalling_flag, when set to 1, specifies the presence of a scaling window offset parameter in PPS; pps_scaling_window_explicit_signalling_flag equal to 0 specifies the absence of a scaling window offset parameter in PPS. The value of pps_scaling_window_explicit_signalling_flag can be 0 when sps_ref_pic_resampling_enabled_flag equals 0. In one embodiment, pps_scaling_win_left_offset, pps_scaling_win_right_offset, pps_scaling_win_top_offset, and pps_scaling_win_bottom_offset can specify offsets applied to the image size for scaling ratio calculation. If they do not exist, it can be inferred that the values ​​of pps_scaling_win_left_offset, pps_scaling_win_right_offset, pps_scaling_win_top_offset, and pps_scaling_win_bottom_offset are equal to pps_conf_win_left_offset, pps_conf_win_right_offset, pps_conf_win_top_offset, and pps_conf_win_bottom_offset, respectively.

[0057] Figure 6This is a system block diagram illustrating an example decoder 600 capable of implementing frame buffering in adaptive resolution management as described in this disclosure. Decoder 600 may include an entropy decoding processor 604, an inverse quantization and inverse transform processor 608, a deblocking filter 612, a frame buffer 616, a motion compensation processor 620, and / or an intra-frame prediction processor 624.

[0058] During operation, still refer to Figure 6 The bitstream 628 can be received by the decoder 600 and input to the entropy decoding processor 604, which can entropy decode a portion of the bitstream into quantization coefficients. The quantization coefficients can be provided to the inverse quantization and inverse transform processor 608, which can perform inverse quantization and inverse transform to create a residual signal. This residual signal can be added to the output of the motion compensation processor 620 or the intra-frame prediction processor 624, depending on the processing mode. The outputs of the motion compensation processor 620 and the intra-frame prediction processor 624 can include block prediction based on previous decoded blocks. The sum of the prediction and the residual can be processed by the deblocking filter 612 and stored in the frame buffer 616.

[0059] Figure 7 This is a flowchart illustrating an exemplary process 700 for encoding video using adaptive resolution management. Process 700 can provide additional flexibility for the video encoder and / or decoder, thereby allowing bitrate savings in a variety of use cases. At step 705, the video frame may undergo initial block segmentation, for example, using a tree-structured macroblock segmentation scheme, which may include dividing the image frame into CTUs and CUs.

[0060] At step 710, block-based adaptive resolution management can be performed, including resolution scaling of frames or portions thereof.

[0061] At step 715, the block can be encoded and included in the bitstream. For example, encoding may include using inter-frame prediction and intra-frame prediction modes.

[0062] Figure 8 This is a system block diagram illustrating an example video encoder 800 capable of implementing frame buffering in adaptive resolution management as described in this disclosure. The example video encoder 800 can receive input video 804, which can be initially segmented or partitioned according to a processing scheme such as a tree-structured macroblock segmentation scheme (e.g., quadtree plus binary tree). Examples of tree-structured macroblock segmentation schemes may include dividing image frames into large blocks called coding tree units (CTUs). In some implementations, each CTU may be further segmented once or multiple times into multiple sub-blocks called coding units (CUs). The result of such segmentation may include a set of sub-blocks called prediction units (PUs). Transform units (TUs) may also be used.

[0063] Still referencing Figure 8 An exemplary video encoder 800 may include an intra-frame prediction processor 808, a motion estimation / compensation processor 812 (also referred to as an inter-frame prediction processor) capable of constructing a motion vector candidate list, including adding global motion vector candidates to the motion vector candidate list, a transform / quantization processor 816, an inverse quantization / inverse transform processor 820, a loop filter 824, a decoded image buffer 828, and / or an entropy coding processor 832. Bitstream parameters may be input to the entropy coding processor 832 to be included in the output bitstream 836.

[0064] Continue to refer to the instructions during operation. Figure 8 For each block of a frame in the input video 804, it can be determined whether the block will be processed via intra-frame prediction or motion estimation / compensation. The block can be provided to either the intra-frame prediction processor 808 or the motion estimation / compensation processor 812. If the block is to be processed via intra-frame prediction, the intra-frame prediction processor 808 can perform processing to output a predictor. If the block is to be processed via motion estimation / compensation, the motion estimation / compensation processor 812 can perform processing, if applicable, including constructing a list of motion vector candidates, including adding global motion vector candidates to the list.

[0065] Further reference Figure 8 The residual can be formed by subtracting the prediction factor from the input video. The residual can be received by a transform / quantization processor 816, which can perform transform processing (e.g., Discrete Cosine Transform (DCT)) to produce coefficients that can be quantized. The quantization coefficients and any associated identification information can be provided to an entropy coding processor 832 for entropy coding and included in the output bitstream 836. The entropy coding processor 832 can support encoding identification information related to the current block. Furthermore, the quantization coefficients can be provided to an inverse quantization / inverse transform processor 820, which can reproduce pixels that can be combined with the prediction factor and processed by a loop filter 824. The output of the loop filter 824 can be stored in a decoded image buffer 828 for use by a motion estimation / compensation processor 812, which is capable of constructing a motion vector candidate list, including adding global motion vector candidates to the motion vector candidate list.

[0066] Continue to refer to Figure 8Although some variations have been described in detail above, other modifications or additions are possible. For example, in some implementations, the current block may include any symmetrical block (8×8, 16×16, 32×32, 64×64, 128×128, etc.) and any asymmetrical block (8×4, 16×8, etc.).

[0067] In some implementations, reference is still made to Figure 8 Quadtree plus binary decision tree (QTBT) can be implemented. For QTBT, at the encoding tree unit level, the splitting parameters of QTBT can be dynamically derived to adapt to local features without any transmission overhead. Subsequently, at the encoding unit level, the joint classifier decision tree structure can eliminate unnecessary iterations and control the risk of incorrect predictions. In some implementations, the LTR frame block update mode can be used as an additional option available at each leaf node of the QTBT.

[0068] In some implementations, reference is still made to Figure 8 Other syntax elements can be identified at different levels of the bitstream. For example, an enable flag can be included for the entire sequence by including an encoded enable flag in the Sequence Parameter Set (SPS). Furthermore, CTU flags can be encoded at the Code Tree Unit (CTU) level.

[0069] Some embodiments may include a non-transitory computer program product (i.e., a physically implemented computer program product) containing instructions that, when executed by one or more data processors of one or more computing systems, cause at least one data processor to perform the operations described herein.

[0070] In some embodiments disclosed herein, the decoder includes circuitry configured to receive a bitstream, decode a first frame using the bitstream, determine a scaled first frame using the first frame and a scaling constant, store the first frame at a first index position in a first image buffer, and store the scaled first frame at a first index position in a second image buffer.

[0071] In some embodiments, the second image buffer may store frames with higher resolution than the first image buffer. In some embodiments, the second image buffer may store frames with lower resolution than the first image buffer. The decoder may be further configured to display the scaled first frame. The decoder may be configured to use the scaled first frame as a reference frame to decode at least a portion of the second frame in an adaptive resolution management mode. The decoder may be configured to decode at least a portion or a portion of the first frame according to the adaptive resolution management mode. The adaptive resolution management mode may be identified in a picture parameter set (PPS). The adaptive resolution management mode may be identified in a sequence parameter set (SPS). The scaling constant may include a vertical scaling component and a horizontal scaling component. The decoder may include: an entropy decoding processor configured to receive a bitstream and decode the bitstream into quantization coefficients; an inverse quantization and inverse transform processor configured to process the quantization coefficients, the processing of the quantization coefficients including performing an inverse discrete cosine transform; a deblocking filter; a frame buffer; and an intra-frame prediction processor.

[0072] In some embodiments disclosed herein, the method may include: receiving a bitstream, decoding a first frame using the bitstream, determining a scaled first frame using the first frame and a scaling constant, storing the first frame at a first index position in a first image buffer, and storing the scaled first frame at the first index position in a second image buffer.

[0073] In some embodiments, the second image buffer may store frames with higher resolution than the first image buffer. The second image buffer may store frames with lower resolution than the first image buffer. Embodiments may include displaying the scaled first frame. Embodiments may include decoding at least a portion of a second frame in an adaptive resolution management mode using the scaled first frame as a reference frame. Embodiments may include decoding at least a portion of the first frame according to an adaptive resolution management mode. The adaptive resolution management mode may be identified in a picture parameter set (PPS). The adaptive resolution management mode may be identified in a sequence parameter set (SPS). The scaling constant may include a vertical scaling component and a horizontal scaling component. At least one of the receiving, decoding, determining, and storing may be performed by a decoder, which may include: an entropy decoding processor configured to receive a bitstream and decode the bitstream into quantization coefficients; an inverse quantization and inverse transform processor configured to process the quantization coefficients, the processing of the quantization coefficients including performing an inverse discrete cosine transform; a deblocking filter; a frame buffer; and an intra-frame prediction processor.

[0074] It should be noted that any one or more aspects and embodiments described herein can be readily implemented using digital electronic circuits, integrated circuits, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), computer hardware, firmware, software, and / or combinations thereof, as implemented and / or carried out in one or more machines programmed according to the teachings of this specification (e.g., one or more computing devices used as user computing devices for electronic documents, one or more server devices such as document servers, etc.), as will be apparent to those skilled in the art of computers. These various aspects or features may include implementations in one or more computer programs and / or software executable and / or interpretable on a programmable system, said programmable system including at least one programmable processor, which may be dedicated or general-purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to send data and instructions to the storage system, at least one input device, and at least one output device. It will be apparent to those skilled in the art of software that a skilled programmer can readily prepare appropriate software code based on the teachings of this disclosure. The aspects and implementation methods of employing software and / or software modules discussed above may also include appropriate hardware for assisting in the implementation of machine-executable instructions for the software and / or software modules.

[0075] Such software can be a computer program product employing a machine-readable storage medium. A machine-readable storage medium can be any medium capable of storing and / or encoding a sequence of instructions executable by a machine (e.g., a computing device) and causing the machine to perform any of the methods and / or embodiments described herein. Examples of machine-readable storage media include, but are not limited to, magnetic disks, optical disks (e.g., CDs, CD-Rs, DVDs, DVD-Rs, etc.), magneto-optical disks, read-only memory "ROM" devices, random access memory "RAM" devices, magnetic cards, optical cards, solid-state storage devices, EPROMs, EEPROMs, programmable logic devices (PLDs), and / or any combination thereof. The use of machine-readable media herein is intended to include both single media and collections of physically separate media, such as collections of optical disks, or one or more hard disk drives combined with computer memory. Machine-readable storage media as used herein do not include temporary forms of signal transmission.

[0076] Such software may also include information (e.g., data) carried as data signals on a data carrier such as a carrier wave. For example, machine-executable information may be included as data-bearing signals contained in a data carrier, wherein the signals encode a sequence of instructions or portions thereof executed by a machine (e.g., a computing device), and any relevant information (e.g., data structures and data) that causes the machine to perform any of the methods and / or embodiments described herein.

[0077] Examples of computing devices include, but are not limited to, e-book readers, computer workstations, terminal computers, server computers, handheld devices (e.g., tablets, smartphones, etc.), network devices, network routers, network switches, bridges, any machine capable of executing a sequence of instructions specifying the action it is to take, and any combination of the foregoing. In one example, a computing device may include and / or be included in a kiosk.

[0078] Figure 9 The illustration shows an embodiment of a computing device in an exemplary form of a computer system 900, wherein a set of instructions can be executed to cause a control system to perform any one or more aspects and / or methods of this disclosure. It is also contemplated that multiple computing devices can be utilized to implement specially configured instruction sets for causing one or more of the devices to perform any one or more aspects and / or methods of this disclosure. The computer system 900 includes a processor 904 and a memory 908, which communicate with each other and with other components via a bus 912. The bus 912 can include any of a variety of bus structures using any of a variety of bus architectures, including but not limited to a memory bus, a memory controller, a peripheral bus, a local bus, and any combination thereof.

[0079] Memory 908 may include various components (e.g., machine-readable media), including but not limited to random access memory components, read-only components, and any combination thereof. In one example, a basic input / output system 916 (BIOS) may be stored in memory 908, the BIOS including basic routines such as those that facilitate the transfer of information between elements within computer system 900 during startup. Memory 908 may also include (e.g., stored on one or more machine-readable media) instructions (e.g., software) 920 embodying any one or more aspects and / or methods of this disclosure. In another example, memory 908 may further include any number of program modules, including but not limited to an operating system, one or more application programs, other program modules, program data, and any combination thereof.

[0080] Computer system 900 may also include storage device 924. Examples of storage devices (e.g., storage device 924) include, but are not limited to, hard disk drives, disk drives, optical disc drives combined with optical media, solid-state storage devices, and any combination thereof. Storage device 924 may be connected to bus 912 via a suitable interface (not shown). Exemplary interfaces include, but are not limited to, SCSI, Advanced Technology Attachment (ATA), Serial ATA, Universal Serial Bus (USB), IEEE 1394 (FireWire), and any combination thereof. In one example, storage device 924 (or one or more components thereof) may be removably connected to computer system 900 (e.g., via an external port connector (not shown)). In particular, storage device 924 and associated machine-readable medium 928 may provide non-volatile and / or volatile storage of machine-readable instructions, data structures, program modules, and / or other data for computer system 900. In one example, software 920 may be stored wholly or partially within machine-readable medium 928. In one example, software 920 may be stored wholly or partially within processor 904.

[0081] Computer system 900 may also include input device 932. In one example, a user of computer system 900 may input commands and / or other information into computer system 900 via input device 932. Examples of input device 932 include, but are not limited to: alphanumeric input devices (e.g., keyboard), pointing devices, joysticks, game controllers, audio input devices (e.g., microphones, voice response systems, etc.), cursor control devices (e.g., mice), touchpads, optical scanners, video capture devices (e.g., still cameras, camcorders), touchscreens, and any combination thereof. Input device 932 may be connected to bus 912 via any of a variety of interfaces (not shown), including but not limited to serial interfaces, parallel interfaces, game ports, USB interfaces, FireWire interfaces, direct interfaces to bus 912, and any combination thereof. Input device 932 may include a touchscreen interface, which may be part of or separate from display 936, as will be discussed further below. Input device 932 may be used as a user selection device for selecting one or more graphical representations in a graphical interface as described above.

[0082] Users can also input commands and / or other information to computer system 900 via storage device 924 (e.g., removable disk drive, flash drive, etc.) and / or network interface device 940. Network interface devices such as network interface device 940 can be used to connect computer system 900 to one or more of various networks such as network 944, and one or more remote devices 948 connected to network 944. Examples of network interface devices include, but are not limited to, network interface cards (e.g., mobile network interface cards, LAN cards), modems, and any combination thereof. Examples of networks include, but are not limited to, wide area networks (e.g., the Internet, corporate networks), local area networks (e.g., networks associated with offices, buildings, campuses, or other relatively small geographical areas), telephone networks, data networks associated with telephone / voice providers (e.g., mobile communication provider data and / or voice networks), direct connections between two computing devices, and any combination thereof. Networks such as network 944 can employ wired and / or wireless communication modes. Typically, any network topology can be used. Information (e.g., data, software 920, etc.) can be transmitted to and / or from computer system 900 via network interface device 940.

[0083] Computer system 900 may further include a video display adapter 952 for transmitting displayable images to a display device such as display device 936. Examples of display devices include, but are not limited to, liquid crystal displays (LCDs), cathode ray tube (CRTs), plasma displays, light-emitting diode (LED) displays, and any combination thereof. Display adapter 952 and display device 936 may be used in conjunction with processor 904 to provide graphical representations of various aspects of this disclosure. In addition to display devices, computer system 900 may include one or more other peripheral output devices, including, but not limited to, audio speakers, printers, and any combination thereof. Such peripheral output devices may be connected to bus 912 via peripheral interface 956. Examples of peripheral interfaces include, but are not limited to, serial ports, USB connections, FireWire connections, parallel connections, and any combination thereof.

[0084] Exemplary embodiments of the present invention have been described in detail above. Various modifications and additions may be made without departing from the spirit and scope of the invention. To provide multiple combinations of features in relevant new embodiments, features of each of the various embodiments described above may be appropriately combined with features of other described embodiments. Furthermore, while many individual embodiments have been described above, the content described herein is merely illustrative of the application of the principles of the invention. Moreover, although specific methods herein may be shown and / or described as being performed in a particular order, the order is highly variable in common techniques for implementing the embodiments disclosed herein. Therefore, this description is intended only as an example and not as limiting the scope of the invention.

[0085] In the foregoing description and claims, phrases such as "at least one" or "one or more" may be followed by a list of connected elements or features. The term "and / or" may also appear in a list of two or more elements or features. Unless implicitly or explicitly contradicting the context in which the phrase is used, such phrases are intended to mean any element or feature listed individually, or a combination of any listed element or feature with any other listed element or feature. For example, the phrases "at least one of A and B," "one or more of A and B," and "A and / or B" are intended to mean "A alone, B alone, or both A and B," respectively. A similar interpretation applies to lists containing three or more items. For example, the phrases "at least one of A, B, and C," "one or more of A, B, and C," and "A, B, and / or C" are intended to mean "A alone, B alone, C alone, both A and B, both A and C, both B and C, or A and B and C," respectively. Furthermore, the use of the term "based on" in the foregoing and claims is intended to mean "at least partially based on," thus allowing for the inclusion of uncited features or elements.

[0086] The subject matter described herein can be embodied in systems, apparatus, methods, and / or articles according to desired configurations. The embodiments set forth in the foregoing description do not represent all embodiments consistent with the subject matter described herein. Rather, they are merely examples of aspects consistent with the described subject matter. Although some variations have been described in detail above, other modifications or additions are possible. In particular, further features and / or variations may be provided in addition to the features and / or variations set forth herein. For example, the above embodiments may be applicable to various combinations and sub-combinations of the disclosed features and / or combinations and sub-combinations of several other features disclosed above. Furthermore, the logical flows described in the drawings and / or described herein do not necessarily require a specific order or sequence to achieve the desired results. Other embodiments are within the scope of the claims.

Claims

1. A decoder for decoding an encoded bitstream, configured to: Execute a block-based mode, in which a reference image of one resolution is used to form a predictor for an image of another resolution; Receive an encoded bitstream encoded by an encoder, the encoded bitstream including a first encoded image having a first spatial resolution, a second encoded image having a second spatial resolution, and a third encoded image having a third spatial resolution; Decode the first encoded image and store the first image as a first reference image; Using the mode, a first prediction factor is used to form the first block of the second coded image from the first reference image, by changing the scaling factor with a first spatial resolution signaled by the encoder to the decoder and a first filter determined for the first block; The first predictor is used to decode the second encoded image, and the second image is stored as a second reference image; Using the aforementioned pattern, a second prediction factor is used to form the second block of the third coded image from the second reference image, employing a scaling factor that is signaled to the decoder by the encoder and a second filter determined for the second block. The third encoded image is decoded using the second predictor.

2. The decoder of claim 1, wherein the first resolution and the third resolution are the same, and wherein the second scaling factor and the second filter, as well as the first scaling factor and the first filter, inversely affect the spatial resolution.

3. The decoder according to claim 1, wherein the first reference image and the second reference image are stored in the decoded image buffer at a position determined by an index.

4. The decoder of claim 3, wherein the index is transmitted as a signal in the encoded bitstream.

5. A video encoder having circuitry configured to generate an encoded bitstream, the encoded bitstream comprising a first encoded image having a first spatial resolution, a second encoded image having a second spatial resolution, and a third encoded image having a third spatial resolution, the bitstream being decodeable by a compatible decoder, wherein the decoder is configured to: Receive the encoded bit stream; Execute a block-based mode, in which a reference image of one resolution is used to form a predictor for an image of another resolution; Decode the first encoded image and store the first image as a first reference image; Using the mode, a first prediction factor is used to form the first block of the second coded image from the first reference image, by changing the scaling factor with a first spatial resolution signaled by the encoder to the decoder and a first filter determined for the first block; The first predictor is used to decode the second encoded image, and the second image is stored as a second reference image; Using the aforementioned mode, a second prediction value for the second block of the third coded image is formed from the second reference image by using a scaling factor changed by a second spatial resolution signaled by the encoder to the decoder and a second filter determined for the second block. The third encoded image is decoded using the second predictor.

6. The video encoder of claim 5, wherein the first resolution and the third resolution are the same, and wherein the second scaling factor and the second filter, as well as the first scaling factor and the first filter, inversely affect the spatial resolution.

7. The video encoder of claim 5, wherein the first reference image and the second reference image are stored in the decoded image buffer at a position determined by an index.

8. The video encoder of claim 7, wherein the index is transmitted as a signal in the encoded bitstream.

Images