Implicit signaling of adaptive resolution management based on frame type
Adaptive Resolution Management (ARM) addresses the inefficiency of current video codecs by allowing frames to be predicted with reference frames of varying resolutions, reducing bit rate and enhancing compression efficiency and display quality.
Patent Information
- Application Number
- JP2024167439
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2019-08-06
- Filing Date
- 2024-09-26
- Publication Date
- 2024-12-05
AI Technical Summary
Current video codecs require recoding and transmitting entire groups of pictures (GOPs) to manage resolution, leading to increased bit costs due to the use of intra-frames, which are resource-intensive.
Adaptive Resolution Management (ARM) allows for predicting frames using reference frames of different resolutions, enabling bitrate savings by eliminating the need for recoding intra-frames and allowing flexibility in video encoding and decoding processes.
ARM reduces bit rate by downsampling or upsampling video resolution, improving compression efficiency and display characteristics while avoiding prediction mismatches and drift issues through implicit and explicit signaling.
Smart Images

Figure 2024169708000001 
Figure 2024169708000002 
Figure 2024169708000003
Abstract
Description
[Technical field]
[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application, filed on August 6, 2019, is incorporated herein by reference in its entirety, and is directed to a method for detecting an implict signaling event (IMPLICIT SIGNALING OF ADAPTIVE This application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 883,484, entitled "RESOLUTION MANAGEMENT BASED ON FRAME TYPE."
[0002] The present invention relates generally to the field of video compression. In particular, the present invention is directed to implicit signaling of adaptive resolution management based on frame type. [Background technology]
[0003] A video codec may include electronic circuitry or software that compresses or decompresses digital video. It may convert uncompressed video into a compressed format, or vice versa. In the context of video compression, a device that compresses video (and / or performs some function thereof) may typically be called an encoder, and a device that decompresses video (and / or performs some function thereof) may be called a decoder.
[0004] The format of the compressed data may conform to standard video compression specifications. The compression may be lossy in that the compressed video may lack certain information present in the original video. Consequences of this may include that the decompressed video may have lower quality than the original uncompressed video because there may be insufficient information to accurately reconstruct the original video.
[0005] There may be a complex relationship between video quality, the amount of data used to represent the video (e.g., as determined by bit rate), the complexity of the encoding and decoding algorithms, sensitivity to data loss and errors, ease of editing, random access, end-to-end delay (e.g., latency), and the like.
[0006] Motion compensation may include an approach for predicting a video frame or a portion thereof given a reference frame, such as a previous and / or future frame, by considering the motion of the camera and / or objects in the video. This may be employed in encoding and decoding video data for video compression, for example, using the Advanced Video Coding (AVC) standard of the Motion Picture Experts Group (MPEG) (also referred to as H.264). Motion compensation may describe a picture in terms of a transformation of a reference picture to a current picture. The reference picture may be temporally earlier as compared to the current picture and / or from the future as compared to the current picture. Summary of the Invention [Means for solving the problem]
[0007] In one aspect, the decoder includes circuitry configured to receive a bitstream, use the bitstream to determine, with respect to a current frame, whether the current frame is available to be used as a reference frame, and in response to determining that the current frame is available to be used as a reference frame, set a variable that characterizes that adaptive resolution management mode is prohibited, and reconstruct pixel data for the current frame, where adaptive resolution management mode is prohibited.
[0008] In another aspect, a method includes receiving a bitstream, determining using the bitstream, with respect to a current frame, whether the current frame is available for use as a reference frame, and in response to determining that the current frame is available for use as a reference frame, setting a variable that characterizes an adaptive resolution management mode being prohibited, and reconstructing pixel data for the current frame, where the adaptive resolution management mode is prohibited.
[0009] The details of one or more variations of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features and advantages of the subject matter described herein will be apparent from the description and drawings, and from the claims. The present invention provides, for example, the following: (Item 1) 13. A decoder, the decoder comprising circuitry comprising: receiving a bitstream; using the bitstream to determine, with respect to a current frame, whether the current frame is available to be used as a reference frame; in response to determining that the current frame is available for use as a reference frame, setting a variable that characterizes an adaptive resolution management mode as being prohibited; reconstructing pixel data of the current frame, wherein the adaptive resolution management mode is disabled; and A decoder configured to: (Item 2) 2. The decoder of claim 1, wherein determining whether the current frame is available for use as a reference frame comprises determining whether a type of the current frame is Type I. (Item 3) 2. The decoder of claim 1, wherein determining whether the current frame is available for use as a reference frame comprises determining whether a type of the current frame is type P. (Item 4) 2. The decoder of claim 1, wherein the type of the current frame is signaled within the bitstream. (Item 5) 2. The decoder of claim 1, wherein the current frame is a slice. (Item 6) 2. The decoder of claim 1, wherein parameters associated with the adaptive resolution management mode are not present in a picture header of the current frame. (Item 7) Item 1. The decoder of item 1, further configured to decode a current block from the bitstream. (Item 8) 2. The decoder of claim 1, wherein the current block comprises a coding tree unit. (Item 9) 2. The decoder of claim 1, wherein the current block comprises a prediction unit. (Item 10) an entropy decoder processor configured to receive the bitstream and decode the bitstream into quantized coefficients; an inverse quantization and inverse transform processor, the inverse quantization and inverse transform processor configured to process the quantized coefficients including performing an inverse discrete cosine; A deblocking filter; A frame buffer; Intra prediction processor Item 1. The decoder of item 1, further comprising: (Item 11) receiving a bitstream; using the bitstream to determine, with respect to a current frame, whether the current frame is available to be used as a reference frame; in response to determining that the current frame is available for use as a reference frame, setting a variable that characterizes an adaptive resolution management mode as being prohibited; reconstructing pixel data of the current frame, wherein the adaptive resolution management mode is disabled; and A method comprising: (Item 12) Item 12. The method of item 11, wherein determining whether the current frame is available for use as a reference frame includes determining whether a type of the current frame is Type I. (Item 13) Item 12. The method of item 11, wherein determining whether the current frame is available for use as a reference frame includes determining whether the type of the current frame is type P. (Item 14) Item 14. The method of item 13, wherein the type of the current frame is signaled within the bitstream. (Item 15) Item 12. The method of item 11, wherein the current frame is a slice. (Item 16) Item 12. The method of item 11, wherein parameters associated with the adaptive resolution management mode are not present in a picture header of the current frame. (Item 17) Item 12. The method of item 11, further comprising decoding a current block from the bitstream. (Item 18) Item 12. The method of item 11, wherein the current block comprises a coding tree unit. (Item 19) Item 12. The method of item 11, wherein the current block comprises a prediction unit. (Item 20) The receiving, determining, setting, and reconstructing at least one of the steps is implemented by a decoder, the decoder comprising: an entropy decoder processor configured to receive the bitstream and decode the bitstream into quantized coefficients; an inverse quantization and inverse transform processor, the inverse quantization and inverse transform processor configured to process the quantized coefficients including performing an inverse discrete cosine; A deblocking filter; A frame buffer; Intra prediction processor Item 12. The method of item 11, comprising: [Brief description of the drawings]
[0010] For the purpose of illustrating the invention, the drawings show aspects of one or more embodiments of the invention, it being understood, however, that the invention is not limited to the precise arrangements and instrumentalities shown in the drawings.
[0011] [Figure 1] FIG. 1 is a diagram of example reference frames and example predicted frames at various resolution scales. [Diagram 2] FIG. 2 is a diagram depicting an example reference frame, an example rescaled reference frame, and an example subsequent block prediction process. [Diagram 3] FIG. 3 is a process flow diagram illustrating an example process according to some implementations of the present subject matter. [Figure 4] FIG. 4 is a system block diagram illustrating an example decoder capable of decoding a bitstream in accordance with some implementations of the present subject matter. [Diagram 5]FIG. 5 is a process flow diagram illustrating an exemplary process for encoding video according to some implementations of the present subject matter. [Figure 6] FIG. 6 is a system block diagram illustrating an example video encoder in accordance with some implementations of the present subject matter. [Figure 7] FIG. 7 is a block diagram of a computing system that may be used to implement any one or more of the methodologies disclosed herein and any one or more portions thereof.
[0012] The drawings are not necessarily to scale and may be illustrated by phantom lines, schematic representations, and partial views. In some instances, details that are not necessary for an understanding of the embodiments or that make other details difficult to perceive may be omitted. Like reference symbols in the various drawings indicate like elements. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0013] Detailed Description In many current state-of-the-art encoders, resolution is managed by recoding and retransmitting entire portions of the video known as Groups of Pictures (GOPs). This requires transmitting intra-frames (I-frames), which can incur additional cost, since those frames account for the majority of the bits within a GOP.
[0014] The embodiments described in this disclosure relate to Adaptive Resolution Management (ARM), a technique that allows additional flexibility for video encoders / decoders, enabling bitrate savings in various use cases. In general, ARM involves performing prediction using a reference frame of a different resolution than the current frame. In current coding standards, the reference frame has the same resolution as the predicted frame. In ARM, the reference frame can be of lower or higher resolution than the frame being predicted. This approach can be used to downscale the video resolution, thus reducing the bitrate, or upscale the video resolution, thus enhancing the display characteristics of the video playback.
[0015] ARM may alternatively, or equivalently, be referred to as reference picture resampling (RPR) for purposes of this disclosure, and RPR and ARM may be used synonymously.
[0016] Some implementations of the present subject matter may include using ARM for any number of frames at any position within a GOP, thus eliminating the requirement for recoding I-frames.
[0017] FIG. 1 is an illustration of reference and predicted frames of various resolution scales. Frame 1 is smaller (lower resolution) than the reference frame, frame 2 is the same size (same resolution), while frame 3 is larger (higher resolution). "Resolution" as used in this disclosure is the number of pixels in a picture, frame, subframe, and / or other displayed area or portion thereof used in video playback, compression, or the like, with a higher number of pixels corresponding to a higher resolution and a lower number of pixels corresponding to a lower resolution. Resolution may be measured in terms of area, for example, but not by way of limitation, by using one or more dimensions of length, measured in pixels, that define an area. For example, a circular subframe or other region may have a resolution defined according to a radius. Alternatively, or in addition, resolution may be defined by a whole number of pixels.
[0018] As an example, and continuing to refer to FIG. 1, if the reference frame and / or subframe have a geometric form whose area spanned can be fully defined in terms of two length parameters, such as, but not limited to, a triangular, parallelogram, and / or rectangular form, the reference frame and / or subframe can have a resolution W×H (W and H can indicate the number of pixels describing the width (or base) and height dimensions of the reference frame and / or subframe, respectively). Each predicted frame can also have a resolution, which can be determined similarly to the resolution of the reference frame, e.g., frame 1 can have a lower resolution WS×HS, frame 2 can have the same resolution as the reference frame W×H, and frame 3 can have a higher resolution WL×HL. The widths and heights of the smaller and larger frames can be obtained by multiplying the reference width and height by an arbitrary rescaling constant (Rc), also referred to as a scaling ratio and / or constant. In the case of smaller frames, Rc can have a value between 0 and 1. In the case of larger frames, Rc may have a value greater than 1, for example, Rc may have a value from 1 to 4. Other values are possible. The rescaling constants may differ from another for one resolution dimension, for example, a rescaling constant Rch may be used to rescale the height while another rescaling constant Rcw may be used to rescale the width.
[0019] Still referring to FIG. 1, ARM may be implemented as a mode. In case of ARM mode activation at some point during decoding, the decoder may have already received a reference frame at resolution W×H and may rescale the predicted frame using a rescaling constant. In some implementations, the encoder may signal the rescaling constant to use to the decoder. The signaling may be performed in the sequence parameter set (SPS) corresponding to the GOP containing the current picture and / or the picture parameter set (PPS) corresponding to the current picture. For example, but not by way of limitation, the encoder may signal the rescaled parameters using fields such as pps_pic_width_in_luma_samples, pps_pic_height_in_luma_samples, pps_scaling_win_left_offset, pps_scaling_win_right_offset, pps_scaling_win_top_offset, pps_scaling_win_bottom_offset, and / or sps_num_subpics_minus1.
[0020] 1 , the W and H parameters as described above may be represented using, but not limited to, the variables CurrPicScalWinWidthL and CurrPicScalWinHeightL, respectively, which may be derived from the signaled parameters as described above using one or more mathematical relationships between the signaled parameters and the variables. For example, but not limited to, CurrPicScalWinWidthL may be derived according to the following equation: CurrPicScalWinWidthL = pps_pic_width_in_luma_samples - SubWidthC * (pps_scaling_win_right_offset + pps_scaling_win_left_offset)
[0021] As a further non-limiting example, CurrPicScalWinHeightL may be derived according to the following equation: CurrPicScalWinWidthL = pps_pic_width_in_luma_samples - SubWidthC * (pps_scaling_win_right_offset + pps_scaling_win_left_offset)
[0022] Those skilled in the art will recognize, upon review of the entirety of this disclosure, various alternative calculations that may be used to derive the variables described above. An encoder may alternatively or additionally directly signal one or more such variables Rc, Rch, and / or Rcw, for example, but not by way of limitation, within the PPS and / or SPS.
[0023] Alternatively, or in addition, still referring to FIG. 1, a rescaling constant and / or set of rescaling constants as described above may be signaled in the bitstream using a reference to a stored scaling constant and / or multiple scaling constants and / or an index of a frame and / or block that has been previously signaled and / or utilized using a scaling constant and / or multiple scaling constants. The reference to an index of a stored scaling constant may be explicitly signaled and / or determined from one or more additional parameters signaled in the bitstream. For example, but not by way of limitation, the decoder may identify a group of pictures that includes a reference frame and / or a current frame, and if a rescaling constant has been previously signaled and / or utilized within such group of pictures with the reference frame signaled as applicable to the current frame and / or current group of pictures, or the like, the decoder may identify that rescaling constant for use as a rescaling constant with the current frame.
[0024] In some implementations, with continued reference to FIG. 1, ARM operations may be performed according to the block level of the encoded frame. For example, a reference frame may be first rescaled, followed by prediction as depicted in FIG. 2, which is a diagram depicting a reference frame, a rescaled reference frame, and a subsequent block prediction process. The block prediction process may be performed on the scaled reference frame (having a scaled resolution) rather than the original reference frame. Rescaling the reference frame may include rescaling according to any parameters signaled by the encoder as described above, for example, but not limited to, if a reference frame to be used with the current picture is signaled via a reference to an index value associated with the reference frame or equivalent, the signaled reference frame may be rescaled according to any method of rescaling described above prior to prediction. The rescaled reference frames may be stored in memory and / or buffers, which may include, but are not limited to, a buffer that identifies the frame contained therein by an index according to a frame retrieval that may be performed, and which may include a decoded picture buffer (DCB) and / or one or more additional buffers implemented by the decoder. The prediction process may include, for example, inter-picture prediction, including motion compensation.
[0025] Some implementations of block-based ARM may allow flexibility to apply optimal filters per block instead of applying the same filter across the entire frame. In some implementations, skip ARM mode may be considered a possibility such that some blocks (e.g., based on uniformity of pixel and bitrate cost) may be in skip ARM mode (such that rescaling would not change the bitrate). Skip ARM mode may be signaled in the bitstream, for example, but not limited to, skip ARM mode may be signaled in the PPS parameters. Alternatively or additionally, the decoder may determine that skip ARM mode is active based on one or more parameters set by the decoder and / or signaled in the bitstream. Spatial filters used within the block-based ARM may include, but are not limited to, bicubic spatial filters applying bicubic interpolation, bilinear spatial filters applying bilinear interpolation, Lanczos filters using Lanczos filtering, and / or sinc filters, Lanczos resampling using a combination of sinc function interpolation and / or signal reconstruction techniques, or equivalents, and those skilled in the art will recognize various filters that may be used consistently with the present disclosure for interpolation upon review of the entire disclosure. Interpolation filters may include, as non-limiting examples, any of the filters described above, such as, but not limited to, low-pass filters that may be used with an upsampling process whereby pixels between pixels of a previous block and / or frame of scaling may be initialized to zero and then the output of a low-pass filter may be taken. Alternatively or additionally, any luma sample interpolation filtering process may be used. Luma sample interpolation may include calculation of an interpolated value at a half-sample interpolation filter index that falls between two consecutive sample values of an unscaled sample array.The calculation of the interpolation value may be performed, without limitation, by reading the coefficients and / or weights from a look-up table, and the selection of the look-up table may be performed as a function of the motion model of the coding unit and / or the scaling ratio amount, for example, as determined using the scaling constants as described above. The calculation may include, without limitation, performing a weighted sum of adjacent pixel values, where the weights are read from a look-up table. The calculated value may alternatively or additionally be shifted, for example, without limitation, the value may be shifted by a minimum value (4, i.e., bit depth -8), 6, a maximum value (2, i.e., 14-bit depth), or the like. Those skilled in the art will recognize various alternative or additional implementations that may be used for the interpolation filter upon review of the entirety of this disclosure.
[0026] Still referring to FIG. 2, adaptive resolution management may enable rescaling of frames and / or regions of frames to improve compression efficiency and / or quality. A key issue is the reproducibility of the rescaling method at the decoder. Depending on the rescaling method used, the decoder and encoder may not be able to reproduce the same scaled-back frame. In such cases, using the rescaled frame as a reference frame may cause prediction mismatch and / or cause the encoder and decoder to drift. The drift may become visible as coding artifacts (quality issues). To avoid these drift issues, ARM implicit and explicit signaling may be utilized as described in this disclosure.
[0027] For example, with further reference to FIG. 2, the implicit signaling of ARM may be based on frame type. For example, in some implementations, the decoder and / or encoder may be configured such that a frame that may be used as a reference frame may not be encoded using ARM, in other words, ARM may be prohibited with respect to a frame available as a reference frame. For example, if the frame type or slice type signaled in the picture header indicates that the frame is available for use as a reference frame, such as a frame signaled in the picture header as being of type I or type P, the decoder and / or encoder may determine that the frame may not be encoded using ARM and that all of the ARM-related parameters may not be present in the picture header of the frame. Parameters such as those described above may be signaled in the SPS and / or PPS, without limitation. For example, the SPS may signal a parameter denoted as sps_res_change_in_clvs_allowed_flag, which when equal to 1, may specify that the picture spatial resolution may change. One aspect of resolution management may signal that the resolution may change within a sequence. As a further non-limiting example, gci_no_ref_pic_resampling_constraint_flag equal to 1 may specify that sps_ref_pic_resampling_enabled_flag for all pictures in OlsInScope shall be equal to 0. gci_no_ref_pic_resampling_constraint_flag equal to 0 may not impose such constraint. In the slice header, sh_slice_type may specify the coding type of the slice between types I, P, and B. ph_non_ref_pic_flag equal to 1 may specify that the current picture is never used as a reference picture. ph_non_ref_pic_flag equal to 0 may specify that the current picture may or may not be used as a reference picture.
[0028] FIG. 3 is a process flow diagram illustrating an example embodiment of a process 300 of adaptive resolution management that may enable additional flexibility for video encoders / decoders, enabling bitrate savings in various use cases.
[0029] At step 305, still referring to FIG. 3, a bitstream is received by the decoder. A current block may be contained within the bitstream that the decoder receives. The bitstream may include data found in a stream of bits that is input to the decoder, for example, when using data compression. The bitstream may include information necessary to decode the video. Receiving may include extracting and / or parsing the block and associated signaling information from the bitstream. In some implementations, the current block may include a coding tree unit (CTU), a coding unit (CU), and / or a prediction unit (PU).
[0030] At step 310, still referring to FIG. 3, it may be determined using the bitstream whether the current frame is available to be used as a reference frame. In some implementations, determining whether the current frame is available to be used as a reference frame includes determining whether the type of the current frame is type I or type P, as described above. As a non-limiting example, in the slice header, sh_slice_type may specify the coding type of the slice between types I, P, and B. The type of the current frame may be signaled in the bitstream. In some implementations, as a non-limiting example, the current frame may include a slice, which is defined as a set of CTUs in a tile and / or a set of an integer number of contiguous tiles.
[0031] Still referring to FIG. 3, in step 315, a variable characterizing that the adaptive resolution management mode is prohibited may be set in response to determining that the current frame is available to be used as a reference frame. In some implementations, parameters associated with the adaptive resolution management mode are not present in the picture header of the current frame. As a non-limiting example, the SPS may signal sps_res_change_in_clvs_allowed_flag equal to 1, which specifies that the picture spatial resolution may change. Slice type-based disabling may be signaled at the encoder based on the picture type.
[0032] At step 340, still referring to FIG. 3, pixel data of the current frame may be reconstructed. Adaptive resolution management mode may be inhibited during reconstruction. In an embodiment, a determination that the current frame is not available as a reference frame may lead to a determination that the ARM is available. For example, the SPS may signal sps_res_change_in_clvs_allowed_flag equal to 1, specifying that the picture spatial resolution may change, e.g., as described above.
[0033] 4 is a system block diagram illustrating an example decoder 400 capable of frame type-based adaptive resolution management as described in this disclosure. The decoder 400 may include an entropy decoder processor 404, an inverse quantization and inverse transform processor 408, a deblocking filter 412, a frame buffer 416, a motion compensation processor 420, and / or an intra prediction processor 424.
[0034] In operation, still referring to FIG. 4, a bitstream 428 may be received by the decoder 400 and input to the entropy decoder processor 404, which may entropy decode a portion of the bitstream into quantized coefficients. The quantized coefficients may be provided to the inverse quantization and inverse transform processor 408, which may perform inverse quantization and inverse transform and generate a residual signal, which may be added to the output of the motion compensation processor 420 or the intra prediction processor 424, depending on the processing mode. The output of the motion compensation processor 420 and the intra prediction processor 424 may include block predictions based on previously decoded blocks. The sum of the predictions and residuals may be processed by the deblocking filter 412 and stored in the frame buffer 416.
[0035] 5 is a process flow diagram illustrating an example process 500 for encoding video with adaptive resolution management, which may allow additional flexibility for a video encoder and / or decoder, enabling bitrate savings in various use cases. At step 505, a video frame may undergo initial block segmentation, for example, using a tree-structured macroblock partitioning scheme, which may include partitioning a picture frame into CTUs and CUs.
[0036] In step 510, block-based adaptive resolution management may be performed, including resolution scaling of the frame or a portion thereof.
[0037] The block may be encoded and included in the bitstream at step 515. The encoding may include, for example, utilizing inter-prediction and intra-prediction modes.
[0038] FIG. 6 is a system block diagram illustrating an example video encoder 600 capable of frame type-based adaptive resolution management as described in this disclosure. The example video encoder 600 may receive an input video 604, which may first be segmented or divided according to a processing scheme such as a tree-structured macroblock partitioning scheme (e.g., quadtree+binarytree). An example of a tree-structured macroblock partitioning scheme may include partitioning a picture frame into large block elements called coding tree units (CTUs). In some implementations, each CTU may be further partitioned one or more times into several sub-blocks, called coding units (CUs). The end result of this partitioning may include a group of sub-blocks, which may be called prediction units (PUs). Transform units (TUs) may also be utilized.
[0039] 6, an exemplary video encoder 600 may include an intra-prediction processor 608, a motion estimation / compensation processor 612, which may also be referred to as an inter-prediction processor, which is capable of building a motion vector candidate list, including adding global motion vector candidates to the motion vector candidate list, a transform / quantization processor 616, an inverse quantization / inverse transform processor 620, an in-loop filter 624, a decoded picture buffer 628, and / or an entropy coding processor 632. Bitstream parameters may be input to the entropy coding processor 632 for inclusion in an output bitstream 636.
[0040] In operation, with continued reference to Figure 6, for each block of a frame of the input video 604, it may be determined whether to process the block via intra-picture prediction or using motion estimation / compensation. The block may be provided to the intra-prediction processor 608 or the motion estimation / compensation processor 612. If the block is to be processed via intra-prediction, the intra-prediction processor 608 may perform processing and output a predictor. If the block is to be processed via motion estimation / compensation, the motion estimation / compensation processor 612 may perform processing including building a motion vector candidate list, including adding global motion vector candidates to the motion vector candidate list, if applicable.
[0041] With further reference to FIG. 6, a residual may be formed by subtracting the predictor from the input video. The residual may be received by a transform / quantization processor 616, which may perform a transform process (e.g., a discrete cosine transform (DCT)) to produce coefficients, which may be quantized. The quantized coefficients and any associated signaling information may be provided to an entropy coding processor 632 for entropy encoding and inclusion in an output bitstream 636. The entropy encoding processor 632 may assist in encoding signaling information related to encoding the current block. In addition, the quantized coefficients may be combined with the predictor and provided to an inverse quantization / inverse transform processor 620, which may reconstruct pixels that may be processed by an in-loop filter 624, the output of which may be stored in a decoded picture buffer 628 for use by a motion estimation / compensation processor 612, which may build a motion vector candidate list, including adding global motion vector candidates to the motion vector candidate list.
[0042] 6, several variations have been described in detail above, but other modifications or additions are possible. For example, in some implementations, the current block may include any symmetric block (8×8, 16×16, 32×32, 64×64, 128×128, and the like) and any asymmetric block (8×4, 16×8, and the like).
[0043] In some implementations, still referring to FIG. 6, a quad-tree plus binary decision tree (QTBT) may be implemented. In QTBT, at the coding tree unit level, the partition parameters of QTBT may be dynamically derived to fit local characteristics without transmitting any overhead. Then, at the coding unit level, a joint classifier decision tree structure may eliminate unnecessary iterations and control the risk of erroneous prediction. In some implementations, an LTR frame block update mode may be available as an additional option available per leaf node of QTBT.
[0044] In some implementations, still referring to Figure 6, additional syntax elements may be signaled at different hierarchical levels of the bitstream. For example, a flag may be enabled throughout a sequence by including an enabled flag coded in a sequence parameter set (SPS). Additionally, a CTU flag may be coded at the coding tree unit (CTU) level.
[0045] Some embodiments may include a non-transitory computer program product (i.e., a physically embodied computer program product) storing instructions that, when executed by one or more data processors of one or more computing systems, cause at least one data processor to perform operations herein.
[0046] In some embodiments disclosed herein, the decoder may include circuitry configured to receive a bitstream, use the bitstream to determine, with respect to a current frame, whether the current frame is available to be used as a reference frame, and in response to determining that the current frame is available to be used as a reference frame, set a variable that characterizes that adaptive resolution management mode is prohibited, and reconstruct pixel data for the current frame, where adaptive resolution management mode is prohibited.
[0047] In some embodiments, determining whether the current frame is available for use as a reference frame may include determining whether a type of the current frame is type I. Determining whether the current frame is available for use as a reference frame may include determining whether a type of the current frame is type P. The type of the current frame may be signaled in the bitstream. The current frame may be a slice. Parameters associated with the adaptive resolution management mode may not be present in a picture header of the current frame. The decoder may further be configured to decode a current block from the bitstream. The current block may include a coding tree unit. The current block may include a prediction unit. The decoder may include an entropy decoder processor configured to receive the bitstream and decode the bitstream into quantized coefficients, an inverse quantization and inverse transform processor configured to process the quantized coefficients, including performing inverse discrete cosine, a deblocking filter, a frame buffer, and an intra prediction processor.
[0048] Embodiments disclosed herein include a method including receiving a bitstream, determining using the bitstream, with respect to a current frame, whether the current frame is available to be used as a reference frame, and in response to determining that the current frame is available to be used as a reference frame, setting a variable that characterizes that adaptive resolution management mode is prohibited, and reconstructing pixel data for the current frame, where adaptive resolution management mode is prohibited.
[0049] In some embodiments, determining whether the current frame is available for use as a reference frame may include determining whether a type of the current frame is type I. Determining whether the current frame is available for use as a reference frame may include determining whether a type of the current frame is type P. The type of the current frame may be signaled in the bitstream. The current frame may be a slice. Parameters associated with the adaptive resolution management mode may not be present in a picture header of the current frame. The method may include decoding the current block from the bitstream. The current block may include a coding tree unit. The current block may include a prediction unit. At least one of receiving, determining, setting, and reconstructing may be performed by a decoder including an entropy decoder processor configured to receive the bitstream and decode the bitstream into quantized coefficients, an inverse quantization and inverse transform processor configured to process the quantized coefficients including performing inverse discrete cosine, a deblocking filter, a frame buffer, and an intra prediction processor.
[0050] It should be noted that any one or more of the aspects and embodiments described herein may be conveniently implemented using digital electronic circuitry, integrated circuitry, specially designed application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), computer hardware, firmware, software, and / or combinations thereof, as embodied and / or implemented in one or more machines (e.g., one or more computing devices utilized as user computing devices for electronic documents, one or more server devices, such as document servers, etc.) programmed according to the teachings herein, as would be apparent to one of ordinary skill in the computer arts. These various aspects or features may include implementation in one or more computer programs and / or software executable and / or interpretable on a programmable system including at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device. Appropriate software coding may be readily prepared by skilled programmers based on the teachings of the present disclosure, as would be apparent to one skilled in the software arts. The aspects and implementations discussed above that employ software and / or software modules may also include appropriate hardware to assist in the implementation of the machine-executable instructions of the software and / or software modules.
[0051] Such software may be a computer program product employing a machine-readable storage medium. A machine-readable storage medium may be any medium capable of storing and / or encoding a sequence of instructions for execution by a machine (e.g., a computing device) and causing the machine to perform any one of the methodologies and / or embodiments described herein. Examples of machine-readable storage media include, but are not limited to, magnetic disks, optical disks (e.g., CDs, CD-Rs, DVDs, DVD-Rs, etc.), magneto-optical disks, read-only memory "ROM" devices, random access memory "RAM" devices, magnetic cards, optical cards, solid-state memory devices, EPROMs, EEPROMs, programmable logic devices (PLDs), and / or any combination thereof. Machine-readable media, as used herein, is intended to include a single medium and a collection of physically separate media, such as, for example, a collection of compact discs or one or more hard disk drives in combination with a computer memory. As used herein, machine-readable storage media does not include a transitory form of signal transmission.
[0052] Such software may also include information (e.g., data) carried as a data signal on a data carrier, such as a carrier wave. For example, machine-executable information may be included as a data carrying signal embodied in a data carrier, which signal encodes a sequence of instructions, or a portion thereof, for execution by a machine (e.g., a computing device), and any associated information (e.g., data structures and data) that causes the machine to perform any one of the methodologies and / or embodiments described herein.
[0053] Examples of computing devices include, but are not limited to, e-book reading devices, computer workstations, terminal computers, server computers, handheld devices (e.g., tablet computers, smart phones, etc.), web appliances, network routers, network switches, network bridges, any machine capable of executing a sequence of instructions that define actions to be taken by the machine, and any combination thereof. In one embodiment, the computing device may include and / or be included within a kiosk.
[0054] 7 shows a diagrammatic representation of one embodiment of a computing device in the exemplary form of a computer system 700 on which a set of instructions for causing a control system to perform any one or more of the aspects and / or methodologies of the present disclosure may be executed. It is also envisioned that multiple computing devices may be utilized to implement a specially configured set of instructions for causing one or more of the devices to perform any one or more of the aspects and / or methodologies of the present disclosure. The computer system 700 includes a processor 704 and a memory 708 that communicate with each other and with other components via a bus 712. The bus 712 may include any of several types of bus structures, including, but not limited to, a memory bus, a memory controller, a peripheral bus, a local bus, and any combination thereof, using any of a variety of bus architectures.
[0055] Memory 708 may include a variety of components (e.g., machine-readable media), including, but not limited to, random access memory components, read-only components, and any combination thereof. In one embodiment, a basic input / output system 716 (BIOS), including the basic routines that help to transfer information between elements within computer system 700, such as during start-up, may be stored in memory 708. Memory 708 may also include (e.g., stored on one or more machine-readable media) instructions (e.g., software) 720 that embody any one or more of the aspects and / or methodologies of the present disclosure. In another embodiment, memory 708 may further include any number of program modules, including, but not limited to, an operating system, one or more application programs, other program modules, program data, and any combination thereof.
[0056] Computer system 700 may also include a storage device 724. Examples of storage devices (e.g., storage device 724) include, but are not limited to, hard disk drives, magnetic disk drives, optical disk drives in combination with optical media, solid-state memory devices, and any combination thereof. Storage device 724 may be connected to bus 712 by an appropriate interface (not shown). Exemplary interfaces include, but are not limited to, SCSI, Advanced Technology Attachment (ATA), Serial ATA, Universal Serial Bus (USB), IEEE 1394 (FIREWIRE®), and any combination thereof. In one embodiment, storage device 724 (or one or more components thereof) may be removably interfaced with computer system 700 (e.g., via an external port connector (not shown)). In particular, storage device 724 and associated machine-readable media 728 may provide non-volatile and / or volatile storage of machine-readable instructions, data structures, program modules, and / or other data for computer system 700. In one embodiment, the software 720 may reside, completely or partially, within the machine-readable medium 728. In another embodiment, the software 720 may reside, completely or partially, within the processor 704.
[0057] Computer system 700 may also include input devices 732. In one embodiment, a user of computer system 700 may type commands and / or other information into computer system 700 via input devices 732. Examples of input devices 732 include, but are not limited to, alphanumeric input devices (e.g., keyboards), pointing devices, joysticks, gamepads, audio input devices (e.g., microphones, voice response systems, etc.), cursor control devices (e.g., mice), touchpads, optical scanners, video capture devices (e.g., still cameras, video cameras), touch screens, and any combination thereof. Input devices 732 may be interfaced to bus 712 via any of a variety of interfaces (not shown), including, but not limited to, a serial interface, a parallel interface, a game port, a USB interface, a FIREWIRE® interface, an interface directly to bus 712, and any combination thereof. Input devices 732 may include a touch screen interface, which may be part of or separate from display 736, discussed further below. The input device 732 may be utilized as a user selection device for selecting one or more graphical representations within a graphical interface such as those described above.
[0058] A user may also input commands and / or other information to computer system 700 via storage device 724 (e.g., a removable disk drive, flash drive, etc.) and / or network interface device 740. A network interface device, such as network interface device 740, may be utilized to connect computer system 700 to one or more of a variety of networks, such as network 744, and one or more remote devices 748 connected thereto. Examples of network interface devices include, but are not limited to, a network interface card (e.g., a mobile network interface card, a LAN card), a modem, and any combination thereof. Examples of networks include, but are not limited to, a wide area network (e.g., the Internet, a corporate network), a local area network (e.g., a network associated with an office, a building, a campus, or other relatively small geographic space), a telephone network, a data network associated with a telephone / voice provider (e.g., a data and / or voice network of a mobile communications provider), a direct connection between two computing devices, and any combination thereof. A network, such as network 744, may employ wired and / or wireless modes of communication. In general, any network topology may be used. Information (eg, data, software 720, etc.) may be communicated to and / or from computer system 700 via network interface device(s) 740.
[0059] Computer system 700 may further include a video display adapter 752 for communicating images displayable on a display device, such as display device 736. Examples of display devices include, but are not limited to, a liquid crystal display (LCD), a cathode ray tube (CRT), a plasma display, a light emitting diode (LED) display, and any combination thereof. Display adapter 752 and display device 736 may be utilized in combination with processor 704 to provide graphical representations of aspects of the disclosure. In addition to a display device, computer system 700 may include one or more other peripheral output devices, including, but not limited to, audio speakers, a printer, and any combination thereof. Such peripheral output devices may be connected to bus 712 via a peripheral interface 756. Examples of peripheral interfaces include, but are not limited to, a serial port, a USB connection, a FIREWIRE® connection, a parallel connection, and any combination thereof.
[0060] The foregoing is a detailed description of exemplary embodiments of the present invention. Various modifications and additions may be made without departing from the spirit and scope of the present invention. Each feature of the various embodiments described above may be combined with features of other described embodiments as appropriate to provide a combination of features in a related new embodiment. Furthermore, while the foregoing describes several separate embodiments, what is described herein is merely illustrative of the application of the principles of the present invention. In addition, although certain methods herein may be illustrated and / or described as being performed in a specific order, the order may be highly variable among those skilled in the art to achieve the embodiments as disclosed herein. Thus, the present description is intended to be taken as an example only, and is not intended to otherwise limit the scope of the present invention.
[0061] In the above description and in the claims, phrases such as "at least one of" or "one or more of" may occur and be followed by a conjunctive enumeration of elements or features. The term "and / or" may also occur within a list of two or more elements or features. Unless otherwise implied or expressly contradicted by the context in which such a phrase is used, it is intended to mean any of the elements or features listed individually or any of the recited elements or features in combination with any of the other recited elements or features. For example, the phrases "at least one of A and B," "one or more of A and B," and "A and / or B" are intended to mean "A alone, B alone, or A and B together," respectively. A similar interpretation is also intended with respect to lists containing three or more items. For example, the phrases "at least one of A, B, and C," "one or more of A, B, and C," and "A, B, and / or C" are intended to mean "A alone, B alone, C alone, A and B together, A and C together, B and C together, or A and B and C together." Additionally, use of the term "based on" above and in the claims is intended to mean "based at least on," such that unrecited features or elements are also permissible.
[0062] The subject matter described herein may be embodied in a system, an apparatus, a method, and / or an article, depending on the desired configuration. The implementations described in the foregoing description do not represent all implementations consistent with the subject matter described herein. Instead, they are merely some examples consistent with aspects related to the subject matter described. Although some variations have been described in detail above, other modifications or additions are possible. In particular, further features and / or variations may be provided in addition to those described herein. For example, the implementations described above may be directed to various combinations and subcombinations of the disclosed features and / or combinations and subcombinations of some further features disclosed above. In addition, the logic flow depicted in the accompanying figures and / or described herein does not necessarily require the particular order shown or sequential order to achieve the desired results. Other implementations may be within the scope of the following claims.
Claims
[Claim 1] The invention described in this specification.
Citation Information
Patent Citations
Adaptive resolution video decoder
EP3057319A1
Apparatus for reference picture resampling generation and method thereof and video decoding system using the same
US20100150231A1
Adaptive motion vector precision for video coding
US20180098089A1
Image encoding and decoding using resolution enhancement technique
WO2019059721A1
An apparatus, a method and a computer program for omnidirectional video
WO2019141907A1
