Adaptive Block Update for Unavailable Reference Frames Using Explicit and Implicit Signaling

Abstract

A decoder includes circuitry configured to: receive a current block; determine that the current block is a block for updating an unavailable reference frame based on the current block; and update the unavailable reference frame using the current block. Related apparatuses, systems, techniques, and articles are also described.

Description

[0001] Cross - Reference to Related Applications

[0002] This application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 772,066, filed on November 27, 2018, entitled "ADAPTIVE BLOCK UPDATE OF LONG - TERM REFERENCE FRAMES USING EXPLICIT SIGNALING", the entire content of which is incorporated herein by reference. This application also claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 771,941, filed on November 27, 2018, entitled "ADAPTIVE BLOCK UPDATE OF LONG TERM REFERENCE FRAMES USING IMPLICIT SIGNALING", the entire content of which is incorporated herein by reference. Field of the Invention

[0003] The present invention generally relates to the field of video compression. Specifically, the present invention relates to adaptive block update of unavailable reference images. Background Art

[0004] A video codec may include electronic circuitry or software for compressing or decompressing digital video. It can convert uncompressed video into a compressed format and vice versa. In the context of video compression, a device that compresses video (and / or performs some of its functions) is generally referred to as an encoder, and a device that decompresses video (and / or performs some of its functions) is referred to as a decoder.

[0005] The format of the compressed data may conform to standard video compression specifications. Compression may be lossy, such that the compressed video may lack some information present in the original video. Such consequences may include that the decompressed video may have lower quality than the original uncompressed video because there is not enough information to accurately reconstruct the original video.

[0006] There may be complex relationships between video quality, the amount of data used to represent the video (e.g., determined by the bit rate), the complexity of the encoding and decoding algorithms, the sensitivity to data loss and errors, the ease of editing, random access, end - to - end latency (e.g., the latency time), etc.

[0007] Motion compensation can include methods of predicting a video frame or a portion of a given reference frame (e.g., a previous frame and / or a future frame) by considering the motion of a camera and / or objects in the video. It can be used for video compression, encoding and decoding video data, for example, encoding and decoding using the Moving Picture Experts Group (MPEG)-2 (also known as Advanced Video Coding (AVC) and H.264) standard. Motion compensation can describe an image based on the transformation from a reference image to the current image. When compared with the current image, the reference image can be temporally previous; or, when compared with the current image, the reference image can be from the future, or the reference image can include a long term reference (LTR) frame. Compression efficiency can be improved when an image can be accurately synthesized based on previously transmitted and / or stored images.

[0008] Current standards such as H.264 and H.265 allow for updating frames such as long term reference frames by signaling a newly decoded frame that is saved and available as a reference frame. This update is signaled by the encoder and the entire frame is updated. However, updating the entire frame is costly, especially in cases where only a small portion of the static background has changed. Partial frame updates are possible, but typically involve complex and computationally expensive processes to implement the frame update. Summary of the Invention

[0009] In one aspect, a decoder includes circuitry configured to: receive a bitstream including a current encoded block; determine a decoded current block; determine that an unavailable reference block update mode is enabled for the current encoded block in the bitstream; and update an unavailable reference frame using the decoded current block.

[0010] In another aspect, a method includes: receiving, by a decoder, a bitstream including a current encoded block. The method includes. The method includes determining a decoded current block. The method includes determining that an unavailable reference block update mode is enabled for the current encoded block in the bitstream. The method includes updating an unavailable reference frame using the decoded current block.

[0011] In another aspect, a decoder includes circuitry configured to: receive a current block; mark the current block for updating an unavailable reference frame based on the current block; and update the unavailable reference frame using the current block.

[0012] In another aspect, a method includes receiving, by a decoder, a current block. The method includes marking the current block for updating an unavailable reference frame based on the current block. The method includes updating the unavailable reference frame using the current block.

[0013] Details of one or more variations of the subject matter described herein are set forth in the following drawings and the detailed description. Other features and advantages of the subject matter described herein will be apparent from the detailed description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] To illustrate the present invention, the drawings show various aspects of one or more embodiments of the present invention. However, it should be understood that the present invention is not limited to the exact arrangements and instrumentalities shown in the drawings, in which:

[0015] Figure 1 is a process flow diagram showing an exemplary process of updating a UR frame block, where the UR frame is partially updated by a decoder based on a decoded current block;

[0016] Figure 2 is a process flow diagram showing an exemplary process of updating a UR frame block using implicit signaling, where the UR frame is partially updated by a decoder based on a decoded current block;

[0017] Figure 3 is a block diagram showing an exemplary frame including a current block X and three adjacent blocks, the three adjacent blocks having associated motion vectors for motion estimation using a UR frame;

[0018] Figure 4 is a block diagram showing another exemplary frame including blocks that do not use a UR frame as a reference and blocks that use a UR frame as a reference;

[0019] Figure 5 is a system block diagram showing an exemplary decoder capable of decoding a bitstream using UR frame block update;

[0020] Figure 6 is a process flow diagram showing an exemplary process of encoding video using UR frame block update according to some aspects of the current subject matter;

[0021] Figure 7 is a process flow diagram showing an exemplary process of encoding video using UR frame block update according to some aspects of the current subject matter, the exemplary process being capable of reducing encoding complexity while improving compression efficiency;

[0022] Figure 8 is a system block diagram showing an exemplary video encoder capable of signaling UR block update for the decoder side; and

[0023] Figure 9 is a block diagram of a computing system that can be used to implement any one or more of the methods and any one or more parts thereof disclosed herein.

[0024] The accompanying drawings are not necessarily drawn to scale and may be illustrated by dashed lines, schematic diagrams, and partial views. In some cases, details that are unnecessary for understanding the embodiments or that make other details difficult to understand may have been omitted. Similar reference numerals in the various drawings denote similar elements. Detailed Description

[0025] Some embodiments of the present subject matter include methods for performing partial updates of unavailable reference (UR) frames on the decoder side. An unavailable reference (UR) frame is a frame and / or image that is used to create predicted frames and / or images in one or more groups of pictures (GOPs), but the frame and / or image itself is not displayed in the video image. A frame marked as a UR frame in the video bitstream can be used as a reference until the frame is explicitly removed by bitstream signaling. UR frames can improve prediction and compression efficiency in scenarios with a static background over a long period of time (e.g., the background of a video conference or a video of a parking lot surveillance). However, over time, the background of the scene gradually changes (e.g., a car becomes part of the background scene when it is parked in an empty location). By utilizing the unavailable reference block update mode, a signal for partial UR frame updates can be explicitly sent in the bitstream; wherein, when the unavailable reference block update mode is enabled for a given current block, the unavailable reference block update mode can indicate that the decoded current block is used to update spatially co-located pixels within the UR frame. Such partial UR frame updates can improve prediction without requiring an entire UR frame update, which can reduce residuals and thus improve compression performance. Additionally, by explicitly sending a signal for UR block updates, UR updates can be simply implemented without the need to utilize inadequate and complex processes.

[0026] Alternatively or additionally, a signal for partial UR frame updates can be implicitly sent such that compared to some alternative methods, explicit signaling in the bitstream is not required or can be reduced. The implicit signaling method can indicate that the decoded current block is used to update spatially co-located pixels within the UR frame. Such partial UR frame updates can improve prediction without requiring an entire UR frame update, which can reduce residuals and thus improve compression performance. Additionally, by implicitly sending a signal for UR block updates, the header overhead can be reduced, thereby reducing the bitrate and improving compression. Further, in some embodiments, UR updates can be simply implemented without the need to utilize inadequate and complex processes.

[0027] In some embodiments, implicit signaling of video blocks identifies cases where the current and adjacent block contexts are used to implicitly determine which blocks in the UR frame are updated (e.g., which parts of the UR frame should be updated). Now refer to Figure 1, which provides a process flow diagram of an exemplary process 100 showing the use of explicit signaling for UR frame block updates, where the UR frame is partially updated by the decoder based on the decoded current block. Such UR block updates can improve prediction without the need for a full UR frame update, which can reduce the residual and thus improve compression performance.

[0028] At step 105, still referring to Figure 1 , the decoder receives a bitstream including the current encoded block. The bitstream can include, for example, the data found in the bitstream, where when data compression is used, the bitstream is the input to the decoder. The bitstream can include the information needed to decode the video. Receiving can include extracting and / or parsing the blocks and associated signaling information in the bitstream. In some embodiments, the current encoded block can include a coding tree unit (CTU), a coding unit (CU), or a prediction unit (PU).

[0029] At step 110, continuing to refer to Figure 1 , the decoded current block can be determined. For example, the received current encoded block can be decoded, for example, by using inter-frame prediction. Decoding via inter-frame prediction can include using a previous frame, a future frame, or a UR frame as a reference for calculating the prediction, which can be combined with the residual included in the bitstream.

[0030] At step 115, continuing to refer to Figure 1 , it can be determined whether the unavailable reference block update mode is enabled for the current encoded block in the bitstream. The UR block update mode can be included in the header of the bitstream and / or a field included in the header. For example, it can be determined that the UR block update mode field in the header of the bitstream is enabled. Headers such as a picture parameter set (PPS) and / or a sequence parameter set (SPS) can be used to selectively enable the signaling of UR frame block updates. A field such as UR_BLOCK_UPDATE can take a true or false value (e.g., 0 or 1), but is not limited thereto.

[0031] Still referring to Figure 1, if the UR block update mode is enabled (e.g., if the field UR_BLOCK_UPDATE in the header is set), then at 120, the decoder can use the decoded current block to update the unavailable reference frames. Such an update can include performing a UR block update, where the decoded current block is used to update (e.g., modify) the pixels in the UR frame that are spatially co-located with the current block for which the UR frame block update mode is enabled. In some embodiments, the block update mechanism can be explicit or implicit. For example, a portion of the UR frame (e.g., a block) can be updated by using the pixel values of the decoded current block to update the co-located pixels (e.g., luminance values) in the UR frame; in other words, updating the UR frame can include replacing the luminance values of the UR frame with the luminance values that are spatially co-located with the decoded current block. In some embodiments, the UR frame can be updated according to another mechanism, such as updating a portion of the UR frame with the average of the initial co-located UR frame pixel values and the current decoded block pixel values. After reviewing the entire disclosure, those skilled in the art will recognize the various other mechanisms that can be employed. In some embodiments, the update can include using multiple decoded blocks to update multiple UR frame blocks, which can be performed using any of the methods described above for updating the UR frame using the decoded current block. The update can include, but is not limited to, using any of the above processes and / or process steps to generate blocks and / or frames with default chrominance and / or luminance values via one or more decoded blocks and replacing and / or updating the default chrominance and / or luminance values; the update can include generating a UR frame, which can be achieved by creating a UR frame with default values and updating the default values as described above.

[0032] Continuing to refer Figure 1 , for subsequent current blocks, the updated UR frame can be used as a reference frame for inter-frame prediction. For example, a second coded block can be received. It can be determined whether the inter-frame prediction mode is enabled for the second coded block. The updated UR frame can be used as a reference frame, and the second decoded block can be determined according to the inter-frame prediction mode. For example, decoding via inter-frame prediction can include using the updated UR frame as a reference for calculating the prediction, which can be combined with the residuals included in the bitstream.

[0033] Still referring Figure 1 , during the decoding process, UR block update can be used for each current block. In some embodiments, for intra-coded frames, the UR block update can be implicitly skipped. For example, the bitstream can include a second current coded block in a frame different from the frame in which the first current coded block is located; the mode of the second current coded block can include intra-frame prediction. In response to determining that the second current coded block is an intra-frame prediction block, the UR block update can be skipped. Skipping can include, for example, not updating the UR frame, or determining whether a field such as UR_BLOCK_UPDATE is set in the header.

[0034] In some embodiments, further referring to Figure 1 , if the UR_BLOCK_UPDATE field is set in the header, the decoder may expect explicit block update signaling bits in the bitstream. In this method, the coded blocks of the video may be signaled in the block header of the video bitstream as the blocks to be updated in the UR frame. In this case, the co-located blocks in the UR frame may be replaced with signaling blocks; for example, the updated UR frame may be used for future motion estimation purposes until the UR frame is subsequently updated.

[0035] Figure 2 FIG. is a process flow diagram of an exemplary embodiment of a process 200 for UR frame block update using implicit signaling, where the UR frame is partially updated by the decoder based on the decoded current block. Implicit signaling may include the case where, compared to explicitly signaling the UR block update mode in the bitstream (e.g., in the header of the bitstream), the current and adjacent block contexts are used to determine which blocks in the UR frame should be updated. This UR block update with implicit signaling can improve prediction and reduce signaling overhead, thereby improving the bitrate and improving compression. Similarly, by performing UR block updates without performing a full UR frame update, some embodiments may reduce the residuals, thereby improving the compression performance.

[0036] At step 205, still referring to Figure 2 , the decoder receives the current block. The current block may be included within the bitstream received by the decoder. The bitstream may include, for example, the data found in the bitstream, which is the input to the decoder when data compression is used. The bitstream may include the information required to decode the video. Receiving may include extracting and / or parsing the blocks and associated signaling information in the bitstream. In some embodiments, the current block may include a coding tree unit (CTU), a coding unit (CU), and / or a prediction unit (PU).

[0037] Continuing to refer to Figure 2 , in some embodiments, still referring to Figure 2 , there may be no signal in the bitstream explicitly sent for the unavailable reference block update mode for the current block. Similarly, in some embodiments, the decoder may use the bitstream to determine whether a signal for the unavailable reference block update mode has been explicitly sent in the bitstream for the current block. In some embodiments, the decoder may determine that a field in the header of the bitstream indicates that the unavailable reference block update mode is disabled or does not exist.

[0038] At step 220, continuing to refer to Figure 2, the current block can be marked to update the UR frame based on the current block; the current block can be marked based on or in response to determining that the current block is implicitly sent a signal for UR frame block update. Marking can include, but is not limited to, setting a variable value to a value indicating that the current block will be used to update the UR frame; for example, a boolean variable associated with the current block and configured to indicate whether the current block is used to update the UR frame can be set to "true" or "1" to indicate marking.

[0039] For example, still referring to Figure 2 , when the current block uses the UR frame as a reference for inter-frame prediction and includes applying illumination compensation, the current block can be marked for update; marking can be performed in response to and / or based on this determination. Illumination compensation can be used to improve prediction in video coding. In cases where block pixels change due to illumination (e.g., clouds passing by reducing illumination), illumination compensation can improve prediction. A block that uses the UR frame as a reference and applies illumination compensation can be determined as an update block and marked accordingly for updating the UR frame.

[0040] As another example, continuing to refer to Figure 2 , when neighboring blocks utilize the UR motion vector reference, based on those neighboring blocks that utilize the UR motion vector reference, the current block can be marked for updating the UR frame. In a representative (e.g., typical) video, a portion of the scene background is updated over time. When such a change occurs, the changed region can be encoded as an intra block when no similar block is found. In such a case, all blocks except the most recently changed region may use the UR frame as a reference. A block with multiple neighboring blocks having UR motion vector references can be marked for implicit UR frame update. Thus, the likelihood that the current block is a UR update block increases as the number of neighboring blocks using the UR frame in the current block increases; this likelihood can be calculated and compared with a threshold number, where a likelihood exceeding the threshold number can indicate that the current block is a UR update block. If the current block is intra-coded, a smaller number of neighboring blocks with UR references can send a signal for UR block update; for example, a smaller threshold number can be used in the threshold comparison as described above. If the current block is inter-coded, a higher number of neighboring blocks with UR references can implicitly send a signal for UR block update, for example, by using a larger threshold number. Thus, as another non-limiting example, in response to and / or based on determining that two or more neighboring blocks of the current block use UR motion vectors for inter-frame prediction, the current block can be marked for updating the UR frame.

[0041] Still referring to Figure 2, in some embodiments, block update candidates may be determined after the entire frame has been decoded, so that the entire block neighborhood can be considered. For example, a block with a non-UR reference that is surrounded by blocks with UR references on two or more sides may be a candidate for UR update. As another example, if the current block is intra-coded and multiple adjacent blocks use a UR frame as a reference and have motion vectors (0, 0) (e.g., zero motion vectors), the current block may be marked for UR update. As yet another example, if the current block is intra-coded and multiple adjacent blocks use a UR frame as a reference and have the same or similar motion vectors (e.g., the motion vectors may differ by less than a predetermined or predefined amount), the current block may be marked for UR update. As yet another example, if the current block has adjacent blocks on at least three sides of the current block that use a UR frame as a reference for inter prediction, the current block may be marked for UR update.

[0042] In some embodiments, continuing to refer to Figure 2 , UR update may also be determined at the coding unit (CU) level. If all adjacent CUs on at least three sides use UR references, all blocks of the current CU may be marked for UR update. In some embodiments, UR update may be determined at the coding tree unit (CTU) level. If all adjacent CTUs on at least three sides use UR references, all blocks of the current CTU that do not use UR references may be marked for update. Figure 3 is a block diagram showing an exemplary embodiment of frame 300 including a current block X and three adjacent blocks, and these three adjacent blocks have associated motion vectors MV1, MV2, MV3 for motion estimation using a UR frame. In the example shown, the current block X may be marked for updating the UR frame.

[0043] Figure 4 is a block diagram showing another exemplary embodiment of frame 400. Frame 400 includes a block marked as X and a block marked as L. The block marked as X does not use a UR frame as a reference, and the block marked as L uses a UR frame as a reference; the block marked as X may be marked for updating the UR frame.

[0044] Referring again to Figure 2, at step 215, the current block can be used to update the UR frame. The update can include performing a UR block update, where the pixels in the UR frame that are spatially co-located with the current block are updated (e.g., modified) using the current block. In some embodiments, the block update mechanism can be explicit or implicit. For example, a portion of the UR frame (e.g., a block) can be updated by updating (e.g., replacing) the co-located pixels (e.g., luminance values) in the UR frame with the pixel values of the current block. In some embodiments, the UR frame can be updated according to another mechanism, such as updating a portion of the UR frame with the average of the initial co-located UR frame pixel values and the current block pixel values. After reviewing the entire content of this disclosure, those skilled in the art will recognize the various other mechanisms that can be employed. In some embodiments, the update can include using multiple decoded blocks to update multiple UR frame blocks, which can be performed using any of the methods described above for updating the UR frame using the decoded current block. The update can include, but is not limited to, using any of the above processes and / or process steps to generate blocks and / or frames with default chrominance and / or luminance values through one or more decoded blocks, and replacing and / or updating the default chrominance and / or luminance values; the update can include generating a UR frame, which can be achieved by creating a UR frame with default values and updating the default values as described above.

[0045] For subsequent current blocks, the updated UR frame can be used as a reference frame for inter-frame prediction. For example, a second encoded block can be received. It can be determined whether the inter-frame prediction mode is enabled for the second encoded block. The updated UR frame can be used as a reference frame, and the second decoded block can be determined according to the inter-frame prediction mode. For example, decoding via inter-frame prediction can include using the updated UR frame as a reference for calculating the prediction, which can be combined with the residuals included in the bitstream.

[0046] Figure 5 is a system block diagram of an exemplary decoder 500 capable of decoding the bitstream 504 using UR frame block updates. The UR frame block updates can include, but are not limited to, explicitly signaling the UR frame block updates, such as but not limited to those referred to above Figure 1 as described. The UR frame block updates can include, but are not limited to, implicitly signaling the UR frame block updates (e.g., UR frame block updates determined according to the context of the current and / or adjacent blocks), such as, for example, as referred to above Figures 2 to 4As described above. The decoder 500 includes an entropy decoding processor 508, an inverse quantization and inverse transformation processor 512, a deblocking filter 516, a frame buffer 520, a motion compensation processor 524, and an intra prediction processor 528. In some embodiments, the bitstream 504 may include a parameter (e.g., a field in the header of the bitstream) that signals a UR frame block update mode. Alternatively or additionally, the bitstream 504 may not explicitly signal the UR frame block update mode for the current block; the bitstream may contain other parameters. The motion compensation processor 524 may use the UR frame to reconstruct pixel information and update the UR frame according to the UR frame block update mode. For example, when the UR frame block update mode is explicitly signaled for the current block, the co-located pixels (e.g., luminance values) in the UR frame may be replaced with the pixel values of the current block. In the case where the block update is implicitly signaled, the motion compensation processor 524 may mark the current block for updating the UR frame (e.g., based on whether the UR frame update is implicitly signaled) and update the UR frame accordingly. For example, when the UR frame block update is implicitly signaled for the current block, the co-located pixels (e.g., luminance values) in the UR frame may be replaced with the pixel values of the current block.

[0047] In operation, the bitstream 504 may be received by the decoder 500 and input to the entropy decoding processor 508, which may entropy decode the bitstream into quantized coefficients. The quantized coefficients are provided to the inverse quantization and inverse transformation processor 512, which may perform inverse quantization and inverse transformation to create a residual signal that may be added to the output of either the motion compensation processor 524 or the intra prediction processor 528 depending on the processing mode. The outputs of the motion compensation processor 524 and the intra prediction processor 528 may include block predictions based on previous decoded blocks or blocks of the UR frame. The sum of the prediction and the residual may be processed by the deblocking filter 516 and stored in the frame buffer 520. For a given block (e.g., CU or PU), when the bitstream 504 explicitly and / or implicitly signals the enabling of the UR frame block update mode, the motion compensation processor 524 may update the UR frame, which may be included in the frame buffer 520, to update the co-located pixels (e.g., luminance values) in the UR frame with the pixel values of the current block.

[0048] In some embodiments, the decoder 500 may include a UR frame block update processor 532, which may generate a UR frame update based on the current block and provide UR frame pixel values for the inter prediction process; this may be implemented according to any of the process steps described in the present disclosure. The UR frame block update processor 532 may directly affect motion compensation. Additionally, for example, when the current block is an intra prediction block, the UR frame update processor 532 may receive information from the intra prediction processor 528.

[0049] Figure 6 is a process flow diagram showing an exemplary embodiment of process 600 for encoding a video with UR frame blocks using explicit signaling according to some aspects of the current subject matter, which can reduce encoding complexity while improving compression efficiency. At step 605, a video frame can undergo an initial block partitioning, for example, using a tree-structured macroblock partitioning scheme that can include dividing an image frame into CTUs and CUs. At step 610, a block for updating a portion of the UR frame can be selected. For example, compared to one or more co-located blocks that are temporally adjacent (e.g., temporally adjacent frames, such as within a predetermined number of frames of the current frame), the block can be selected based on the frequency content and / or metric of the motion within the block. The determination of the metric of the frequency content can include the determination of a transform matrix, such as but not limited to a discrete cosine transform matrix. The determination of the metric of the frequency content of the block can include the determination of a generalized discrete cosine transform matrix. For example, in the case where the above block is a 4×4 pixel block, the generalized discrete cosine transform matrix can include a generalized discrete cosine transform II matrix having the following form:

[0050]

[0051] where α is 1 / 2, b is and C is

[0052] In some embodiments, an integer approximation of the transform matrix can be utilized, which can be used for efficient hardware and software implementations. For example, in the case where the above block is a 4×4 pixel block, the generalized discrete cosine transform matrix can include a generalized discrete cosine transform II matrix having the following form:

[0053]

[0054] For block B i , the following formula can be used to calculate the frequency content of the block:

[0055] F Bi = T x B i x T'.

[0056] where T' is the transpose of the cosine transform matrix T, B i is the block represented as a numerical matrix corresponding to the pixels in the block, such as a 4×4 matrix representing the above 4×4 block, and the operation × represents matrix multiplication. The selection can include identifying, according to a metric rule, the block that will be used to update a portion of the UR frame at the decoder. The metric rule can include a comparison of a metric of texture and / or motion (including but not limited to any metric such as the above frequency content, etc.) with a predetermined and / or stored threshold. As a non-limiting example, in F as defined above BiIn the case of exceeding a certain value, an update can be triggered, for example because very high frequencies may not be perceivable. Alternatively or additionally, any process and / or process step described with reference to Figure 7 can be used to perform the selection. At step 615, the block can be encoded and included in the bitstream.

[0057] At step 620, explicit UR frame block update parameters can be determined and included in the bitstream to signal that the UR frame block update mode is enabled for the current block. For example, a field in the PPS or SPS can be set (e.g., enabled). For example, a field such as the UR_BLOCK_UPDATE field can be set to indicate that the UR frame block update mode is enabled for the current block.

[0058] Figure 7 is a process flow diagram of an exemplary embodiment of a process 700 for encoding video with UR frame block updates using implicit signaling, according to some aspects of the current subject matter. This process can reduce encoding complexity while improving compression efficiency. At step 705, a video frame can undergo an initial block segmentation, for example, segmented using a tree-structured macroblock segmentation scheme that can include dividing the image frame into CTUs and CUs.

[0059] In step 710, still referring to Figure 7 blocks can be selected to update a portion of the UR frame; for example, the blocks can be selected based on implicit UR frame update conditions. This selection can include identifying, according to the conditions, the blocks that will be used to update a portion of the UR frame at the decoder. These conditions can include, for example, when the current block uses the UR frame as a reference for inter-frame prediction and includes applying illumination compensation. As another example, if the current block is inter-frame encoded and two or more adjacent blocks of the current block use UR motion vectors for inter-frame prediction, then these conditions can relate to, for example, which adjacent blocks utilize the UR motion vector reference when the block has multiple adjacent blocks with UR motion vector references. As another example, if a block with a non-UR reference is surrounded by multiple blocks with UR references on two or more sides, this can indicate that the block with the non-UR reference will be used to update a portion of the UR frame. As another example, if the current block is intra-frame encoded, and multiple adjacent blocks use the UR frame as a reference and have a motion vector (0, 0) (e.g., a zero motion vector), then the current block can be identified as a block for updating a portion of the UR frame. As yet another example, if the current block is intra-frame encoded, and multiple adjacent blocks use the UR frame as a reference and have the same or similar motion vectors (e.g., the motion vectors can differ by less than a predetermined or predefined amount), then the current block can be identified as a block for updating the UR frame.

[0060] As another example, continuing to refer to Figure 7, in some embodiments, a block may include a coding unit (CU), and the determination condition may include the following condition, which stipulates that if all adjacent CUs on at least three sides use the UR reference, all blocks of the current CU may be selected. For another example, in some embodiments, the block may include a coding tree unit (CTU) level, and the condition may include that if all adjacent CTUs on at least three sides use the UR reference, all blocks of the current CTU that do not use the UR reference may be selected as the blocks for updating the UR frame.

[0061] At step 730, the selected blocks may be encoded and included in the bitstream. Encoding may include, for example, using an inter-frame prediction mode and an intra-frame prediction mode. In some embodiments, explicit UR frame block update parameters may not be included in the bitstream of the current block.

[0062] Figure 8 is a system block diagram showing an exemplary embodiment of a video encoder 800, which is capable of sending explicit and / or implicit signaling for UR block update on the decoder side. The exemplary video encoder 800 receives an input video 804, and the input video 804 may be initially segmented or partitioned according to a processing scheme such as a tree-structured macroblock segmentation scheme (e.g., quadtree plus binary tree). Examples of tree-structured macroblock segmentation schemes may include dividing an image frame into large block elements called coding tree units (CTUs). In some embodiments, each CTU may be further segmented into multiple sub-blocks called coding units (CUs) one or more times; the result of such segmentation may include a set of sub-blocks called prediction units (PUs). A transform unit (TU) may also be used. The "transform unit" used in the present disclosure is the basic unit on which frequency transformation is applied, and this basic unit may include, but is not limited to, a CU sub-block suitable for frequency transformation operations. As a non-limiting example, a TU may include a transform block of the luminance samples of an image and two corresponding transform blocks of the chrominance samples, as well as a syntax structure for transforming the samples of the transform block, where a "transform block" is defined as a rectangular M×N sample block generated by transformation during the decoding process.

[0063] Still referring to Figure 8 , the exemplary video encoder 800 includes an intra-frame prediction processor 815, a motion estimation / compensation processor 812 (also referred to as an inter-frame prediction processor) capable of supporting explicit and / or implicit UR frame block update at the decoder, a transform / quantization processor 816, an inverse quantization / inverse transform processor 820, a loop filter 824, a decoded image buffer 828, and an entropy coding processor 832. In some embodiments, the motion estimation / compensation processor 812 may, for example, according to the above reference Figure 6and / or one or more of the conditions of 7, determine for the current block that the UR frame should be updated at the decoder; the motion estimation / compensation processor 812 may set parameters to explicitly signal the enabling of the UR frame block update mode. The bitstream parameters signaling the UR frame block update mode may be input to the entropy coding processor 832 to be included in the output bitstream 836. Alternatively or additionally, the motion estimation / compensation processor 812 may update the encoder UR frame with the current block and not include explicit UR frame block update signaling in the bitstream. The bitstream parameters may be input to the entropy coding processor 832 to be included in the output bitstream 836. A portion of the encoder-side UR frame may be updated for use by the motion estimation / compensation processor 812 for encoding additional blocks that may utilize the UR frame as a reference for inter prediction.

[0064] In operation, still referring to Figure 8 , for each block of a frame of the input video 804, it can be determined whether to process the block by intra prediction or using motion estimation / compensation. The block may be provided to the intra prediction processor 808 or the motion estimation / compensation processor 812. If the block is to be processed by intra prediction, the intra prediction processor 808 may perform processing to output a predictor. If the block is to be processed by motion estimation / compensation, the motion estimation / compensation processor 812 may perform processing including using the encoder-side UR frame as a reference for inter prediction (if applicable).

[0065] Still referring to Figure 8 , a residual may be formed by subtracting the predictor from the input video. The residual may be received by the transform / quantization processor 816, and the transform / quantization processor 816 may perform transform processing (e.g., discrete cosine transform (DCT)) to produce coefficients that may be quantized. The quantized coefficients and any associated signaling information may be provided to the entropy coding processor 832 for entropy coding and to be included in the output bitstream 836. The entropy coding processor 832 may support the coding of signaling information related to the UR frame block update mode. Additionally, the quantized coefficients may be provided to the inverse quantization / inverse transform processor 820, and the inverse quantization / inverse transform processor 820 may reproduce pixels that may be combined with the predictor and processed by the loop filter 824, and the output of the loop filter 824 may be stored in the decoded picture buffer 828 for use by the motion estimation / compensation processor 812, where the motion estimation / compensation processor 812 is capable of supporting UR frame block updates at the decoder.

[0066] Although some variations have been described in detail above, other modifications or additions are possible. For example, in some embodiments, the current block may include any symmetric block (8×8, 16×16, 32×32, 64×64, 128×128, etc.) and any asymmetric block (8×4, 16×8, etc.).

[0067] In some embodiments, a quadtree plus binary decision tree (QTBT) may be implemented. For the QTBT, at the coding tree unit level, the splitting parameters of the QTBT can be dynamically derived to adapt to local features without transmitting any overhead. Subsequently, at the coding unit level, the joint classifier decision tree structure can eliminate unnecessary iterations and control the risk of misprediction. In some embodiments, the UR frame block update mode may be available as an additional option at each leaf node of the QTBT. As described above, the current decoded block may be included in the QTBT; for example, but not limited to, the current decoded block may be a non-leaf node of the QTBT.

[0068] In some embodiments, signals of other syntax elements may be sent at different levels of the bitstream. By including an enable flag encoded in the sequence parameter set (SPS), UR frame block updates can be enabled for the entire sequence, where the UR frame block updates may include, but are not limited to, explicit UR frame block updates and / or implicit UR frame block updates. Additionally, a CTU flag may be encoded at the coding tree unit (CTU) level to indicate whether any coding unit (CU) uses the UR frame block update mode. A CU flag may be encoded to indicate whether the current coding unit utilizes the UR frame block update mode. A CTU flag may be encoded at the coding tree unit (CTU) level to indicate whether any coding unit (CU) uses explicit UR frame block update signaling. A CTU flag may be encoded at the coding tree unit (CTU) level to indicate whether any coding unit (CU) uses explicit UR frame block update signaling. Although the above-described disclosed embodiments relate to updates of UR frames, the above-described disclosed embodiments may alternatively or additionally be applied to other frames, images (including but not limited to long-term reference frames).

[0069] The subject matter described herein provides many technical advantages. For example, some embodiments of the current subject matter may provide decoding of blocks using UR frames, which may include updating parts of the UR frame without having to update the entire UR frame. Such an approach can reduce complexity while increasing compression efficiency. Additionally, by implicitly signaling UR block updates, the header overhead can be reduced, thereby reducing the bitrate and improving compression.

[0070] It should be noted that any one or more aspects and embodiments described herein can be conveniently implemented using digital electronic circuits, integrated circuits, application-specific integrated circuits (ASICs) designed specifically, field-programmable gate arrays (FPGAs), computer hardware, firmware, software, and / or combinations of the foregoing, such as implemented and / or executed in one or more machines programmed according to the teachings of this specification (e.g., one or more computing devices serving as user computing devices for electronic documents, one or more server devices such as document servers, etc.), which is obvious to those of ordinary skill in the computer field. These various aspects or features can include implementations in one or more computer programs and / or software executable and / or interpretable on a programmable system, the programmable system including at least one programmable processor, which can be dedicated or general-purpose, and is coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to send data and instructions to the storage system, at least one input device, and at least one output device. It is obvious to those of ordinary skill in the software field that a skilled programmer can easily prepare appropriate software code based on the teachings of this disclosure. The aspects and implementations using software and / or software modules discussed above can also include appropriate hardware for assisting in the implementation of the machine-executable instructions of the software and / or software modules.

[0071] Such software can be a computer program product employing a machine-readable storage medium. A machine-readable storage medium can be any medium capable of storing and / or encoding a sequence of instructions executable by a machine (e.g., a computing device) and causing the machine to execute any of the methods and / or embodiments described herein. Examples of machine-readable storage media include, but are not limited to, magnetic disks, optical discs (e.g., CDs, CD-Rs, DVDs, DVD-Rs, etc.), magneto-optical discs, read-only memory "ROM" devices, random access memory "RAM" devices, magnetic cards, optical cards, solid-state storage devices, EPROMs, EEPROMs, programmable logic devices (PLDs), and / or any combination of the foregoing. The machine-readable media used herein are intended to include a single medium as well as a collection of physically separate media (e.g., a collection of optical discs, or one or more hard disk drives combined with computer memory). The machine-readable storage media used herein do not include transient forms of signal transmission.

[0072] Such software may also include information (e.g., data) carried as a data signal on a data carrier such as a carrier wave. For example, machine-executable information may be included as a data-carrying signal contained in a data carrier, where the signal encodes: a sequence of instructions or portions thereof to be executed by a machine (e.g., a computing device), and any associated information (e.g., data structures and data) that causes the machine to execute any one of the methods and / or embodiments described herein.

[0073] Examples of computing devices include but are not limited to e-reader devices, computer workstations, terminal computers, server computers, handheld devices (e.g., tablet computers, smartphones, etc.), network devices, network routers, network switches, bridges, any machine capable of executing a sequence of instructions specifying actions to be taken by it, and any combination of the foregoing. In one example, a computing device may be included in and / or included within a kiosk.

[0074] Figure 9 FIG. shows an illustration of an embodiment of a computing device in an exemplary form of a computer system 900, in which a set of instructions can be executed to cause a control system to perform any one or more aspects and / or methods of the present disclosure. It is also contemplated that multiple computing devices may be utilized to implement a specifically configured instruction set for causing one or more of the devices to execute any one or more of the aspects and / or methods of the present disclosure. The computer system 900 includes a processor 904 and a memory 908 that communicate with each other and with other components via a bus 912. The bus 912 may include any one of a variety of types of bus structures (including but not limited to a memory bus, a memory controller, a peripheral bus, a local bus, and any combination of the foregoing) using any one of a variety of bus architectures.

[0075] The memory 908 may include various components (e.g., machine-readable media), including but not limited to random access memory components, read-only components, and any combination of the foregoing. In one example, a basic input / output system 916 (BIOS) may be stored in the memory 908, which includes basic routines such as those that help transfer information between elements within the computer system 900 during startup. The memory 908 may also include instructions (e.g., software) 920 embodying any one or more of the aspects and / or methods of the present disclosure (e.g., stored on one or more machine-readable media). In another example, the memory 908 may further include any number of program modules, including but not limited to an operating system, one or more applications, other program modules, program data, and any combination of the foregoing.

[0076] The computer system 900 may also include a storage device 924. Examples of storage devices (e.g., storage device 924) include, but are not limited to, hard disk drives, disk drives, optical disc drives in combination with optical media, solid state storage devices, and any combination of the foregoing. The storage device 924 may be connected to the bus 912 via a suitable interface (not shown). Exemplary interfaces include, but are not limited to, SCSI, High Technology Attachment (ATA), Serial ATA, Universal Serial Bus (USB), IEEE 1394 (FIREWIRE), and any combination of the foregoing. In one example, the storage device 924 (or one or more of its components) may be removably connected to the computer system 900 (e.g., via an external port connector (not shown)). In particular, the storage device 924 and the associated machine-readable medium 928 may provide non-volatile and / or volatile storage of machine-readable instructions, data structures, program modules, and / or other data for the computer system 900. In one example, the software 920 may be stored in whole or in part within the machine-readable medium 928. In another example, the software 920 may be stored in whole or in part within the processor 904.

[0077] The computer system 900 may also include an input device 932. In one example, a user of the computer system 900 may input commands and / or other information into the computer system 900 via the input device 932. Examples of input devices 932 include, but are not limited to: alphanumeric input devices (e.g., keyboards), pointing devices, joysticks, gamepads, audio input devices (e.g., microphones, voice response systems, etc.), cursor control devices (e.g., mice), touchpads, optical scanners, video capture devices (e.g., still cameras, video cameras), touchscreens, and any combination of the foregoing. The input device 932 may be connected to the bus 912 via any one of a variety of interfaces (not shown), the variety of interfaces including, but not limited to, serial interfaces, parallel interfaces, game ports, USB interfaces, FireWire interfaces, direct interfaces to the bus 912, and any combination of the foregoing. The input device 932 may include a touchscreen interface, which may be part of or separate from the display 936, which will be discussed further below. The input device 932 may be used as a user selection device for selecting one or more graphical representations in a graphical interface as described above.

[0078] A user may also input commands and / or other information to the computer system 900 via a storage device 924 (e.g., a removable disk drive, a flash drive, etc.) and / or a network interface device 940. A network interface device such as the network interface device 940 may be used to connect the computer system 900 to one or more of various networks such as a network 944, and to one or more remote devices 948 connected to the network 944. Examples of network interface devices include, but are not limited to, network interface cards (e.g., mobile network interface cards, LAN cards), modems, and any combination thereof. Examples of networks include, but are not limited to, wide area networks (e.g., the Internet, enterprise networks), local area networks (e.g., networks associated with an office, a building, a campus, or other relatively small geographical spaces), telephone networks, data networks associated with telephone / voice providers (e.g., mobile communication provider data and / or voice networks), direct connections between two computing devices, and any combination of the above. Networks such as the network 944 may employ wired and / or wireless communication modes. Generally, any network topology may be used. Information (e.g., data, software 920, etc.) may be transmitted to and / or from the computer system 900 via the network interface device 940.

[0079] The computer system 900 may further include a video display adapter 952 for transmitting a displayable image to a display device such as a display device 936. Examples of display devices include, but are not limited to, liquid crystal displays (LCDs), cathode ray tubes (CRTs), plasma displays, light emitting diode (LED) displays, and any combination of the above. The display adapter 952 and the display device 936 may be used in conjunction with the processor 904 to provide a graphical representation of various aspects of the present disclosure. In addition to the display device, the computer system 900 may include one or more other peripheral output devices, including but not limited to audio speakers, printers, and any combination of the above. Such peripheral output devices may be connected to the bus 912 via a peripheral interface 956. Examples of peripheral interfaces include, but are not limited to, serial ports, USB connections, FireWire connections, parallel connections, and any combination of the above.

[0080] Exemplary embodiments of the present invention have been described in detail above. Various modifications and additions can be made without departing from the gist and scope of the present invention. In order to provide various feature combinations in related new embodiments, the features of each of the above various embodiments can be appropriately combined with the features of other described embodiments. Further, although many individual embodiments have been described above, the content described herein is merely an illustration of the application of the principles of the present invention. Additionally, although a particular method herein may be shown and / or described as being performed in a particular order, in the ordinary technology for implementing the embodiments disclosed herein, the order is highly variable. Accordingly, this description is meant as an example only and is not intended to limit the scope of the present invention.

[0081] In the above description and claims, a list of elements or features may follow phrases such as "at least one" or "one or more". The term "and / or" may also appear in a list of two or more elements or features. Unless implicitly or explicitly contradicted by the context in which the phrase is used, such phrases are intended to mean any of the elements or features listed individually, or any combination of any listed element or feature with any other listed element or feature. For example, the phrases "at least one of A and B", "one or more of A and B", and "A and / or B" are each intended to mean "A alone, B alone, or both A and B". Similar interpretations apply to lists containing three or more items. For example, the phrases "at least one of A, B, and C", "one or more of A, B, and C", "A, B, and / or C" are each intended to mean "A alone, B alone, C alone, both A and B, both A and C, both B and C, or A, B, and C". Additionally, the term "based on" as used above and in the claims is intended to mean "at least partially based on", such that unrecited features or elements are also permitted.

[0082] The subject matter described herein can be embodied in a system, apparatus, method, and / or article according to a desired configuration. The embodiments set forth in the foregoing description do not represent all embodiments consistent with the subject matter described herein. Rather, they are only some examples consistent with aspects related to the subject matter described. Although some variations have been described in detail above, other modifications or additions are possible. In particular, further features and / or variations can be provided in addition to the features and / or variations set forth herein. For example, the above embodiments can be directed to various combinations and sub - combinations of the disclosed features and / or combinations and sub - combinations of several other features disclosed above. Additionally, the logical flows described in the figures and / or herein do not necessarily require the particular order or sequence shown to achieve the desired result. Other embodiments are within the scope of the claims.

Claims

1. A decoder, the decoder comprising circuitry configured to: Receive a bitstream comprising a first current coded block, a second current coded block, and signaling information in a sequence parameter set (SPS); Determine a decoded first current block from the first current coded block; Determine that an unavailable reference block partial update mode is enabled in the SPS of the bitstream; And If the unavailable reference block partial update mode is enabled in the SPS, partially update an unavailable reference frame using the decoded first current block; The decoder is further configured to determine that the second current coded block includes intra prediction, and for the second current coded block, update that the unavailable reference frame is skipped.

2. The decoder according to claim 1, wherein, The first current coded block includes a coding tree unit, a coding unit, or a prediction unit.

3. The decoder according to claim 1, wherein, Updating the unavailable reference frame includes: replacing the luminance values of the unavailable reference frame with the corresponding luminance values of the decoded first current block.

4. The decoder according to claim 1, further comprising: An entropy decoding processor configured to receive the bitstream and decode the bitstream into quantized coefficients; An inverse quantization and inverse transform processor configured to process the quantized coefficients, including performing an inverse discrete cosine; A deblocking filter; A frame buffer; and An intra prediction processor.

5. The decoder according to claim 1, wherein, The decoded first current block forms part of a quadtree binary decision tree.

6. The decoder according to claim 1, wherein, The decoded first current block is a non-leaf node of the quadtree binary decision tree.

7. A method, comprising: Receiving, by a decoder, a bitstream comprising a first current coded block, a second current coded block, and signaling information in a sequence parameter set (SPS); Determining a decoded first current block from the first current coded block; Determining that an unavailable reference block partial update mode is enabled in the SPS of the bitstream; And If the unavailable reference block partial update mode is enabled in the SPS, partially update an unavailable reference frame using the decoded first current block; The method further comprises: Determining that the second current coded block includes intra prediction; And For the second current coded block, updating that the unavailable reference frame is skipped.

8. The method according to claim 7, wherein The first current coded block includes a coding tree unit, a coding unit, or a prediction unit.

9. The method according to claim 7, wherein, Updating the unavailable reference frame includes: replacing the luminance values of the unavailable reference frame with the corresponding luminance values of the decoded first current block.

10. The method according to claim 7, the decoder further comprising: An entropy decoding processor configured to receive the bitstream and decode the bitstream into quantized coefficients; An inverse quantization and inverse transform processor configured to process the quantized coefficients, including performing an inverse discrete cosine; A deblocking filter; A frame buffer; and An intra prediction processor.

11. The method according to claim 7, wherein, The decoded first current block forms part of a quadtree binary decision tree.

12. The method according to claim 7, wherein The decoded first current block is a non-leaf node of the quadtree binary decision tree.

Images