Adaptive block update for unavailable reference frames using explicit and implicit signaling

Abstract

A decoder includes circuitry configured to: receive a current block; determining that the current block is a block for updating an unavailable reference frame based on the current block; and updating the unavailable reference frame using the current block. Related apparatus, systems, techniques, and articles are also described.

Description

[0001] Divisional application

[0002] The parent application of this application is the Chinese patent invention application with application number 2019800780165. The name of the invention of the parent application is "Adaptive block update of unavailable reference frames using explicit and implicit signaling" and the application date is November 27, 2019.

[0003] CROSS-REFERENCE TO RELATED APPLICATIONS

[0004] This application claims priority to U.S. Provisional Patent Application No. 62 / 772,066, filed on November 27, 2018, entitled “ADAPTIVE BLOCK UPDATE OF LONG TERM REFERENCE FRAMES USING EXPLICIT SIGNALING,” the entire contents of which are incorporated herein by reference. This application also claims priority to U.S. Provisional Patent Application No. 62 / 771,941, filed on November 27, 2018, entitled “ADAPTIVE BLOCK UPDATE OF LONG TERM REFERENCE FRAMES USING IMPLICIT SIGNALING,” the entire contents of which are incorporated herein by reference. Technical Field

[0005] The present invention relates generally to the field of video compression and, more particularly, to adaptive block updating for unavailable reference pictures. Background Art

[0006] A video codec can include electronic circuitry or software that compresses or decompresses digital video. It can convert uncompressed video into a compressed format, and vice versa. In the context of video compression, a device that compresses video (and / or performs certain functions thereof) can generally be referred to as an encoder, and a device that decompresses video (and / or performs certain functions thereof) can be referred to as a decoder.

[0007] The format of the compressed data can conform to standard video compression specifications. Compression can be lossy, meaning that the compressed video lacks some information present in the original video. This can result in the decompressed video being of lower quality than the original, uncompressed video, as there is insufficient information to accurately reconstruct the original video.

[0008] There may be a complex relationship between video quality, the amount of data used to represent the video (e.g., determined by the bit rate), the complexity of the encoding and decoding algorithms, sensitivity to data loss and errors, ease of editing, random access, end-to-end latency (e.g., delay time), etc.

[0009] Motion compensation may include methods for predicting a video frame or a portion of a given reference frame (e.g., a previous frame and / or a future frame) by taking into account the motion of the camera and / or objects in the video. It may be used for video compression, encoding and decoding of video data, such as encoding and decoding using the Moving Picture Experts Group (MPEG)-2 (also known as Advanced Video Coding (AVC) and H.264) standard. Motion compensation may describe an image based on a transformation from a reference image to the current image. The reference image may be temporally previous when compared to the current image, or from the future when compared to the current image, or may include a long term reference (LTR) frame. Compression efficiency may be improved when an image can be accurately synthesized based on previously transmitted and / or stored images.

[0010] Current standards such as H.264 and H.265 allow for updating frames such as long-term reference frames by signaling a newly decoded frame that is saved and can be used as a reference frame. This update is signaled by the encoder, and the entire frame is updated. However, updating the entire frame is expensive, especially when only a small portion of the static background has changed. Partial frame updates are possible, but typically involve complex and computationally expensive processes to implement the frame update. Summary of the Invention

[0011] In one aspect, a decoder includes circuitry configured to: receive a bitstream including a current coding block; determine a decoded current block; determine that an unavailable reference block update mode is enabled for the current coding block in the bitstream; and update an unavailable reference frame using the decoded current block.

[0012] In another aspect, a method includes receiving, by a decoder, a bitstream including a current coding block. The method includes determining a decoded current block. The method includes determining that an unavailable reference block update mode is enabled for the current coding block in the bitstream. The method includes updating an unavailable reference frame using the decoded current block.

[0013] In another aspect, a decoder includes circuitry configured to: receive a current block; mark the current block for updating an unavailable reference frame based on the current block; and update the unavailable reference frame using the current block.

[0014] In another aspect, a method includes receiving, by a decoder, a current block, marking the current block for updating an unavailable reference frame based on the current block, and updating the unavailable reference frame using the current block.

[0015] The details of one or more variations of the subject matter described herein are set forth in the following drawings and detailed description. Other features and advantages of the subject matter described herein will be apparent from the detailed description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] For the purpose of illustrating the invention, the accompanying drawings show various aspects of one or more embodiments of the invention. It should be understood, however, that the invention is not limited to the precise arrangements and instrumentalities shown in the drawings, in which:

[0017] Figure 1 is a process flow diagram illustrating an exemplary process for UR frame block updating, wherein the UR frame is partially updated by the decoder based on the decoded current block;

[0018] Figure 2 is a process flow diagram illustrating an exemplary process for UR frame block update using implicit signaling, wherein the UR frame is partially updated by the decoder based on the decoded current block;

[0019] Figure 3 is a block diagram illustrating an exemplary frame including a current block X and three neighboring blocks having associated motion vectors for motion estimation using a UR frame;

[0020] Figure 4 is a block diagram illustrating another exemplary frame including blocks that do not use a UR frame as a reference and blocks that use a UR frame as a reference;

[0021] Figure 5 is a system block diagram illustrating an exemplary decoder capable of decoding a bitstream using UR frame block updates;

[0022] Figure 6 is a process flow diagram illustrating an exemplary process for encoding a video using UR frame block updates according to some aspects of the present subject matter;

[0023] Figure 7 is a process flow diagram illustrating an exemplary process for encoding a video using UR frame block updates, which can reduce encoding complexity while improving compression efficiency, according to some aspects of the present subject matter;

[0024] Figure 8 is a system block diagram illustrating an exemplary video encoder capable of signaling decoder-side UR block updates; and

[0025] Figure 9 is a block diagram of a computing system that can be used to implement any one or more of the methods disclosed herein, or any one or more portions thereof.

[0026] The accompanying drawings are not necessarily drawn to scale and may be illustrated using dashed lines, schematic diagrams, and fragmentary views. In some cases, details that are not necessary for understanding the embodiments or that make other details difficult to understand may have been omitted. Like reference numerals in the various drawings represent like elements. DETAILED DESCRIPTION

[0027] Some embodiments of the current subject matter include methods for performing partial updates of unavailable reference (UR) frames at the decoder side. Unavailable reference (UR) frames are frames and / or images that are used to create predicted frames and / or images in one or more groups of pictures (GOPs), but are not themselves displayed in the video. Frames marked as UR frames in a video bitstream can be used as references until they are explicitly removed by bitstream signaling. UR frames can improve prediction and compression efficiency in scenes with a static background for extended periods of time, such as those in video conferences or parking lot surveillance. However, over time, the background of a scene gradually changes (for example, a car becomes part of the background scene while parked in an empty space). Partial UR frame updates can be explicitly signaled in the bitstream by utilizing an unavailable reference block update mode; when enabled for a given current block, the unavailable reference block update mode can indicate that, after decoding, the current block should be used to update spatially co-located pixels within the UR frame. This partial UR frame update can improve prediction without requiring an entire UR frame update, which can reduce residual errors and thus improve compression performance. Furthermore, by explicitly signaling UR block updates, UR updates can be implemented simply without utilizing inefficient and complex procedures.

[0028] Alternatively or additionally, partial UR frame updates can be implicitly signaled, eliminating or reducing explicit signaling in the bitstream compared to some alternative approaches. This implicit signaling approach can indicate that, after decoding, the current block is used to update spatially co-located pixels within the UR frame. This partial UR frame update can improve prediction without requiring an entire UR frame update, which can reduce residuals and thus improve compression performance. Furthermore, by implicitly signaling UR block updates, header overhead can be reduced, thereby lowering bitrate and improving compression. Furthermore, in some embodiments, UR updates can be implemented simply, without requiring the use of inefficient and complex processes.

[0029] In some embodiments, implicit signaling of video blocks identifies situations where current and neighboring block contexts are used to implicitly determine which blocks are updated in a UR frame (e.g., which portions of a UR frame should be updated). Figure 1, which provides a process flow diagram illustrating an exemplary process 100 for UR frame block updating using explicit signaling, wherein a UR frame is partially updated by the decoder based on the decoded current block. This UR block update can improve prediction without requiring an entire UR frame update, which can reduce residual errors and thus improve compression performance.

[0030] At step 105, still referring to Figure 1 , a decoder receives a bitstream including a current coding block. The bitstream may, for example, include data found in a bitstream that is input to the decoder when data compression is used. The bitstream may include information required to decode the video. Receiving may include extracting and / or parsing blocks in the bitstream and associated signaling information. In some embodiments, the current coding block may include a coding tree unit (CTU), a coding unit (CU), or a prediction unit (PU).

[0031] At step 110, continue with reference to Figure 1 , the decoded current block can be determined. For example, the received current coded block can be decoded, for example, by using inter-frame prediction. Decoding via inter-frame prediction may include using a previous frame, a future frame, or a UR frame as a reference for calculating the prediction, which may be combined with a residual included in the bitstream.

[0032] At step 115, continue with reference to Figure 1 , it can be determined whether the unavailable reference block update mode is enabled for the current coding block in the bitstream. The UR block update mode can be included in the header of the bitstream and / or a field included in the header. For example, it can be determined that the UR block update mode field in the header of the bitstream is enabled. The signaling of UR frame block updates can be selectively enabled using headers such as picture parameter set (PPS) and / or sequence parameter set (SPS). Fields such as UR_BLOCK_UPDATE can take true or false values (e.g., 0 or 1), but are not limited to this.

[0033] Still refer to Figure 1If the UR block update mode is enabled (e.g., if the UR_BLOCK_UPDATE field in the header is set), then at 120, the decoder may use the decoded current block to update the unavailable reference frame. This updating may include performing a UR block update, wherein the decoded current block is used to update (e.g., modify) pixels in the UR frame that are spatially co-located with the current block for which the UR frame block update mode is enabled. In some embodiments, the block update mechanism may be explicit or implicit. For example, a portion of the UR frame (e.g., a block) may be updated by updating co-located pixels (e.g., luma values) in the UR frame using pixel values of the decoded current block; in other words, updating the UR frame may include replacing the luma values of the UR frame with the luma values that are spatially co-located with the decoded current block. In some embodiments, the UR frame may be updated according to another mechanism, such as updating a portion of the UR frame with the average of the initial co-located UR frame pixel values and the currently decoded block pixel values. After reviewing the entire disclosure, one skilled in the art will recognize various other mechanisms that may be employed. In some embodiments, updating may include updating multiple UR frame blocks using multiple decoded blocks, which may be performed using any of the methods described above for updating a UR frame using a decoded current block. Updating may include, but is not limited to, generating a block and / or frame with default chroma and / or luma values using one or more decoded blocks using any of the processes and / or process steps described above, and replacing and / or updating the default chroma and / or luma values; updating may include generating a UR frame, which may be achieved by creating a UR frame with default values and updating the default values as described above.

[0034] Continue to refer Figure 1 , for a subsequent current block, the updated UR frame may be used as a reference frame for inter-frame prediction. For example, a second coded block may be received. It may be determined whether inter-frame prediction mode is enabled for the second coded block. The updated UR frame may be used as a reference frame, and a second decoded block may be determined based on the inter-frame prediction mode. For example, decoding via inter-frame prediction may include using the updated UR frame as a reference for calculating a prediction, which may be combined with a residual included in the bitstream.

[0035] Still refer to Figure 1 , during the decoding process, a UR block update may be applied to each current block. In some embodiments, for intra-coded frames, a UR block update may be implicitly skipped. For example, a bitstream may include a second current coded block in a frame different from the frame in which the first current coded block is located; the mode of the second current coded block may include intra-frame prediction. In response to determining that the second current coded block is an intra-frame predicted block, the UR block update may be skipped. Skipping may include, for example, not updating the UR frame, or determining whether a field such as UR_BLOCK_UPDATE is set in the header.

[0036] In some embodiments, further reference is made to Figure 1 If the UR_BLOCK_UPDATE field is set in the header, the decoder can expect explicit block update signaling bits in the bitstream. In this approach, a coded block of the video can be signaled in the block header of the video bitstream as a block to be updated in the UR frame. In this case, the co-located block in the UR frame can be replaced with a signaling block; for example, the updated UR frame can be used for future motion estimation purposes until the UR frame is subsequently updated.

[0037] Figure 2 is a process flow diagram illustrating an exemplary embodiment of a process 200 for UR frame block updates using implicit signaling, wherein a UR frame is partially updated by the decoder based on the decoded current block. Implicit signaling can include situations where, compared to explicitly signaling the UR block update mode in the bitstream (e.g., in the bitstream header), the current and neighboring block context is used to determine which blocks in the UR frame should be updated. This UR block update with implicit signaling can improve prediction and reduce signaling overhead, thereby improving bitrate and compression. Similarly, by performing UR block updates without performing an entire UR frame update, some embodiments can reduce residual errors, thereby improving compression performance.

[0038] At step 205, still referring to Figure 2 , the decoder receives the current block. The current block may be contained in a bitstream received by the decoder. The bitstream may, for example, include data found in a bitstream that is input to the decoder when data compression is used. The bitstream may include information required to decode the video. Receiving may include extracting and / or parsing blocks in the bitstream and associated signaling information. In some embodiments, the current block may include a coding tree unit (CTU), a coding unit (CU), and / or a prediction unit (PU).

[0039] Continue to refer Figure 2 , in some embodiments, still referring to Figure 2 , the unavailable reference block update mode may not be explicitly signaled in the bitstream for the current block. Similarly, in some embodiments, the decoder may use the bitstream to determine whether the unavailable reference block update mode is explicitly signaled in the bitstream for the current block. In some embodiments, the decoder may determine that a field in the header of the bitstream indicates that the unavailable reference block update mode is disabled or does not exist.

[0040] At step 220, continue with reference to Figure 2The current block may be marked to update the UR frame based on the current block; the current block may be marked based on or in response to determining that the current block is implicitly signaled for UR frame block update. The marking may include, but is not limited to, setting a variable value to a value indicating that the current block will be used to update the UR frame; for example, a Boolean variable associated with the current block and configured to indicate whether the current block is used to update the UR frame may be set to "true" or "1" to indicate the marking.

[0041] For example, still referring to Figure 2 When the current block uses the UR frame as a reference for inter-frame prediction and includes applying illumination compensation, the current block may be marked for updating; marking may be performed responsive to and / or based on the determination. Illumination compensation may be used to improve prediction in video coding. Illumination compensation may improve prediction when block pixels change due to illumination (e.g., a cloud passing by reduces illumination). Blocks that use the UR frame as a reference and apply illumination compensation may be determined to be update blocks and marked accordingly for updating the UR frame.

[0042] For another example, continue to refer to Figure 2 When neighboring blocks utilize UR motion vector references, the current block can be marked for UR frame update based on those neighboring blocks that utilize UR motion vector references. In representative (e.g., typical) videos, portions of the scene background update over time. When such changes occur, the changed region can be encoded as an intra block if no similar blocks are found. In such cases, all blocks except the most recently changed region may use the UR frame as a reference. Blocks with multiple neighboring blocks that reference the UR motion vector can be marked for implicit UR frame update. Therefore, the likelihood of the current block being a UR update block increases as the number of neighboring blocks in the current block that use the UR frame increases. This likelihood can be calculated and compared to a threshold value, where a likelihood exceeding the threshold value can indicate that the current block is a UR update block. If the current block is intra-coded, a smaller number of neighboring blocks with UR references can signal a UR block update; for example, a smaller threshold value can be used in the threshold comparison described above. If the current block is inter-coded, a higher number of neighboring blocks with UR references can implicitly signal a UR block update, for example, by using a larger threshold value. Thus, as another non-limiting example, in response to and / or based on determining that two or more neighboring blocks of the current block use the UR motion vector for inter prediction, the current block may be marked for updating the UR frame.

[0043] Still refer to Figure 2In some embodiments, block update candidates can be determined after the entire frame has been decoded, so that the entire block neighborhood can be considered. For example, a block with a non-UR reference that is surrounded on two or more sides by blocks with a UR reference can be a candidate for a UR update. For another example, if the current block is intra-coded and multiple neighboring blocks use a UR frame as a reference and have a motion vector of (0, 0) (e.g., a zero motion vector), the current block can be marked for a UR update. For another example, if the current block is intra-coded and multiple neighboring blocks use a UR frame as a reference and have the same or similar motion vectors (e.g., the motion vectors may differ by less than a predetermined or predefined amount), the current block can be marked for a UR update. For another example, if the current block has neighboring blocks on at least three sides of the current block that use a UR frame as a reference for inter prediction, the current block can be marked for a UR update.

[0044] In some embodiments, continue with reference to Figure 2 UR updates can also be determined at the coding unit (CU) level. If all neighboring CUs on at least three sides use UR references, all blocks of the current CU can be marked for UR updates. In some embodiments, UR updates can be determined at the coding tree unit (CTU) level. If all neighboring CTUs on at least three sides use UR references, all blocks of the current CTU that do not use UR references can be marked for updates. Figure 3 is a block diagram illustrating an exemplary embodiment of a frame 300 including a current block X and three neighboring blocks having associated motion vectors MV1, MV2, MV3 for motion estimation using a UR frame. In the example shown, the current block X may be marked for updating the UR frame.

[0045] Figure 4 is a block diagram illustrating another exemplary embodiment of a frame 400, which includes blocks marked as X and blocks marked as L, where the blocks marked as X do not use the UR frame as a reference and the blocks marked as L use the UR frame as a reference; the blocks marked as X can be marked for updating the UR frame.

[0046] Reference again Figure 2At step 215, the UR frame may be updated using the current block. The update may include performing a UR block update, wherein the current block is used to update (e.g., modify) pixels in the UR frame that are spatially co-located with the current block. In some embodiments, the block update mechanism may be explicit or implicit. For example, a portion (e.g., a block) of the UR frame may be updated by updating (e.g., replacing) co-located pixels (e.g., luminance values) in the UR frame with pixel values of the current block. In some embodiments, the UR frame may be updated according to another mechanism, such as updating a portion of the UR frame using an average of initially co-located UR frame pixel values and current block pixel values. After reviewing the entire disclosure of this disclosure, one skilled in the art will recognize various other mechanisms that may be employed. In some embodiments, the update may include updating multiple UR frame blocks using multiple decoded blocks, which may be performed using any of the above-described methods for updating a UR frame using a decoded current block. The updating may include, but is not limited to, generating blocks and / or frames having default chroma and / or luminance values by one or more decoded blocks using any of the above processes and / or process steps, and replacing and / or updating the default chroma and / or luminance values; the updating may include generating a UR frame, which may be achieved by creating a UR frame having default values and updating the default values as described above.

[0047] For a subsequent current block, the updated UR frame can be used as a reference frame for inter-frame prediction. For example, a second coded block can be received. It can be determined whether inter-frame prediction mode is enabled for the second coded block. The updated UR frame can be used as a reference frame, and a second decoded block can be determined based on the inter-frame prediction mode. For example, decoding via inter-frame prediction can include using the updated UR frame as a reference for calculating a prediction, which can be combined with a residual included in the bitstream.

[0048] Figure 5 5 is a system block diagram illustrating an exemplary decoder 500 capable of decoding a bitstream 504 using UR frame block updates. UR frame block updates may include, but are not limited to, explicitly signaling UR frame block updates, such as, but not limited to, the above-referenced Figure 1 The UR frame block update may include, but is not limited to, implicitly sending a signal for a UR frame block update (e.g., a UR frame block update determined based on the context of the current and / or neighboring blocks), for example, as described above with reference to Figures 2 to 4The decoder 500 includes an entropy decoding processor 508, an inverse quantization and inverse transform processor 512, a deblocking filter 516, a frame buffer 520, a motion compensation processor 524, and an intra-frame prediction processor 528. In some embodiments, the bitstream 504 may include parameters (e.g., a field in the bitstream header) that signal a UR frame block update mode. Alternatively or additionally, the bitstream 504 may not explicitly signal the UR frame block update mode for the current block; the bitstream may include other parameters. The motion compensation processor 524 may use the UR frame to reconstruct pixel information and update the UR frame according to the UR frame block update mode. For example, when the UR frame block update mode is explicitly signaled for the current block, the co-located pixels (e.g., luminance values) in the UR frame may be replaced with the pixel values of the current block. In the case where a block update is implicitly signaled, the motion compensation processor 524 may mark the current block for updating the UR frame (e.g., based on whether the UR frame update is implicitly signaled) and update the UR frame accordingly. For example, when a UR frame block update is implicitly signaled for a current block, the co-located pixels (eg, luma values) in the UR frame may be replaced with pixel values of the current block.

[0049] In operation, a bitstream 504 may be received by the decoder 500 and input to an entropy decoding processor 508, which may entropy decode the bitstream into quantized coefficients. The quantized coefficients are provided to an inverse quantization and inverse transform processor 512, which may perform inverse quantization and inverse transform to create a residual signal. The residual signal may be added to the output of a motion compensation processor 524 or an intra-frame prediction processor 528, depending on the processing mode. The output of the motion compensation processor 524 and the intra-frame prediction processor 528 may include a block prediction based on a previously decoded block or UR frame. The sum of the prediction and the residual may be processed by a deblocking filter 516 and stored in a frame buffer 520. For a given block (e.g., a CU or PU), when the bitstream 504 explicitly and / or implicitly signals the activation of a UR frame block update mode, the motion compensation processor 524 may update the UR frame, which may be contained in the frame buffer 520, to update the co-located pixels (e.g., luma values) in the UR frame with the pixel values of the current block.

[0050] In some embodiments, the decoder 500 may include a UR frame block update processor 532 that can generate a UR frame update based on the current block and provide UR frame pixel values for the inter-frame prediction process; this can be implemented according to any of the process steps described in this disclosure. The UR frame block update processor 532 can directly affect motion compensation. In addition, the UR frame update processor 532 can receive information from the intra-frame prediction processor 528, for example, when the current block is an intra-frame predicted block.

[0051] Figure 6 6 is a process flow diagram illustrating an exemplary embodiment of a process 600 for encoding a video using UR frame block updates using explicit signaling, according to some aspects of the present subject matter, which can reduce coding complexity while improving compression efficiency. At step 605, the video frame can undergo initial block partitioning, for example, using a tree-structured macroblock partitioning scheme, which can include dividing the image frame into CTUs and CUs. At step 610, a block can be selected for updating a portion of the UR frame. For example, the block can be selected based on frequency content and / or a measure of motion within the block compared to one or more temporally adjacent co-located blocks (e.g., temporally adjacent frames, such as within a predetermined number of frames of the current frame). Determining the measure of frequency content can include determining using a transform matrix, such as, but not limited to, a discrete cosine transform matrix. Determining the measure of frequency content of the block can include determining using a generalized discrete cosine transform matrix. For example, where the block is a 4×4 pixel block, the generalized discrete cosine transform matrix can include employing a generalized discrete cosine transform II matrix having the following form:

[0052]

[0053] Among them, α is 1 / 2, b is and C is

[0054] In some embodiments, integer approximations of the transform matrix may be utilized, which may allow for efficient hardware and software implementations. For example, where the block is a 4×4 pixel block, the generalized discrete cosine transform matrix may include employing a generalized discrete cosine transform II matrix having the following form:

[0055]

[0056] For block B i , the frequency content of the block can be calculated using the following formula:

[0057] F Bi =T x B i x T'.

[0058] Where T' is the transpose of the cosine transform matrix T, B i is a block represented as a matrix of values corresponding to the pixels in the block, for example a 4×4 matrix representing the 4×4 block described above, and the operation × represents a matrix multiplication. The selection may include identifying a block to be used to update a portion of the UR frame at the decoder according to a metric rule. The metric rule may include a comparison of a metric of texture and / or motion (including but not limited to any metric of frequency content described above) with a predetermined and / or stored threshold. As a non-limiting example, in the case of F as defined above BiIf a certain value is exceeded, an update may be triggered, for example because very high frequencies may not be perceived. Alternatively or additionally, the following reference may be used Figure 7 The selection may be performed by any of the processes and / or process steps described in

[0066] At step 615, the block may be encoded and included in a bitstream.

[0059] At step 620, explicit UR frame block update parameters may be determined and included in the bitstream to signal that the UR frame block update mode is enabled for the current block. For example, a field in the PPS or SPS may be set (e.g., enabled). For example, a field such as the UR_BLOCK_UPDATE field may be set to indicate that the UR frame block update mode is enabled for the current block.

[0060] Figure 7 7 is a process flow diagram illustrating an exemplary embodiment of a process 700 for encoding a video using implicit signaling with UR frame block updates, which can reduce coding complexity while improving compression efficiency, according to some aspects of the present subject matter. At step 705, the video frame can undergo initial block partitioning, for example, using a tree-structured macroblock partitioning scheme, which can include partitioning the image frame into CTUs and CUs.

[0061] In step 710, still referring to Figure 7 A block may be selected to update a portion of a UR frame; for example, the block may be selected based on implicit UR frame update conditions. This selection may include identifying a block to be used to update a portion of the UR frame at the decoder based on conditions. These conditions may include, for example, when the current block uses a UR frame as a reference for inter prediction and when illumination compensation is applied. For another example, if the current block is inter-coded and two or more neighboring blocks of the current block use a UR motion vector for inter prediction, the conditions may relate to which neighboring blocks utilize the UR motion vector reference, for example, when the block has multiple neighboring blocks with UR motion vector references. For another example, if a block with non-UR references is surrounded on two or more sides by multiple blocks with UR references, this may indicate that the block with non-UR references will be used to update a portion of the UR frame. For another example, if the current block is intra-coded and multiple neighboring blocks use a UR frame as a reference and have a motion vector of (0, 0) (e.g., a zero motion vector), the current block may be identified as a block to be used to update a portion of the UR frame. For another example, if the current block is intra-coded and multiple neighboring blocks use the UR frame as a reference and have the same or similar motion vectors (e.g., the motion vectors may differ by less than a predetermined or predefined amount), the current block may be identified as a block for updating the UR frame.

[0062] For another example, continue to refer to Figure 7In some embodiments, the block may include a coding unit (CU), and the determination condition may include the following condition, which stipulates that if all neighboring CUs on at least three sides use UR references, then all blocks of the current CU may be selected. For another example, in some embodiments, the block may include a coding tree unit (CTU) level, and the condition may include that if all neighboring CTUs on at least three sides use UR references, then all blocks of the current CTU that do not use UR references may be selected as blocks for updating the UR frame.

[0063] At step 730, the selected block may be encoded and included in the bitstream. Encoding may, for example, include utilizing inter-frame prediction mode and intra-frame prediction mode. In some embodiments, explicit UR frame block update parameters may not be included in the bitstream for the current block.

[0064] Figure 8 is a system block diagram illustrating an exemplary embodiment of a video encoder 800 capable of sending explicit and / or implicit signaling for decoder-side UR block updates. The exemplary video encoder 800 receives an input video 804, which may be initially partitioned or divided according to a processing scheme such as a tree-structured macroblock partitioning scheme (e.g., a quadtree plus a binary tree). Examples of tree-structured macroblock partitioning schemes may include dividing an image frame into large block elements called coding tree units (CTUs). In some embodiments, each CTU may be further partitioned one or more times into multiple sub-blocks called coding units (CUs); the result of such partitioning may include a set of sub-blocks called prediction units (PUs). Transform units (TUs) may also be used. A "transform unit" as used in the present disclosure is a basic unit on which a frequency transform is applied, which may include, but is not limited to, CU sub-blocks suitable for frequency transform operations. As a non-limiting example, a TU may include a transform block of luma samples and two corresponding transform blocks of chroma samples of an image, and a syntax structure for transforming the transform block samples, where a "transform block" is defined as a rectangular M×N block of samples resulting from a transform in the decoding process.

[0065] Still refer to Figure 8 , the exemplary video encoder 800 includes an intra-frame prediction processor 815, a motion estimation / compensation processor 812 (also referred to as an inter-frame prediction processor) capable of supporting explicit and / or implicit UR frame block updates at the decoder, a transform / quantization processor 816, an inverse quantization / inverse transform processor 820, a loop filter 824, a decoded picture buffer 828, and an entropy coding processor 832. In some embodiments, the motion estimation / compensation processor 812 may be, for example, based on the above reference Figure 6and / or one or more conditions described in 7, determines that the UR frame should be updated at the decoder for the current block; the motion estimation / compensation processor 812 can set parameters to explicitly signal that the UR frame block update mode is enabled. The bitstream parameters that signal the UR frame block update mode can be input to the entropy coding processor 832 for inclusion in the output bitstream 836. Alternatively or additionally, the motion estimation / compensation processor 812 can update the encoder UR frame with the current block and not include explicit UR frame block update signaling in the bitstream. The bitstream parameters can be input to the entropy coding processor 832 for inclusion in the output bitstream 836. A portion of the encoder-side UR frame can be updated for use by the motion estimation / compensation processor 812 for encoding additional blocks that can use the UR frame as a reference for inter-frame prediction.

[0066] While operating, continue to refer to Figure 8 For each block of a frame of the input video 804, it can be determined whether the block is to be processed by intra-image prediction or using motion estimation / compensation. The block can be provided to an intra-frame prediction processor 808 or a motion estimation / compensation processor 812. If the block is to be processed by intra-frame prediction, the intra-frame prediction processor 808 can perform processing to output a predictor. If the block is to be processed by motion estimation / compensation, the motion estimation / compensation processor 812 can perform processing including using the encoder-side UR frame as a reference for inter-frame prediction (if applicable).

[0067] Still refer to Figure 8 , a residual can be formed by subtracting the prediction factor from the input video. The residual can be received by the transform / quantization processor 816, which can perform a transform process (e.g., a discrete cosine transform (DCT)) to produce coefficients that can be quantized. The quantized coefficients and any associated signaling information can be provided to the entropy coding processor 832 for entropy coding and included in the output bitstream 836. The entropy coding processor 832 can support encoding signaling information related to the UR frame block update mode. In addition, the quantized coefficients can be provided to the inverse quantization / inverse transform processor 820, which can reproduce pixels, which can be combined with the prediction factor and processed by the loop filter 824, the output of which can be stored in the decoded picture buffer 828 for use by the motion estimation / compensation processor 812, wherein the motion estimation / compensation processor 812 can support UR frame block updates at the decoder.

[0068] Although some variations have been described in detail above, other modifications or additions are possible. For example, in some embodiments, the current block may include any symmetric block (8×8, 16×16, 32×32, 64×64, 128×128, etc.) and any asymmetric block (8×4, 16×8, etc.).

[0069] In some embodiments, a quadtree plus binary decision tree (QTBT) can be implemented. For QTBT, at the coding tree unit level, the segmentation parameters of the QTBT can be dynamically derived to adapt to local features without transmitting any overhead. Subsequently, at the coding unit level, the joint classifier decision tree structure can eliminate unnecessary iterations and control the risk of incorrect predictions. In some embodiments, the UR frame block update mode can be an additional option available at each leaf node of the QTBT. As described above, the current decoding block can be included in the QTBT; for example, but not limited to, the current decoding block can be a non-leaf node of the QTBT.

[0070] In some embodiments, other syntax elements may be signaled at different hierarchical levels of the bitstream. UR frame block updates may be enabled for the entire sequence by including an enable flag encoded in a sequence parameter set (SPS), where the UR frame block updates may include, but are not limited to, explicit UR frame block updates and / or implicit UR frame block updates. Furthermore, a coding tree unit (CTU) flag may be encoded at the CTU level to indicate whether any coding unit (CU) utilizes the UR frame block update mode. A CU flag may be encoded to indicate whether the current coding unit utilizes the UR frame block update mode. A CTU flag may be encoded at the CTU level to indicate whether any coding unit (CU) utilizes explicit UR frame block update signaling. A CTU flag may be encoded at the CTU level to indicate whether any coding unit (CU) utilizes explicit UR frame block update signaling. While the disclosed embodiments have been described with respect to updates to UR frames, the disclosed embodiments may alternatively or additionally be applied to other frames, pictures, including, but not limited to, long-term reference frames.

[0071] The subject matter described herein provides numerous technical advantages. For example, some embodiments of the present subject matter may provide for decoding blocks using UR frames, which may include updating portions of UR frames without having to update the entire UR frame. Such an approach can reduce complexity while increasing compression efficiency. Furthermore, by implicitly signaling UR block updates, header overhead can be reduced, thereby lowering bitrate and improving compression.

[0072] It should be noted that any one or more aspects and embodiments described herein can be conveniently implemented using digital electronic circuits, integrated circuits, specially designed application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), computer hardware, firmware, software, and / or combinations thereof, as implemented and / or implemented in one or more machines programmed according to the teachings of this specification (e.g., one or more computing devices used as user computing devices for electronic documents, one or more server devices such as document servers, etc.), as will be apparent to one of ordinary skill in the computer arts. These various aspects or features can include implementation in one or more computer programs and / or software executable and / or interpretable on a programmable system, the programmable system comprising at least one programmable processor, which can be either dedicated or general purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to send data and instructions to the storage system, at least one input device, and at least one output device. It will be apparent to one of ordinary skill in the software arts that a skilled programmer can readily prepare appropriate software coding based on the teachings of this disclosure. The aspects and embodiments discussed above that employ software and / or software modules may also include appropriate hardware to facilitate implementation of the machine-executable instructions of the software and / or software modules.

[0073] This software can be a computer program product using a machine-readable storage medium. A machine-readable storage medium can be an instruction sequence that can be stored and / or encoded and executed by a machine (e.g., a computing device), and causes the machine to execute any medium of any one of the methods and / or embodiments described herein. The example of a machine-readable storage medium includes, but is not limited to, a disk, an optical disc (e.g., CD, CD-R, DVD, DVD-R, etc.), a magneto-optical disc, a read-only memory "ROM" device, a random access memory "RAM" device, a magnetic card, an optical card, a solid-state storage device, an EPROM, an EEPROM, a programmable logic device (PLD), and / or any combination of the above items. The machine-readable medium used herein is intended to include a single medium and a set of physically separated media (e.g., a set of optical discs, or one or more hard disk drives combined with a computer memory). The machine-readable storage medium used herein does not include a temporary form of signal transmission.

[0074] Such software may also include information (e.g., data) carried as a data signal on a data carrier such as a carrier wave. For example, machine-executable information may be included as a data-bearing signal embodied in a data carrier, where the signal encodes a sequence of instructions, or portion thereof, for execution by a machine (e.g., a computing device), and any associated information (e.g., data structures and data) that causes the machine to perform any of the methods and / or embodiments described herein.

[0075] Examples of computing devices include, but are not limited to, electronic book reading devices, computer workstations, terminal computers, server computers, handheld devices (e.g., tablet computers, smartphones, etc.), network devices, network routers, network switches, network bridges, any machine capable of executing a sequence of instructions specifying actions to be taken, and any combination thereof. In one example, the computing device may include and / or be included in a kiosk.

[0076] Figure 9 A diagram of one embodiment of a computing device in the exemplary form of a computer system 900 is shown, wherein a set of instructions for causing a control system to perform any one or more aspects and / or methods of the present disclosure may be executed. It is also contemplated that a plurality of computing devices may be utilized to implement a specially configured instruction set for causing one or more of the devices to perform any one or more aspects and / or methods of the present disclosure. The computer system 900 includes a processor 904 and a memory 908 that communicate with each other and with other components via a bus 912. The bus 912 may include any of a variety of types of bus structures (including but not limited to a memory bus, a memory controller, a peripheral bus, a local bus, and any combination thereof) using any of a variety of bus architectures.

[0077] The memory 908 may include various components (e.g., machine-readable media), including, but not limited to, random access memory components, read-only components, and any combination thereof. In one example, a basic input / output system 916 (BIOS) including basic routines such as those that help transfer information between components within the computer system 900 during startup may be stored in the memory 908. The memory 908 may also include instructions (e.g., software) 920 (e.g., stored on one or more machine-readable media) that embody any one or more of the aspects and / or methods of the present disclosure. In another example, the memory 908 may further include any number of program modules, including, but not limited to, an operating system, one or more application programs, other program modules, program data, and any combination thereof.

[0078] The computer system 900 may also include a storage device 924. Examples of storage devices (e.g., storage device 924) include, but are not limited to, hard disk drives, disk drives, optical disk drives combined with optical media, solid-state storage devices, and any combination thereof. The storage device 924 may be connected to the bus 912 via an appropriate interface (not shown). Exemplary interfaces include, but are not limited to, SCSI, Advanced Technology Attachment (ATA), Serial ATA, Universal Serial Bus (USB), IEEE 1394 (FIREWIRE), and any combination thereof. In one example, the storage device 924 (or one or more of its components) may be removably connected to the computer system 900 (e.g., via an external port connector (not shown)). In particular, the storage device 924 and associated machine-readable media 928 may provide non-volatile and / or volatile storage of machine-readable instructions, data structures, program modules, and / or other data for the computer system 900. In one example, the software 920 may be stored in whole or in part in the machine-readable media 928. In another example, the software 920 may be stored in whole or in part in the processor 904.

[0079] The computer system 900 may also include an input device 932. In one example, a user of the computer system 900 can enter commands and / or other information into the computer system 900 via the input device 932. Examples of the input device 932 include, but are not limited to, an alphanumeric input device (e.g., a keyboard), a pointing device, a joystick, a game controller, an audio input device (e.g., a microphone, a voice response system, etc.), a cursor control device (e.g., a mouse), a touchpad, an optical scanner, a video capture device (e.g., a still camera, a video camera), a touch screen, and any combination thereof. The input device 932 may be connected to the bus 912 via any of a variety of interfaces (not shown), including, but not limited to, a serial interface, a parallel interface, a game port, a USB interface, a FireWire interface, a direct interface to the bus 912, and any combination thereof. The input device 932 may include a touch screen interface, which may be part of or separate from the display 936, as will be discussed further below. The input device 932 may be used as a user selection device for selecting one or more graphical representations in the graphical interface as described above.

[0080] The user can also input commands and / or other information to the computer system 900 via a storage device 924 (e.g., a removable disk drive, a flash drive, etc.) and / or a network interface device 940. A network interface device such as the network interface device 940 can be used to connect the computer system 900 to one or more of various networks such as the network 944, as well as one or more remote devices 948 connected to the network 944. Examples of network interface devices include, but are not limited to, a network interface card (e.g., a mobile network interface card, a LAN card), a modem, and any combination thereof. Examples of networks include, but are not limited to, a wide area network (e.g., the Internet, an enterprise network), a local area network (e.g., a network associated with an office, a building, a campus, or other relatively small geographic space), a telephone network, a data network associated with a telephone / voice provider (e.g., a mobile communications provider data and / or voice network), a direct connection between two computing devices, and any combination thereof. A network such as the network 944 can employ wired and / or wireless communication modes. Generally, any network topology can be used. Information (eg, data, software 920 , etc.) can be transferred to and / or from computer system 900 via network interface device 940 .

[0081] Computer system 900 may further include a video display adapter 952 for transmitting displayable images to a display device such as a display device 936. The example of a display device includes but is not limited to a liquid crystal display (LCD), a cathode ray tube (CRT), a plasma display, a light emitting diode (LED) display, and any combination thereof. Display adapter 952 and display device 936 may be used in conjunction with processor 904 to provide graphical representations of various aspects of the present disclosure. In addition to the display device, computer system 900 may include one or more other peripheral output devices, including but not limited to audio speakers, printers, and any combination thereof. Such peripheral output devices may be connected to bus 912 via a peripheral interface 956. The example of a peripheral interface includes but is not limited to a serial port, a USB connection, a FireWire connection, a parallel connection, and any combination thereof.

[0082] The exemplary embodiments of the present invention have been described in detail above. Various modifications and additions may be made without departing from the spirit and scope of the present invention. In order to provide a variety of feature combinations in relevant new embodiments, the features of each of the various embodiments described above may be appropriately combined with the features of the other described embodiments. In addition, although many separate embodiments have been described above, what is described herein is merely an illustration of the application of the principles of the present invention. In addition, although the specific methods herein may be shown and / or described as being performed in a specific order, in the common art of implementing the embodiments disclosed herein, the order is highly variable. Therefore, this description is meant only as an example and is not intended to limit the scope of the present invention.

[0083] In the above description and claims, phrases such as "at least one" or "one or more" may be followed by a connected list of elements or features. The term "and / or" may also appear in a list of two or more elements or features. Unless implicitly or explicitly contradicted by the context in which the phrase is used, such a phrase is intended to represent any element or feature listed separately, or a combination of any listed element or feature with any other listed element or feature. For example, the phrases "at least one of A and B," "one or more of A and B," and "A and / or B" are intended to represent "alone A, alone B, or both A and B," respectively. A similar interpretation also applies to lists containing three or more items. For example, the phrases "at least one of A, B, and C," "one or more of A, B, and C," and "A, B, and / or C" are intended to represent "alone A, alone B, alone C, both A and B, both A and C, both B and C, or A and B and C," respectively. In addition, the term "based on" used above and in the claims is intended to represent "based at least in part on," so that unreferenced features or elements are also permitted.

[0084] The subject matter described herein may be embodied in systems, devices, methods and / or articles depending on the desired configuration. The embodiments set forth in the foregoing description do not represent all embodiments consistent with the subject matter described herein. Rather, they are merely some examples consistent with aspects related to the described subject matter. Although some variations have been described in detail above, other modifications or additions are possible. In particular, in addition to the features and / or variations set forth herein, further features and / or variations may be provided. For example, the above embodiments may be directed to various combinations and subcombinations of the disclosed features and / or combinations and subcombinations of several other features disclosed above. In addition, the logical flows described in the accompanying drawings and / or described herein do not necessarily require the specific order or sequential order shown to achieve the desired results. Other embodiments may be within the scope of the claims.

Claims

1. The decoder is configured as: receiving a bitstream generated by an encoder, the bitstream including signaling information, a first coded picture, and a second coded picture, the first coded picture being used as a reference picture, the second coded picture including a first block and a second block, the signaling information including a sequence parameter set (SPS), the sequence parameter set indicating a mode capable of modifying a portion of the reference picture; decoding and storing the first image as a reference image; using said mode to modify only a portion of said reference picture in response to information generated by said encoder and included in said bitstream; using the modified portion of the reference image as a predictor for the first block of the second image; decoding the first block by adding residual pixels to the predictor; as well as A second block of the second picture is decoded without using the modified portion of any reference picture. 2 . The decoder of claim 1 , wherein the first block and the second block are N×N coding units. The decoder of claim 1 , wherein the second block of the second image is an intra-coded block.

4. The encoder is configured as: receiving a video signal; Generate a coded bitstream, the coded bitstream including signaling information, a first coded image, and a second coded image, the first coded image being used as a reference image, the second coded image including a first block and a second block, the signaling information including a sequence parameter set (SPS), the sequence parameter set indicating a mode capable of modifying a portion of the reference image; the bitstream can be decoded by a decoder receiving the bitstream, and is configured to: decoding and storing the first image as a reference image; using said mode to modify only a portion of said reference picture in response to information generated by said encoder and included in said bitstream; using the modified portion of the reference image as a predictor for a first block of the second image; decoding the first block by adding residual pixels to a predictor; as well as A second block of the second picture is decoded without using the modified portion of any reference picture. The encoder of claim 4 , wherein the first block and the second block are N×N coding units. The encoder of claim 4 , wherein the second block of the second image is an intra-coded block.

7. Video decoder, configured as: receiving a bitstream comprising a first picture and a sequence parameter set, the sequence parameter set including information indicating that a mode is enabled in which a first region of the picture may be extracted and replaced with a second region of another picture; using the pattern, modifying the first image by extracting the first region and replacing the extracted region with the second region; storing the modified first image; forming a predictor using the modified first image as a reference image; as well as The predictor is used to decode blocks of subsequent images.

8. The video decoder of claim 7, wherein the first region and the second region are one coding tree unit.

9. The video decoder according to claim 8, wherein: The first region and the second region are rectangular.

10. The video decoder according to claim 7, wherein: The first region and the second region are rectangular. 11.Decoder, configured as: storing a reference picture in a decoded picture buffer, the reference picture formed by extracting a rectangular region from a picture and replacing the extracted rectangular region with a different rectangular region from another picture, receiving a bitstream comprising a current block and a sequence parameter set, the sequence parameter set including an indication that a region of a picture may be extracted and replaced with a different region, A predictor is formed from a reference image, and the current block is decoded using the predictor.

12. The decoder of claim 11, wherein the extracted region is a coding tree unit and the replacement region is a coding tree unit.

Images