Adaptive Temporal Filter for Unavailable Reference Images

Abstract

A decoder includes circuitry configured to receive a bitstream, decode a plurality of video frames from the bitstream, determine that a long-term reference block update mode is enabled for a current block of a current frame, determine a long-term reference block update including pixel values and using the plurality of video frames, and update a portion of a long-term reference frame using the long-term reference block update. Related apparatus, systems, techniques, and articles are also described.

Description

[0001] Cross - Reference to Related Applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 771,918, filed on November 27, 2018, titled "ADAPTIVE TEMPORAL FILTER FOR AN UNAVAILABLE REFERENCE PICTURE", the entire content of which is incorporated herein by reference. Technical Field

[0003] The present invention generally relates to the field of video compression. Specifically, the present invention relates to an adaptive temporal filter for an unavailable reference image. Background Art

[0004] A video codec may include an electronic circuit or software for compressing or decompressing digital video. It can convert uncompressed video into a compressed format and vice versa. In the context of video compression, a device that compresses video (and / or performs some of its functions) is generally referred to as an encoder, while a device that decompresses video (and / or performs some of its functions) is referred to as a decoder.

[0005] There are complex relationships among video quality, the amount of data used to represent the video (e.g., determined by the bit rate), the complexity of the encoding and decoding algorithms, the sensitivity to data loss and errors, the ease of editing, random access, end - to - end latency (e.g., the delay time), etc.

[0006] Motion compensation may include methods for predicting a video frame or a portion of a given reference frame (e.g., a previous frame and / or a future frame) by considering the motion of the camera and / or objects in the video. It can be used for video compression, encoding and decoding video data, for example, encoding and decoding using the Moving Picture Experts Group (MPEG) - 2 (also known as Advanced Video Coding (AVC) and H.264) standard. Motion compensation can describe an image based on the transformation from a reference image to the current image. When compared with the current image, the reference image can be temporally previous; alternatively, when compared with the current image, the reference image can be from the future, or the reference image can include a long - term reference (LTR) frame. Compression efficiency can be improved when an image can be accurately synthesized based on previously transmitted and / or stored images.

[0007] Current standards such as H.264 and H.265 allow for the signaling of newly decoded frames to be saved and used as reference frames, updating reference frames such as long-term reference frames. This update is signaled by the encoder and the entire frame is updated. However, updating the entire frame is costly, especially when only a small portion of the static background has changed. Partial frame updates are possible but typically involve complex and computationally expensive processes to achieve frame updates. Additionally, such updates can typically involve copying a portion of the current frame to the frame when a portion of the background changes, which may require frequent updates and may not reflect the future frame background, resulting in relatively poor bitrate performance. Summary of the Invention

[0008] In one aspect, a decoder includes circuitry configured to: receive a bitstream, decode a plurality of video frames from the bitstream, determine that an unavailable reference block update mode is enabled for a current block of a current frame, determine an unavailable reference block update including pixel values and using the plurality of video frames, and update a portion of an unavailable frame using the unavailable reference block update.

[0009] In another aspect, a method includes receiving a bitstream. The method includes decoding a plurality of video frames from the bitstream. The method includes determining that an unavailable reference block update mode is enabled for a current block of a current frame. The method includes determining an unavailable reference block update including pixel values and using the plurality of video frames. The method includes updating a portion of an unavailable reference frame using the unavailable reference block update.

[0010] Details of one or more variations of the subject matter described herein are set forth in the following drawings and detailed description. Other features and advantages of the subject matter described herein will be apparent from the detailed description and drawings, and from the claims. Brief Description of the Drawings

[0011] To illustrate the invention, the drawings show various aspects of one or more embodiments of the invention. However, it should be understood that the invention is not limited to the exact arrangements and instrumentalities shown in the drawings, in which:

[0012] Figure 1 is a process flow diagram illustrating an exemplary process of using a plurality of video frames to update a portion (e.g., a block) of an unavailable reference frame;

[0013] Figure 2 shows two exemplary unavailable reference frame buffers, one for continuous mode and the other for reset mode;

[0014] Figure 3is a process flow diagram showing an exemplary process using a continuous mode according to some embodiments of the present subject matter;

[0015] Figure 4 is a process flow diagram showing an exemplary process using a reset mode according to some embodiments of the present subject matter;

[0016] Figure 5 is a process flow diagram showing an exemplary process for updating unavailable reference frames using a time filter according to some aspects of the present subject matter;

[0017] Figure 6 is a system block diagram showing an exemplary decoder capable of decoding a bitstream using an unavailable reference frame block update;

[0018] Figure 7 is a process flow diagram showing an exemplary process for encoding a video with unavailable reference frame block updates using multiple frames according to some aspects of the present subject matter, which exemplary process can improve compression efficiency;

[0019] Figure 8 is a system block diagram showing an exemplary video encoder capable of signaling unavailable reference frame block updates on the decoder side using multiple frames; and

[0020] Figure 9 is a block diagram of a computing system that can be used to implement any one or more of the methods and any one or more portions thereof disclosed herein.

[0021] The drawings are not necessarily to scale and may be illustrated by broken lines, schematic views, and partial views. In some instances, details that are not necessary for an understanding of the embodiments or that render other details difficult to perceive may have been omitted. Like reference numerals in the various drawings denote like elements. Detailed Description

[0022] The embodiments described in this disclosure relate to the reception, update, and operation of unavailable reference frames. An unavailable reference (UR) frame is a frame and / or image that is used to create predicted frames and / or images in one or more groups of pictures (GOPs), but the frame and / or image itself is not displayed in the video image. A frame marked as a UR frame in a video bitstream can be used as a reference until the frame is explicitly removed by bitstream signaling. UR frames can improve prediction and compression efficiency in scenarios with a static background over a long period of time (e.g., the background of a video conference or a video of a parking lot surveillance). However, over time, the background of the scene gradually changes (e.g., a car becomes part of the background scene when it is parked in an empty space). Therefore, by allowing better prediction, updating the UR frame can improve compression performance.

[0023] Current standards such as H.264 and H.265 allow the update of similar frames such as LTR frames by signaling that a newly decoded frame is saved and available as a reference frame. This update is signaled by the encoder, and the entire frame is updated. However, updating the entire frame is costly, especially when only a small portion of the static background has changed.

[0024] Some existing compression techniques only use a portion of the previous frame to update a frame such as an LTR frame, which may result in a degradation of prediction performance. Some embodiments of the current subject matter include using multiple video frames to update a portion (e.g., a block) of an LTR frame. For example, an LTR frame can be updated by applying a temporal filter to a buffer of decoded frames to compute statistics of co-located pixels. For example, the per-pixel mean, median, and / or mode of multiple decoded frames can be computed and used to update a portion of the LTR frame. In some embodiments, the current subject matter can support both a continuous mode and a reset mode. By using multiple frames to update frames such as UR and / or LTR frames instead of only using a portion of the previous frame, prediction can be improved, thereby reducing the residual and improving the bitrate performance.

[0025] Figure 1 is a process flow diagram of an exemplary process 100 for illustrating using multiple video frames to update a portion (e.g., a block) of a UR frame. By using multiple frames to update a UR frame instead of only using a portion of the previous frame, prediction can be improved, thereby reducing the residual and improving the bitrate performance.

[0026] Still referring to Figure 1, at step 105, the decoder receives a bitstream. The bitstream can include, for example, but is not limited to: the data found in the bitstream, which is the input to the decoder when data compression is used. The bitstream can include the information required to decode the video. Receiving can include extracting and / or parsing the blocks and associated signaling information in the bitstream. In some embodiments, the bitstream can include encoded video frames (which include encoded blocks). Each encoded block can include a coding tree unit (CTU), a coding unit (CU), and / or a prediction unit (PU).

[0027] At step 110, continuing to refer to Figure 1 , a video frame is decoded from (e.g., using) the bitstream. For example, the video frame can be decoded by using inter-frame prediction. Decoding via inter-frame prediction can include using a previous frame, a future frame, or a UR frame as a reference for calculating the prediction, and the reference can be combined with the residuals included in the bitstream. Other techniques and tools for decoding can be utilized, such as intra-frame prediction.

[0028] At step 115, still referring to Figure 1 , the decoder determines whether the unavailable reference block update mode is enabled for the current block in the bitstream. For example, it can be determined that the UR block update mode field in the header of the bitstream is enabled. Headers such as the picture parameter set (PPS) or the sequence parameter set (SPS) can be used to selectively enable the signaling of UR frame block updates. Fields such as UR_BLOCK_UPDATE can take a true or false value (e.g., 0 or 1); the absence of this field in the header may imply a false value (e.g., 0). In some embodiments, the unavailable reference block update mode can be sent implicitly.

[0029] Continuing to refer to Figure 1, at 120, the decoder uses video frames to determine unavailable reference block updates. The UR block update may include modifying pixel values of a portion of the UR frame. Determining the unavailable reference block update may include calculating statistics of collocated pixels of the plurality of video frames, where the statistics may include an average, a median, and / or a mode. Statistics may be calculated for a set of collocated pixels, such as a set of collocated pixels in multiple frames and / or blocks; as a non-limiting example, the average of the luminance and / or chrominance of pixels having a given set of fixed coordinates in a set of multiple frames may be calculated by collecting the chrominance and / or luminance values of the pixels in each frame at the fixed coordinates and then calculating the average of the set of chrominance and / or luminance values. Statistics of one or more attributes of a pixel may be calculated. For example but not limited to, statistics of luminance may be calculated by calculating, for example, the average, median, and / or mode of the luminance of pixels in one or more of the plurality of video frames. As another non-limiting example, statistics of chrominance may be calculated by calculating, for example, the average, median, and / or mode of the chrominance of pixels in one or more of the plurality of video frames. In an embodiment, calculations using chrominance may be more computationally efficient than calculations using luminance, while calculations using luminance may be more accurate than calculations using chrominance. In an embodiment, the statistics may be directly used to update pixel values in the updated UR frame and / or block; the statistics may represent the "closest" representation of the time series of pixels because the statistics may result in a minimum amount of residual because the "average" pixel is statistically most similar to each individual pixel and has the smallest variance. This may minimize the residual, resulting in a higher degree of compression.

[0030] For example, as described in further detail below, a temporal filtering operation may be applied to the plurality of video frames to determine the unavailable reference block update.

[0031] Still referring to Figure 1 , in some embodiments, a UR frame buffer may be employed. The UR update may be constructed using operations on the frames present in the UR buffer. The UR buffer may include multiple frames. For example, Figure 2 Two exemplary embodiments of the UR frame buffer 200 are shown, where one is for continuous mode and the other is for reset mode.

[0032] In continuous mode, Figure 3An exemplary process 300 is shown, where the UR buffer can be updated periodically by removing the first frame from the buffer and adding the current frame (Ft) to the buffer, which is similar to a first-in-first-out (FIFO) queue. The current frame can include a newly decoded frame. The buffer can include a predefined length (e.g., a predefined maximum allowable number of frames) t-b. Each block filter can be updated; each block filter can be any filter suitable for filtering the blocks and / or frames described in this disclosure. The decoder can be configured to determine that the unavailable reference buffer has reached or exceeded the predefined maximum allowable buffer size and remove frames from the unavailable reference buffer. Each current frame can be added to the consecutive mode buffer when being processed, and this process can be repeated for all frames. In some embodiments, the signal of the consecutive mode can be sent in the bitstream.

[0033] In the reset mode, Figure 4 An exemplary process 400 is shown, where the decoder can determine that the reset mode is enabled. The UR buffer can be updated in a manner similar to the consecutive mode update described above, except that when a significant change is detected between two consecutive frames, a scene change can be detected and the UR buffer can be emptied. For example, the determination can be made by determining the inter-frame similarity between consecutive frames and comparing the similarity with a threshold. When the similarity value is lower than the threshold, a scene change can be detected. When the similarity value is higher than the threshold, a scene change may not be detected. Those skilled in the art will realize, after reading the entire disclosure, that various alternative or additional ways of performing such threshold comparisons can be carried out, including but not limited to comparing the difference with a threshold, where a difference higher than the threshold indicates a scene change, while a difference lower than the threshold indicates that the scene has not changed. In some embodiments, the scene reset and / or change can alternatively or additionally be signaled in the bitstream. The decoder may empty the UR buffer. After emptying the UR buffer, the buffer filling process can start from the current frame. In some embodiments, the reset mode can be signaled in the bitstream.

[0034] Therefore, still referring to Figure 4 , the consecutive mode can allow a longer buffer and a wider set of reference samples, and the UR frame update can be calculated based on the wider set of reference samples. The reset mode can allow buffer size control based on frame similarity.

[0035] As Figure 2 shown, in the case of the reset mode, after each scene change, the size of the UR buffer can become smaller and then increase to the maximum size of b frames until a subsequent scene change occurs.

[0036] Figure 5is a process flow diagram showing an exemplary process 500 for updating a UR frame using a time filter in accordance with some aspects of the present subject matter. Video frames that have been divided into blocks can be filtered in time. Each block can have a size of width W and height H, where W can be equal to or can be not equal to H. A filter can be selected from a plurality of filters, which can include but are not limited to a set of three filters including median, mean, and mode. In a non-limiting example, the update UR block operation can be implemented based on the number of frames in a buffer. For example, at step 505, it can be determined whether there are no frames in the buffer; if so, at 510, the UR frame can be constructed by copying all blocks from the current frame. At 515, it can be determined whether there is one frame in the buffer, and at 520, the UR frame can be constructed by applying a filtering operation to matching (e.g., located at the same position) blocks in the UR buffer frame and the current frame. This operation can be implemented according to UR(Ft) = Filter_2(UR(Ft-l, Ft)). It can be determined at step 525 whether there is more than one frame in the buffer; at step 530, the filter can be applied to the previously calculated UR frame and the current frame according to the following formula: UR(Ft) = Filter_n(UR(Ft-l), Ft). At step 535, the UR update (e.g., the result of applying the filter as described above) can be saved to the UR frame. In some embodiments, the filtering operation can be selected based on performance requirements (e.g., processing speed, robustness to quantization and / or word length limitations, passband and / or stopband distortion and / or ripple, phase change and / or delay characteristics, attenuation level and / or speed and / or passband and stopband interconversion, etc.), and the signal of the filtering operation can be sent in a bitstream or any other suitable process for filtering selection.

[0037] In some embodiments, still referring to Figure 5 , the update can be enforced over the entire UR frame or one or more blocks of the UR frame. In some embodiments, per-pixel updates are possible (e.g., the block can have a size of 1×1).

[0038] The filter can be calculated in the following ways, but is not limited thereto:

[0039] The median filter can be calculated according to the following:

[0040] Filter_2median(Px,y,t) = Px,y,t-1

[0041] Filter_nmedian(Px,y,t) = median(Px,y,t-1,Px,y,t-1,...,Px,y,t-n)

[0042] Among them, median() represents the median value of a sorted array of numbers.

[0043] The average filter can be calculated as follows:

[0044] Filter_2mean(Px,y,t) = Px,y,t-1

[0045]

[0046] The mode filter can be calculated as follows:

[0047] Filter_2mode(Px,y,t) = Px,y,t-1

[0048] Filter_nmode(Px,y,t) = mode(Px,y,t-1,Px,y,t-1,···,Px,y,t-n)

[0049] Among them, mode() represents the value that appears most frequently in a given array of numbers.

[0050] Referring again to Figure 1 , at step 125, the decoder can update the unavailable reference frame using the determined UR block update. Such an update can include performing a UR block update, where the pixels (e.g., luminance values) spatially co-located with the current block for which the UR frame block update mode is enabled in the UR frame are updated (e.g., modified) using the determined UR block update (e.g., including the determined pixel values). The pixels can be updated in the following ways, but are not limited thereto: by replacing the pixels with co-located pixels from one or more blocks (including the current block) as described above, by updating the pixel values with values (e.g., chrominance and / or luminance values) from one or more blocks (including the current block) as described above, by replacing and / or updating the pixels according to and / or using statistical values (e.g., but not limited to the average value calculated for co-located pixels as described above), and / or by replacing and / or updating the pixels according to and / or using filter outputs (e.g., temporal filter outputs) as described in the present disclosure. In some embodiments, the update can include updating multiple UR frame blocks using multiple decoded blocks, which can be performed using any of the methods for updating the UR frame using the decoded current block described above. The update can include, but is not limited to, using any of the above processes and / or process steps to generate blocks and / or frames with default chrominance and / or luminance values through one or more decoded blocks and replacing and / or updating the default chrominance and / or luminance values; the update can include generating a UR frame, which can be achieved by creating a UR frame with default values as described above and updating the default values.

[0051] Still referring to Figure 1, for a subsequent current block, the updated UR frame can be used as a reference frame for inter - frame prediction. For example, an encoded block can be received. It can be determined whether the inter - frame prediction mode is enabled for the encoded block. The updated UR frame can be used as a reference frame and the decoded block can be determined according to the inter - frame prediction mode. For example, decoding via inter - frame prediction can include using the updated UR frame as a reference for calculating the prediction, which can be combined with the residual included in the bitstream.

[0052] Continuing to refer to 1, during the decoding process, UR block updates can be used for each current block. In some embodiments, for intra - coded frames, UR block updates can be implicitly skipped. For example, the bitstream can include a second current encoded block within a frame different from the first current encoded block. The mode of the second current encoded block can include intra - frame prediction. In response to determining that the second current encoded block is an intra - frame prediction block, the UR block update can be skipped. Skipping can include, for example, not updating the UR frame, or determining whether a field such as UR_BLOCK_UPDATE is set in the header.

[0053] In some embodiments, continuing to refer to Figure 1 , if the UR_BLOCK_UPDATE field is set in the header, the decoder can expect explicit block update signaling bits in the bitstream. In this method, the encoded blocks of the video can be signaled in the block header of the video bitstream as the blocks to be updated in the UR frame. In that case, the co - located blocks in the UR frame can be replaced with the result of temporal filtering of the frame buffer. As a non - limiting example, the updated UR frame can be used for future motion - estimation purposes until the UR frame is subsequently updated.

[0054] Figure 6FIG. 0 is a system block diagram showing an exemplary decoder 600 capable of decoding a bitstream 604 using UR frame block updates. The decoder 600 can include an entropy decoding processor 608, an inverse quantization and inverse transform processor 612, a deblocking filter 616, a frame buffer 620, a motion compensation processor 624, and an intra prediction processor 628. In some embodiments, the bitstream 604 includes a parameter (e.g., a field in the header of the bitstream) that signals a UR frame block update mode. The motion compensation processor 624 can use the UR frame to reconstruct pixel information and update the UR frame using multiple frames according to the UR frame block update mode. For example, when the UR frame block update mode is signaled explicitly for a current block, the co-located pixels (e.g., luminance values) in the UR frame can be replaced with pixel values calculated by applying a temporal filter to frames in the UR frame buffer. Such an update can include performing a UR block update, where the decoded current block is used to update (e.g., modify) the co-located pixels in the UR frame that are spatially co-located with the current block. In some embodiments, the block update mechanism can be explicit or implicit. For example, the co-located pixels (e.g., luminance values) in the UR frame can be updated by using the pixel values of the decoded current block, updating a portion (e.g., a block) of the UR frame; in other words, updating the UR frame can include replacing the luminance values of the UR frame with the luminance values that are spatially co-located with the decoded current block. In some embodiments, the UR frame can be updated according to another mechanism, such as updating a portion of the UR frame to an average of the initial co-located UR frame pixel values and / or the current decoded block pixel values. In an example, each pixel that is updated can be replaced and / or updated to the corresponding filter output (e.g., the result of filtering a set of co-located pixels). After reviewing the entire disclosure, those skilled in the art will recognize the various other mechanisms that can be employed. In some embodiments, the update can include updating multiple UR frame blocks using multiple decoded blocks, which can be performed using any of the methods described above for updating the UR frame using the decoded current block. In some embodiments, the update can include updating multiple UR frame blocks using multiple decoded blocks, which can be performed using any of the methods described above for updating the UR frame using the decoded current block. The update can include, but is not limited to, generating a block and / or frame with default chrominance and / or luminance values using one or more decoded blocks and replacing and / or updating the default chrominance and / or luminance values using any of the processes and / or process steps described above; the update can include generating a UR frame, which can be achieved by creating a UR frame with default values and updating the default values as described above.

[0055] In operation, still referring to Figure 6, the bitstream 604 can be received by the decoder 600 and input to the entropy decoding processor 608, which entropy decodes the bitstream 604 into quantized coefficients. The quantized coefficients are provided to the inverse quantization and inverse transform processor 612, which can perform inverse quantization and / or inverse transform to create a residual signal that can be added to the output of the motion compensation processor 624 or the intra prediction processor 628 depending on the processing mode. The output of the motion compensation processor 624 and / or the intra prediction processor 628 can include a prediction based on previous decoded blocks or blocks of the UR frame. The sum of the prediction and the residual can be processed by the deblocking filter 616 and stored in the frame buffer 620. For a given block (e.g., CU or PU), when the bitstream 604 explicitly signals the enabling of the UR frame block update mode, the motion compensation processor 624 can update the UR frame, which can be included in the frame buffer 620, to update the co-located pixels (e.g., luminance values) in the UR frame based on pixel values calculated from multiple frames (e.g., frames included in the frame buffer 620). The pixel values (e.g., UR block update) can be determined by applying a temporal filter to the frames in the frame buffer 620. The temporal filter can alternatively or additionally be applied to blocks of the video frame, e.g., after segmentation, to calculate unavailable reference block updates; as a non-limiting example, one or more pixels and / or blocks can be replaced and / or updated by the blocks and / or pixels output by the temporal filter.

[0056] In some embodiments, continuing to refer to Figure 6 , the decoder 600 can include a UR frame block update processor 632 that generates a UR frame update based on the current block and provides UR frame pixel values for the inter prediction process. The UR frame block update processor 632 can directly affect motion compensation. Additionally, for example, when the current block is an intra prediction block, the UR frame update processor 632 can receive information from the intra prediction processor.

[0057] Figure 7FIG. 0 is a process flow diagram of an exemplary embodiment of a process 700 for encoding a video with UR frame block updates using multiple frames, which exemplary process can improve compression efficiency. At step 705, a video frame may undergo an initial block segmentation, e.g., segmented using a tree-structured macroblock segmentation scheme, which tree-structured macroblock segmentation scheme may include dividing an image frame into CTUs and CUs. At step 710, a block for updating a portion of the UR frame may be selected. For example, the block may be selected based on the frequency content and / or metric of the motion within the block compared to one or more collocated blocks that are temporally adjacent (e.g., frames that are temporally adjacent within a predetermined number of frames of the current frame). The selection may include identifying, according to a metric rule, the block that will be used to update a portion of the UR frame at the decoder. At step 715, the block may be encoded and included in the bitstream. In some embodiments, the encoder-side UR frame may be updated using multiple frames, e.g., by applying a temporal filter to the frames within a frame buffer. The encoder-side UR frame may be used for motion estimation and compensation in the encoder.

[0058] At step 720, still referring to Figure 7 , explicit UR frame block update parameters may be determined and included in the bitstream to signal that the UR frame block update mode is enabled for the current block. For example, a field in the PPS or SPS may be set (e.g., enabled). For example, a field such as the UR_BLOCK_UPDATE field may be set to indicate that the UR frame block update mode is enabled for the current block.

[0059] Figure 8 FIG. 9 is a system block diagram of an exemplary embodiment of a video encoder 800 capable of signaling decoder-side UR frame block updates using multiple frames. The video encoder 800 may receive an input video 804, which input video 804 may be initially segmented or partitioned according to a processing scheme such as a tree-structured macroblock segmentation scheme (e.g., quadtree binary decision tree). Examples of tree-structured macroblock segmentation schemes may include dividing an image frame into large block elements called coding tree units (CTUs). In some implementations, each CTU may be further divided one or more times into multiple sub-blocks called coding units (CUs). The final result of this partitioning may include a set of sub-blocks that may be referred to as prediction units (PUs). Transform units (TUs) may also be used.

[0060] Still referring to Figure 8, the exemplary video encoder 800 may include an intra prediction processor 808, a motion estimation / compensation processor 812 (also referred to as an inter prediction processor) capable of supporting UR frame block update at the decoder, a transform / quantization processor 816, an inverse quantization / inverse transform processor 820, a loop filter 824, a decoded picture buffer 828, and an entropy coding processor 832. In some embodiments, the motion estimation / compensation processor 812 may determine for a current block that the UR frame should be updated at the decoder and set parameters to explicitly signal the enabling of the UR frame block update mode. In some embodiments, implicit signaling may be performed. The bitstream parameters signaling the UR frame block update mode may be input to the entropy coding processor 832 to be included in the output bitstream 836. A portion of the encoder-side UR frame may be updated for use by the motion estimation / compensation processor 812 for encoding additional blocks that may utilize the UR frame as a reference for inter prediction. The encoder-side UR frame may be updated using multiple frames, e.g., by applying a temporal filter to the frames within the frame buffer.

[0061] In operation, for each block of a frame of the input video 804, it may be determined whether to process the block via intra prediction or using motion estimation / compensation. The block may be provided to the intra prediction processor 808 or the motion estimation / compensation processor 812. If the block is to be processed via intra prediction, the intra prediction processor 808 may perform processing to output a predictor. If the block is to be processed via motion estimation / compensation, the motion estimation / compensation processor 812 may perform processing including using the encoder-side UR frame as a reference for inter prediction (if applicable).

[0062] A residual can be formed by subtracting a predictor from an input video; the residual can be received by a transform / quantization processor 816, which can perform a transform process (e.g., a discrete cosine transform (DCT)) to produce coefficients that can be quantized. The quantized coefficients and any associated signaling information can be provided to an entropy coding processor 832 for entropy coding and inclusion in an output bitstream 836. The entropy coding processor 832 can support encoding of signaling information related to a UR frame block update mode. Additionally, the quantized coefficients can be provided to an inverse quantization / inverse transform processor 820, which can reproduce pixels that can be combined with the predictor and processed by a loop filter 824, the output of which can be stored in a decoded picture buffer 828 for use by a motion estimation / compensation processor 812, which is capable of supporting UR frame block updates at a decoder. The decoded picture buffer 828 can include UR frames, and the motion estimation / compensation processor 812 can use the UR frames as references for inter prediction. Encoder-side UR frames can be updated using multiple frames, e.g., by applying a temporal filter to frames within a frame buffer.

[0063] Continuing to refer Figure 8 , although some variations have been described in detail above, other modifications or additions are possible. For example, in some embodiments, a block can include any symmetric block (8×8, 16×16, 32×32, 64×64, 128×128, etc.) and any asymmetric block (8×4, 16×8, etc.).

[0064] In some embodiments, still referring Figure 8 , a quadtree plus binary decision tree (QTBT) can be implemented. For a QTBT, at the coding tree unit level, the QTBT splitting parameters can be dynamically derived to adapt to local characteristics without transmitting any overhead. Subsequently, at the coding unit level, a joint classifier decision tree structure can eliminate unnecessary iterations and control the risk of misprediction. In some embodiments, a UR frame block update mode can be available as an additional option at each leaf node of the QTBT.

[0065] In some embodiments, still referring Figure 8, other syntax elements can be signaled at different levels of the bitstream. The UR frame block update can be enabled for an entire sequence by including an enable flag encoded in the sequence parameter set (SPS). Additionally, a CTU flag can be encoded at the coding tree unit (CTU) level to indicate whether any coding unit (CU) uses the UR frame block update mode. The CU flag can be encoded to indicate whether the current coding unit utilizes the UR frame block update mode. Although the above-described disclosed embodiments relate to the update of the UR frame, the above-described disclosed embodiments can alternatively or additionally be applied to other frames, images, including but not limited to long-term reference frames.

[0066] The subject matter described herein provides many technical advantages. For example, some implementations of the current subject matter can improve the bitrate of the bitstream by reducing the number of bits required to encode video. Such an improvement can be achieved by improving the UR frame, which in turn can improve prediction and reduce the residual. Additionally, in some implementations, the UR does not need to be updated as frequently, thereby reducing decoder computational requirements. In some implementations, the current subject matter does not increase the use of memory because the frame buffer can be utilized by other operations of the encoder and / or decoder. Some implementations of the current subject matter can provide for decoding blocks using the UR frame, which can include updating portions of the UR frame without having to update the entire UR frame. Such an approach can reduce complexity while increasing compression efficiency.

[0067] It should be noted that any one or more aspects and embodiments described herein can be conveniently implemented using digital electronic circuits, integrated circuits, application-specific integrated circuits (ASICs) designed specifically, field-programmable gate arrays (FPGAs), computer hardware, firmware, software, and / or combinations of the above, such as implemented and / or executed in one or more machines programmed according to the teachings of this specification (e.g., one or more computing devices serving as user computing devices for electronic documents, one or more server devices such as document servers, etc.), which is obvious to those of ordinary skill in the computer art. These various aspects or features can include implementations in one or more computer programs and / or software that are executable and / or interpretable on a programmable system, the programmable system including at least one programmable processor, which can be dedicated or general-purpose, and is coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to send data and instructions to the storage system, at least one input device, and at least one output device. It is obvious to those of ordinary skill in the software art that a skilled programmer can easily prepare appropriate software code based on the teachings of this disclosure. The aspects and implementations using software and / or software modules discussed above can also include appropriate hardware for assisting in implementing the machine-executable instructions of the software and / or software modules.

[0068] Such software can be a computer program product employing a machine-readable storage medium. A machine-readable storage medium can be any medium that can store and / or encode a sequence of instructions executable by a machine (e.g., a computing device) and that causes the machine to execute any one of the methods and / or embodiments described herein. Examples of machine-readable storage media include, but are not limited to, magnetic disks, optical discs (e.g., CDs, CD-Rs, DVDs, DVD-Rs, etc.), magneto-optical discs, read-only memory "ROM" devices, random-access memory "RAM" devices, magnetic cards, optical cards, solid-state storage devices, EPROMs, EEPROMs, programmable logic devices (PLDs), and / or any combination of the above. The machine-readable media used herein are intended to include a single medium as well as a collection of physically separate media (e.g., a collection of optical discs, or one or more hard disk drives combined with computer memory). The machine-readable storage media used herein do not include transient forms of signal transmission.

[0069] Such software may also include information (e.g., data) carried as a data signal on a data carrier such as a carrier wave. For example, machine-executable information may be included as a data-bearing signal contained in the data carrier, where the signal encodes: a sequence of instructions or portions thereof to be executed by a machine (e.g., a computing device), and any associated information (e.g., data structures and data) that causes the machine to execute any one of the methods and / or embodiments described herein.

[0070] Examples of computing devices include, but are not limited to, e-reader devices, computer workstations, terminal computers, server computers, handheld devices (e.g., tablet computers, smartphones, etc.), network devices, network routers, network switches, bridges, any machine capable of executing a sequence of instructions specifying actions to be taken by it, and any combination of the foregoing. In one example, a computing device may include and / or be included in a kiosk.

[0071] Figure 9 FIG. shows an illustration of one embodiment of a computing device in an exemplary form of a computer system 900, in which a set of instructions can be executed to cause a control system to perform any one or more aspects and / or methods of the present disclosure. It is also contemplated that multiple computing devices may be utilized to implement a specifically configured set of instructions for causing one or more of the devices to perform any one or more of the aspects and / or methods of the present disclosure. The computer system 900 includes a processor 904 and a memory 908 that communicate with each other and with other components via a bus 912. The bus 912 may include any one of a variety of types of bus structures (including, but not limited to, memory buses, memory controllers, peripheral buses, local buses, and any combination of the foregoing) using any one of a variety of bus architectures.

[0072] The memory 908 may include various components (e.g., machine-readable media), including but not limited to random access memory components, read-only components, and any combination of the foregoing. In one example, a basic input / output system 916 (BIOS) may be stored in the memory 908, which includes basic routines such as those that help transfer information between elements within the computer system 900 during startup. The memory 908 may also include instructions (e.g., software) 920 embodying any one or more of the aspects and / or methods of the present disclosure (e.g., stored on one or more machine-readable media). In another example, the memory 908 may further include any number of program modules, including but not limited to an operating system, one or more applications, other program modules, program data, and any combination of the foregoing.

[0073] The computer system 900 may also include a storage device 924. Examples of storage devices (e.g., storage device 924) include, but are not limited to, hard disk drives, disk drives, optical disc drives in combination with optical media, solid state storage devices, and any combination of the foregoing. The storage device 924 may be connected to the bus 912 via a suitable interface (not shown). Exemplary interfaces include, but are not limited to, SCSI, High Technology Attachment (ATA), Serial ATA, Universal Serial Bus (USB), IEEE 1394 (FIREWIRE), and any combination of the foregoing. In one example, the storage device 924 (or one or more components thereof) may be removably connected to the computer system 900 (e.g., via an external port connector (not shown)). In particular, the storage device 924 and the associated machine-readable medium 928 may provide non-volatile and / or volatile storage of machine-readable instructions, data structures, program modules, and / or other data for the computer system 900. In one example, the software 920 may be stored in whole or in part within the machine-readable medium 928. In another example, the software 920 may be stored in whole or in part within the processor 904.

[0074] The computer system 900 may also include an input device 932. In one example, a user of the computer system 900 may input commands and / or other information into the computer system 900 via the input device 932. Examples of input devices 932 include, but are not limited to: alphanumeric input devices (e.g., keyboards), pointing devices, joysticks, gamepads, audio input devices (e.g., microphones, voice response systems, etc.), cursor control devices (e.g., mice), touchpads, optical scanners, video capture devices (e.g., still cameras, video cameras), touchscreens, and any combination of the foregoing. The input device 932 may be connected to the bus 912 via any of a variety of interfaces (not shown), the various interfaces including, but not limited to, serial interfaces, parallel interfaces, game ports, USB interfaces, FireWire interfaces, direct interfaces to the bus 912, and any combination of the foregoing. The input device 932 may include a touchscreen interface, which may be part of or separate from the display 936, which will be discussed further below. The input device 932 may be used as a user selection device for selecting one or more graphical representations in a graphical interface as described above.

[0075] The user can also input commands and / or other information to the computer system 900 via a storage device 924 (e.g., a removable disk drive, a flash drive, etc.) and / or a network interface device 940. A network interface device such as the network interface device 940 can be used to connect the computer system 900 to one or more of various networks such as a network 944, and to one or more remote devices 948 connected to the network 944. Examples of network interface devices include, but are not limited to, network interface cards (e.g., mobile network interface cards, LAN cards), modems, and any combination thereof. Examples of networks include, but are not limited to, wide area networks (e.g., the Internet, enterprise networks), local area networks (e.g., networks associated with an office, a building, a campus, or other relatively small geographical spaces), telephone networks, data networks associated with telephone / voice providers (e.g., mobile communication provider data and / or voice networks), direct connections between two computing devices, and any combination of the above. Networks such as the network 944 can employ wired and / or wireless communication modes. Generally, any network topology can be used. Information (e.g., data, software 920, etc.) can be transmitted to and / or from the computer system 900 via the network interface device 940.

[0076] The computer system 900 can further include a video display adapter 952 for transmitting a displayable image to a display device such as a display device 936. Examples of display devices include, but are not limited to, liquid crystal displays (LCDs), cathode ray tubes (CRTs), plasma displays, light emitting diode (LED) displays, and any combination of the above. The display adapter 952 and the display device 936 can be used in combination with the processor 904 to provide a graphical representation of various aspects of the present disclosure. In addition to the display device, the computer system 900 can include one or more other peripheral output devices, including but not limited to audio speakers, printers, and any combination of the above. Such peripheral output devices can be connected to the bus 912 via a peripheral interface 956. Examples of peripheral interfaces include, but are not limited to, serial ports, USB connections, FireWire connections, parallel connections, and any combination of the above.

[0077] Exemplary embodiments of the present invention have been described in detail above. Various modifications and additions can be made without departing from the gist and scope of the present invention. In order to provide various feature combinations in related new embodiments, the features of each of the above various embodiments can be appropriately combined with the features of other described embodiments. In addition, although many individual embodiments have been described above, the content described herein is merely an illustration of the application of the principles of the present invention. Moreover, although the specific methods herein may be shown and / or described as being performed in a particular order, in the ordinary technology for implementing the embodiments disclosed herein, the order is highly variable. Therefore, this description is only meant as an example and does not limit the scope of the present invention.

[0078] In the above description and claims, a list of elements or features may follow phrases such as "at least one" or "one or more". The term "and / or" may also appear in a list of two or more elements or features. Unless implicitly or explicitly contradicted by the context in which the phrase is used, such phrases are intended to mean any element or feature listed individually, or any combination of any listed element or feature with any other listed element or feature. For example, the phrases "at least one of A and B", "one or more of A and B", and "A and / or B" are each intended to mean "A alone, B alone, or both A and B". Similar interpretations also apply to lists containing three or more items. For example, the phrases "at least one of A, B, and C", "one or more of A, B, and C", "A, B, and / or C" are each intended to mean "A alone, B alone, C alone, both A and B, both A and C, both B and C, or A, B, and C". In addition, the term "based on" used above is intended to mean "at least partially based on", such that unrecited features or elements are also permitted.

[0079] The subject matter described herein can be embodied in a system, apparatus, method, and / or article according to a desired configuration. The embodiments set forth in the foregoing description do not represent all embodiments consistent with the subject matter described herein. Rather, they are merely some examples consistent with aspects related to the subject matter described. Although some variations have been described in detail above, other modifications or additions are possible. In particular, further features and / or variations can be provided in addition to the features and / or variations set forth herein. For example, the above embodiments can be directed to various combinations and sub - combinations of the disclosed features and / or combinations and sub - combinations of several other features disclosed above. Moreover, the logical flows described in the figures and / or herein do not necessarily require the particular order or sequence shown to achieve the desired result. Other embodiments are within the scope of the claims.

Claims

1. A decoding method, comprising: Receive a bitstream that includes signaling information, an encoded image that serves as a reference image, and an image sequence that includes a first current coding block and a second current coding block, where the signaling information includes a sequence parameter set (SPS), and the SPS indicates an enabled mode in which a portion of the stored reference image can be changed; Decode the encoded image and store the image as a reference image; Using the mode indicated as enabled in the SPS, change a portion of the reference image; Using the changed portion of the reference image, form a predictor for the first current coding block; Reconstruct the first current coding block by adding residual pixels to the predictor; And Reconstruct the second current coding block without using any changed portion of the reference image.

2. The decoding method according to claim 1, wherein the second current coding block is an intra coding block.

3. The decoding method according to claim 2, wherein the reference image is a reference image that is not output for display.

Images