Adaptive temporal filter for unavailable reference images

Abstract

A decoder includes circuitry configured to receive a bitstream, decode a plurality of video frames from the bitstream, determine that a long-term reference block update mode is enabled for a current block of a current frame, determine that a long-term reference block update including pixel values and using the plurality of video frames, and determine that a long-term reference block update using the plurality of video frames is enabled for a current block of the current frame. And updating a portion of a long-term reference frame with the long-term reference block update. Related apparatus, systems, techniques, and articles are also described.

Description

[0001] This application is a divisional application of the application filed on November 27, 2019, with Chinese National Application No. 2019800779628 and the invention title of "Adaptive Temporal Filter for Unavailable Reference Images".

[0002] Cross - reference to related applications

[0003] This application claims the priority of U.S. Provisional Patent Application No. 62 / 771,918, titled "ADAPTIVE TEMPORAL FILTER FOR AN UNAVAILABLE REFERENCE PICTURE", filed on November 27, 2018, the entire content of which is incorporated herein by reference. Technical field

[0004] The present invention generally relates to the field of video compression. Specifically, the present invention relates to an adaptive temporal filter for unavailable reference images. Background art

[0005] A video codec may include electronic circuits or software for compressing or decompressing digital video. It can convert uncompressed video into a compressed format and vice versa. In the context of video compression, a device that compresses video (and / or performs some of its functions) is typically referred to as an encoder, while a device that decompresses video (and / or performs some of its functions) is referred to as a decoder.

[0006] There are complex relationships among video quality, the amount of data used to represent the video (e.g., determined by the bit rate), the complexity of the encoding algorithm and the decoding algorithm, the sensitivity to data loss and errors, the ease of editing, random access, end - to - end latency (e.g., the delay time), etc.

[0007] Motion compensation may include methods for predicting a video frame or a portion of a given reference frame (e.g., a previous frame and / or a future frame) by considering the motion of the camera and / or objects in the video. It can be used for video compression, encoding and decoding video data, for example, encoding and decoding using the Moving Picture Experts Group (MPEG) - 2 (also known as Advanced Video Coding (AVC) and H.264) standard. Motion compensation can describe an image based on the transformation from a reference image to the current image. When compared with the current image, the reference image can be temporally previous; or, when compared with the current image, the reference image can be from the future, or the reference image can include a long - term reference (LTR) frame. When an image can be precisely synthesized based on previously transmitted and / or stored images, the compression efficiency can be improved.

[0008] Current standards such as H.264 and H.265 allow the signaling of newly decoded frames to be saved and used as reference frames, updating reference frames such as long-term reference frames. This update is signaled by the encoder and the entire frame is updated. However, updating the entire frame is costly, especially when only a small portion of the static background has changed. Partial frame updates are possible but typically involve complex and computationally expensive processes to implement the frame update. Additionally, such updates can typically involve copying a portion of the current frame to the frame when a portion of the background changes, which may require frequent updates and may not reflect the future frame background, resulting in relatively poor bitrate performance. Summary of the Invention

[0009] In one aspect, a decoder includes circuitry configured to: receive a bitstream, decode a plurality of video frames from the bitstream, determine that an unavailable reference block update mode is enabled for a current block of a current frame, determine an unavailable reference block update including pixel values and using the plurality of video frames, and update a portion of an unavailable frame using the unavailable reference block update.

[0010] In another aspect, a method includes receiving a bitstream. The method includes decoding a plurality of video frames from the bitstream. The method includes determining that an unavailable reference block update mode is enabled for a current block of a current frame. The method includes determining an unavailable reference block update including pixel values and using the plurality of video frames. The method includes updating a portion of an unavailable reference frame using the unavailable reference block update.

[0011] Details of one or more variations of the subject matter described herein are set forth in the following drawings and detailed description. Other features and advantages of the subject matter described herein will be apparent from the detailed description and drawings, and from the claims. Brief Description of the Drawings

[0012] To illustrate the invention, the drawings show various aspects of one or more embodiments of the invention. However, it is to be understood that the invention is not limited to the exact arrangements and instrumentalities shown in the drawings, in which:

[0013] Figure 1 is a process flow diagram illustrating an exemplary process of using a plurality of video frames to update a portion (e.g., a block) of an unavailable reference frame;

[0014] Figure 2 shows two exemplary unavailable reference frame buffers, one for continuous mode and the other for reset mode;

[0015] Figure 3is a process flow diagram showing an exemplary process using a continuous mode according to some embodiments of the present subject matter;

[0016] Figure 4 is a process flow diagram showing an exemplary process using a reset mode according to some embodiments of the present subject matter;

[0017] Figure 5 is a process flow diagram showing an exemplary process for updating unavailable reference frames using a time filter according to some aspects of the present subject matter;

[0018] Figure 6 is a system block diagram showing an exemplary decoder capable of decoding a bitstream using an unavailable reference frame block update;

[0019] Figure 7 is a process flow diagram showing an exemplary process for encoding a video with unavailable reference frame block updates using multiple frames according to some aspects of the present subject matter, which exemplary process can improve compression efficiency;

[0020] Figure 8 is a system block diagram showing an exemplary video encoder capable of signaling unavailable reference frame block updates on the decoder side using multiple frames; and

[0021] Figure 9 is a block diagram of a computing system that can be used to implement any one or more of the methods and any one or more portions thereof disclosed herein.

[0022] The drawings are not necessarily to scale and may be illustrated by broken lines, schematic views, and partial views. In some cases, details unnecessary for understanding the embodiments or details that make other details difficult to understand may have been omitted. Like reference numerals in the various drawings denote like elements. Detailed Description

[0023] The embodiments described in this disclosure relate to the reception, update, and operation of unavailable reference frames. An unavailable reference (UR) frame is a frame and / or image that is used to create predicted frames and / or images within one or more groups of pictures (GOPs), but the frame and / or image itself is not displayed in the video image. A frame marked as a UR frame in a video bitstream can be used as a reference until the frame is explicitly removed by bitstream signaling. UR frames can improve prediction and compression efficiency in scenarios with a static background over a long period of time (e.g., the background of a video conference or a parking lot surveillance video). However, over time, the background of the scene gradually changes (e.g., a car becomes part of the background scene when it is parked in an empty space). Therefore, updating the UR frame can improve compression performance by allowing better prediction.

[0024] Current standards such as H.264 and H.265 allow the update of similar frames such as long-term reference (LTR) frames by signaling that a newly decoded frame is saved and available as a reference frame. This update is signaled by the encoder, and the entire frame is updated. However, updating the entire frame is costly, especially when only a small portion of the static background has changed.

[0025] Some existing compression techniques only use a portion of the previous frame to update frames such as LTR frames, which may lead to a degradation in prediction performance. Some embodiments of the present subject matter include using multiple video frames to update a portion (e.g., a block) of an LTR frame. For example, an LTR frame can be updated by applying a temporal filter to a buffer of decoded frames to compute statistics of co-located pixels. For example, the per-pixel mean, median, and / or mode of multiple decoded frames can be computed and used to update a portion of the LTR frame. In some embodiments, the present subject matter can support both a continuous mode and a reset mode. By using multiple frames to update frames such as UR and / or LTR frames instead of only using a portion of the previous frame, prediction can be improved, thereby reducing the residual and improving the bitrate performance.

[0026] Figure 1 is a process flow diagram illustrating an exemplary process 100 for using multiple video frames to update a portion (e.g., a block) of a UR frame. By using multiple frames to update a UR frame instead of only using a portion of the previous frame, prediction can be improved, thereby reducing the residual and improving the bitrate performance.

[0027] Still referring to Figure 1, at step 105, the decoder receives a bitstream. The bitstream can, for example, include but is not limited to: the data found in the bitstream, which is the input to the decoder when data compression is used. The bitstream can include information required to decode the video. Receiving can include extracting and / or parsing the blocks and associated signaling information in the bitstream. In some embodiments, the bitstream can include encoded video frames (which include encoded blocks). Each encoded block can include a coding tree unit (CTU), a coding unit (CU), and / or a prediction unit (PU).

[0028] At step 110, still referring to Figure 1 , a video frame is decoded from (e.g., using) the bitstream. For example, the video frame can be decoded by using inter-frame prediction. Decoding via inter-frame prediction can include using a previous frame, a future frame, or a UR frame as a reference for calculating the prediction, and the reference can be combined with the residuals included in the bitstream. Other techniques and tools for decoding, such as intra-frame prediction, can be utilized.

[0029] At step 115, still referring to Figure 1 , the decoder determines whether the unavailable reference block update mode is enabled for the current block in the bitstream. For example, it can be determined that the UR block update mode field in the header of the bitstream is enabled. Headers such as the picture parameter set (PPS) or the sequence parameter set (SPS) can be used to selectively enable the signaling of UR frame block updates. A field such as UR_BLOCK_UPDATE can take a true or false value (e.g., 0 or 1); the absence of this field in the header may imply a false value (e.g., 0). In some embodiments, the unavailable reference block update mode can be sent implicitly.

[0030] Still referring to Figure 1, at 120, the decoder uses the video frames to determine unavailable reference block updates. The UR block updates may include modifying pixel values of a portion of the UR frame. Determining the unavailable reference block updates may include calculating statistics of the collocated pixels of the plurality of video frames, for example, the statistics may include mean, median, and / or mode. Statistics may be calculated for a set of collocated pixels, such as a set of collocated pixels in multiple frames and / or blocks; as a non-limiting example, the mean of the luminance and / or chrominance of the pixels having a given set of fixed coordinates in a set of multiple frames may be calculated by collecting the chrominance and / or luminance values of the pixels in each frame at the fixed coordinates and then calculating the mean of the set of chrominance and / or luminance values. Statistics of one or more attributes of the pixels may be calculated. For example but not limited to, statistics of luminance may be calculated, for example, by calculating the mean, median, and / or mode of the luminance of the pixels in one or more of the plurality of video frames. As another non-limiting example, statistics of chrominance may be calculated, for example, by calculating the mean, median, and / or mode of the chrominance of the pixels in one or more of the plurality of video frames. In an embodiment, calculations using chrominance may be more computationally efficient than calculations using luminance, while calculations using luminance may be more accurate than calculations using chrominance. In an embodiment, the statistics may be directly used to update the pixel values in the updated UR frame and / or block; the statistics may represent the "closest" representation of the time series of the pixels because the statistics may result in a minimum amount of residual because the "average" pixel is statistically most similar to each individual pixel and has the smallest variance. This may minimize the residual, resulting in a higher degree of compression.

[0031] For example, as described in further detail below, a temporal filtering operation may be applied to the plurality of video frames to determine the unavailable reference block updates.

[0032] Still referring to Figure 1 , in some embodiments, a UR frame buffer may be employed. The UR updates may be constructed using operations on the frames present in the UR buffer. The UR buffer may include multiple frames. For example, Figure 2 Two exemplary embodiments of the UR frame buffer 200 are shown, where one is for the continuous mode and the other is for the reset mode.

[0033] In the continuous mode, Figure 3An exemplary process 300 is shown where the UR buffer can be updated periodically by removing a first frame from the buffer and adding the current frame (Ft) to the buffer, which is similar to a first-in-first-out (FIFO) queue. The current frame can include a newly decoded frame. The buffer can include a predefined length (e.g., a predefined maximum allowed number of frames) t-b. Each block filter can be updated; each block filter can be any filter suitable for filtering the blocks and / or frames described in this disclosure. The decoder can be configured to determine that the unavailable reference buffer has reached or exceeded a predefined maximum allowed buffer size and remove frames from the unavailable reference buffer. Each current frame can be added to the continuous mode buffer when being processed, and this process can be repeated for all frames. In some embodiments, the signal of the continuous mode can be sent in the bitstream.

[0034] In the reset mode, Figure 4 An exemplary process 400 is shown where the decoder can determine that the reset mode is enabled. The UR buffer can be updated in a manner similar to the continuous mode update described above, except that when a significant change is detected between two consecutive frames, a scene change can be detected and the UR buffer can be emptied. For example, the determination can be made by determining the inter-frame similarity between consecutive frames and comparing the similarity with a threshold. When the similarity value is below the threshold, a scene change can be detected. When the similarity value is above the threshold, a scene change may not be detected. Those skilled in the art will realize, after reading the entire disclosure, that various alternative or additional ways of performing such threshold comparisons can be implemented, including but not limited to comparing the difference with a threshold, where a difference above the threshold indicates a scene change and a difference below the threshold indicates that no scene change has occurred. In some embodiments, the scene reset and / or change can alternatively or additionally be signaled in the bitstream. The decoder may empty the UR buffer. After emptying the UR buffer, the buffer filling process can start from the current frame. In some embodiments, the reset mode can be signaled in the bitstream.

[0035] Therefore, still referring to Figure 4 , the continuous mode can allow a longer buffer and a wider reference sample set, and the UR frame update can be calculated based on the wider reference sample set. The reset mode can allow buffer size control based on frame similarity.

[0036] As Figure 2 shown, in the case of the reset mode, after each scene change, the size of the UR buffer can become smaller and then increase to the maximum size of b frames until a subsequent scene change occurs.

[0037] Figure 5is a process flowchart showing an exemplary process 500 for updating a UR frame using a time filter according to some aspects of the present subject matter. Video frames that have been divided into blocks can be filtered in time. Each block can have a size of width W and height H, where W can be equal to or not equal to H. A filter can be selected from a plurality of filters, which can include but are not limited to a set of three filters including median, mean, and mode. In a non-limiting example, the UR block update operation can be implemented based on the number of frames in a buffer. For example, at step 505, it can be determined whether there are no frames in the buffer; if so, at 510, the UR frame can be constructed by copying all blocks from the current frame. At 515, it can be determined whether there is one frame in the buffer, and at 520, the UR frame can be constructed by applying a filtering operation to matching (e.g., located at the same position) blocks in the UR buffer frame and the current frame. This operation can be implemented according to UR(Ft) = Filter_2(UR(Ft-l, Ft)). It can be determined at step 525 whether there is more than one frame in the buffer; at step 530, the filter can be applied to the previously calculated UR frame and the current frame according to the following formula: UR(Ft) = Filter_n(UR(Ft-l), Ft). At step 535, the UR update (e.g., the result of applying the filter as described above) can be saved to the UR frame. In some embodiments, the filtering operation can be selected based on performance requirements (such as processing speed, robustness to quantization and / or word length limitations, passband and / or stopband distortion and / or ripple, phase change and / or delay characteristics, attenuation level and / or speed and / or passband and stopband interconversion, etc.), and the signal of the filtering operation can be sent in the bitstream or any other suitable process for filtering selection.

[0038] In some embodiments, still referring to Figure 5 , the update can be enforced on the entire UR frame or one or more blocks of the UR frame. In some embodiments, per-pixel update is possible (e.g., the block can have a size of 1×1).

[0039] The filter can be calculated in the following ways, but is not limited thereto:

[0040] The median filter can be calculated according to the following:

[0041] Filter_2median(Px,y,t) = Px,y,t-1

[0042] Filter_nmedian(Px,y,t) = median(Px,y,t-1, Px,y,t-1,..., Px,y,t-n)

[0043] Among them, median() represents the median value of a sorted array of numbers.

[0044] The average filter can be calculated as follows:

[0045] Filter_2mean(Px,y,t) = Px,y,t-1

[0046]

[0047] The mode filter can be calculated as follows:

[0048] Filter_2mode(Px,y,t) = Px,y,t-1

[0049] Filter_nmode(Px,y,t) = mode(Px,y,t-1, Px,y,t-1, ···, Px,y,t-n)

[0050] Among them, mode() represents the value that appears most frequently in a given array of numbers.

[0051] Referring again to Figure 1 , at step 125, the decoder can update the unavailable reference frame using the determined UR block update. Such an update can include performing a UR block update, where the pixels (e.g., luminance values) spatially co-located with the current block for which the UR frame block update mode is enabled in the UR frame are updated (e.g., modified) using the determined UR block update (e.g., including the determined pixel values). The pixels can be updated in the following ways, but are not limited thereto: by replacing the pixels with co-located pixels from one or more blocks (including the current block) as described above, by updating the pixel values with values (e.g., chrominance and / or luminance values) from one or more blocks (including the current block) as described above, by replacing and / or updating the pixels according to and / or using statistical values (e.g., but not limited to the average value calculated for co-located pixels as described above), and / or by replacing and / or updating the pixels according to and / or using filter outputs (e.g., temporal filter outputs) as described in the present disclosure. In some embodiments, the update can include updating multiple UR frame blocks using multiple decoded blocks, which can be performed using any of the methods for updating UR frames using the decoded current block described above. The update can include, but is not limited to, using any of the above processes and / or process steps to generate blocks and / or frames with default chrominance and / or luminance values through one or more decoded blocks and replacing and / or updating the default chrominance and / or luminance values; the update can include generating a UR frame, which can be achieved by creating a UR frame with default values as described above and updating the default values.

[0052] Still referring to Figure 1, for a subsequent current block, the updated UR frame can be used as a reference frame for inter - frame prediction. For example, an encoded block can be received. It can be determined whether the inter - frame prediction mode is enabled for the encoded block. The updated UR frame can be used as a reference frame and the decoded block can be determined according to the inter - frame prediction mode. For example, decoding via inter - frame prediction can include using the updated UR frame as a reference for calculating the prediction, which can be combined with the residual included in the bitstream.

[0053] Continuing with reference 1, during the decoding process, UR block updates can be used for each current block. In some embodiments, for intra - coded frames, UR block updates can be implicitly skipped. For example, the bitstream can include a second current encoded block within a frame different from the first current encoded block. The mode of the second current encoded block can include intra - frame prediction. In response to determining that the second current encoded block is an intra - frame prediction block, the UR block update can be skipped. Skipping can include, for example, not updating the UR frame or determining whether a field such as UR_BLOCK_UPDATE is set in the header.

[0054] In some embodiments, continuing with reference Figure 1 , if the UR_BLOCK_UPDATE field is set in the header, the decoder can expect explicit block update signaling bits in the bitstream. In this method, the encoded blocks of the video can be signaled in the block header of the video bitstream as the blocks to be updated in the UR frame. In that case, the co - located blocks in the UR frame can be replaced with the result of temporal filtering of the frame buffer. As a non - limiting example, the updated UR frame can be used for future motion estimation purposes until the UR frame is subsequently updated.

[0055] Figure 6FIG. 0 is a system block diagram showing an exemplary decoder 600 capable of decoding a bitstream 604 using UR frame block updates. The decoder 600 may include an entropy decoding processor 608, an inverse quantization and inverse transform processor 612, a deblocking filter 616, a frame buffer 620, a motion compensation processor 624, and an intra prediction processor 628. In some embodiments, the bitstream 604 includes a parameter (e.g., a field in the header of the bitstream) that signals a UR frame block update mode. The motion compensation processor 624 may use the UR frame to reconstruct pixel information and update the UR frame using multiple frames according to the UR frame block update mode. For example, when the UR frame block update mode is signaled explicitly for a current block, the co-located pixels (e.g., luminance values) in the UR frame may be replaced with pixel values calculated by applying a temporal filter to frames in the UR frame buffer. Such an update may include performing a UR block update, where the decoded current block is used to update (e.g., modify) the co-located pixels in the UR frame that are spatially co-located with the current block. In some embodiments, the block update mechanism may be explicit or implicit. For example, the co-located pixels (e.g., luminance values) in the UR frame may be updated with the pixel values of the decoded current block, updating a portion (e.g., a block) of the UR frame; in other words, updating the UR frame may include replacing the luminance values of the UR frame with the luminance values that are spatially co-located with the decoded current block. In some embodiments, the UR frame may be updated according to another mechanism, such as a portion of the UR frame being updated to an average of the initial co-located UR frame pixel values and / or the current decoded block pixel values. In an example, each pixel being updated may be replaced and / or updated to a corresponding filter output (e.g., the result of filtering a set of co-located pixels). After reviewing the entire disclosure, those skilled in the art will recognize various other mechanisms that may be employed. In some embodiments, the update may include updating multiple UR frame blocks using multiple decoded blocks, which may be performed using any of the methods described above for updating the UR frame using the decoded current block. In some embodiments, the update may include updating multiple UR frame blocks using multiple decoded blocks, which may be performed using any of the methods described above for updating the UR frame using the decoded current block. The update may include, but is not limited to, generating a block and / or frame with default chrominance and / or luminance values using one or more decoded blocks using any of the processes and / or process steps described above, and replacing and / or updating the default chrominance and / or luminance values; the update may include generating a UR frame, which may be achieved by creating a UR frame with default values and updating the default values as described above.

[0056] In operation, still referring to Figure 6, the bitstream 604 can be received by the decoder 600 and input into the entropy decoding processor 608, which entropy decodes the bitstream 604 into quantized coefficients. The quantized coefficients are provided to the inverse quantization and inverse transform processor 612, which can perform inverse quantization and / or inverse transform to create a residual signal that can be added to the output of the motion compensation processor 624 or the intra prediction processor 628 according to the processing mode. The output of the motion compensation processor 624 and / or the intra prediction processor 628 can include block predictions based on previous decoded blocks or blocks of the UR frame. The sum of the prediction and the residual can be processed by the deblocking filter 616 and stored in the frame buffer 620. For a given block (e.g., CU or PU), when the bitstream 604 explicitly signals the enabling of the UR frame block update mode, the motion compensation processor 624 can update the UR frame, which can be included in the frame buffer 620, to update the co-located pixels (e.g., luminance values) in the UR frame based on pixel values calculated from multiple frames (e.g., frames included in the frame buffer 620). Pixel values (e.g., UR block update) can be determined by applying a temporal filter to the frames in the frame buffer 620. The temporal filter can alternatively or additionally be applied to blocks of the video frame, e.g., after segmentation, to calculate unavailable reference block updates; as a non-limiting example, one or more pixels and / or blocks can be replaced and / or updated by the blocks and / or pixels output by the temporal filter.

[0057] In some embodiments, continuing to refer to Figure 6 , the decoder 600 can include a UR frame block update processor 632 that generates a UR frame update based on the current block and provides UR frame pixel values for the inter prediction process. The UR frame block update processor 632 can directly affect motion compensation. Additionally, for example, when the current block is an intra prediction block, the UR frame update processor 632 can receive information from the intra prediction processor.

[0058] Figure 7FIG. 700 is a process flow diagram of an exemplary embodiment of a process for encoding a video with UR frame block updates using multiple frames according to some aspects of the present subject matter, which exemplary process may improve compression efficiency. At step 705, a video frame may undergo an initial block partitioning, e.g., partitioned using a tree-structured macroblock partitioning scheme, which may include dividing an image frame into coding tree units (CTUs) and coding units (CUs). At step 710, a block for updating a portion of the UR frame may be selected. For example, the block may be selected based on the frequency content and / or metric of the motion within the block as compared to one or more collocated blocks that are temporally adjacent (e.g., frames that are temporally adjacent within a predetermined number of frames of the current frame). The selection may include identifying, according to a metric rule, the block that will be used to update a portion of the UR frame at the decoder. At step 715, the block may be encoded and included in the bitstream. In some embodiments, the encoder-side UR frame may be updated using multiple frames, e.g., by applying a temporal filter to the frames within a frame buffer. The encoder-side UR frame may be used for motion estimation and compensation in the encoder.

[0059] At step 720, still referring to Figure 7 , explicit UR frame block update parameters may be determined and included in the bitstream to signal that the UR frame block update mode is enabled for the current block. For example, a field in the Picture Parameter Set (PPS) or Sequence Parameter Set (SPS) may be set (e.g., enabled). For example, a field such as the UR_BLOCK_UPDATE field may be set to indicate that the UR frame block update mode is enabled for the current block.

[0060] Figure 8 FIG. 800 is a system block diagram of an exemplary embodiment of a video encoder 800 capable of signaling decoder-side UR frame block updates using multiple frames. The video encoder 800 may receive an input video 804, which may be initially partitioned or divided according to a processing scheme such as a tree-structured macroblock partitioning scheme (e.g., quadtree binary decision tree). Examples of tree-structured macroblock partitioning schemes may include dividing an image frame into large block elements called coding tree units (CTUs). In some implementations, each CTU may be further divided one or more times into multiple sub-blocks called coding units (CUs). The final result of this partitioning may include a set of sub-blocks that may be referred to as prediction units (PUs). Transform units (TUs) may also be used.

[0061] Still referring to Figure 8, the exemplary video encoder 800 may include an intra prediction processor 808, a motion estimation / compensation processor 812 (also referred to as an inter prediction processor) capable of supporting UR frame block update at the decoder, a transform / quantization processor 816, an inverse quantization / inverse transform processor 820, a loop filter 824, a decoded picture buffer 828, and an entropy coding processor 832. In some embodiments, the motion estimation / compensation processor 812 may determine for a current block that the UR frame should be updated at the decoder and set parameters to explicitly signal an enabled UR frame block update mode. In some embodiments, implicit signaling may be performed. The bitstream parameter signaling the UR frame block update mode may be input to the entropy coding processor 832 to be included in the output bitstream 836. A portion of the encoder-side UR frame may be updated for use by the motion estimation / compensation processor 812 for encoding additional blocks that may utilize the UR frame as a reference for inter prediction. The encoder-side UR frame may be updated using multiple frames, e.g., by applying a temporal filter to frames within a frame buffer.

[0062] In operation, for each block of a frame of the input video 804, it may be determined whether to process the block via intra prediction or using motion estimation / compensation. The block may be provided to the intra prediction processor 808 or the motion estimation / compensation processor 812. If the block is to be processed via intra prediction, the intra prediction processor 808 may perform processing to output a predictor. If the block is to be processed via motion estimation / compensation, the motion estimation / compensation processor 812 may perform processing including using the encoder-side UR frame as a reference for inter prediction if applicable.

[0063] A residual can be formed by subtracting a predictor from an input video; the residual can be received by a transform / quantization processor 816, which can perform transform processing (e.g., discrete cosine transform (DCT)) to produce coefficients that can be quantized. The quantized coefficients and any associated signaling information can be provided to an entropy coding processor 832 for entropy coding and included in an output bitstream 836. The entropy coding processor 832 can support encoding of signaling information related to the UR frame block update mode. Additionally, the quantized coefficients can be provided to an inverse quantization / inverse transform processor 820, which can reproduce pixels that can be combined with the predictor and processed by a loop filter 824, and the output of the loop filter 824 can be stored in a decoded picture buffer 828 for use by a motion estimation / compensation processor 812, which is capable of supporting UR frame block update at the decoder. The decoded picture buffer 828 can include UR frames, and the motion estimation / compensation processor 812 can use the UR frames as references for inter prediction. Encoder-side UR frames can be updated using multiple frames, e.g., by applying a temporal filter to frames within a frame buffer.

[0064] Continuing to refer to Figure 8 , although some variations have been described in detail above, other modifications or additions are possible. For example, in some embodiments, a block can include any symmetric block (8×8, 16×16, 32×32, 64×64, 128×128, etc.) and any asymmetric block (8×4, 16×8, etc.).

[0065] In some embodiments, still referring to Figure 8 , a quadtree plus binary decision tree (QTBT) can be implemented. For the QTBT, at the coding tree unit level, the QTBT splitting parameters can be dynamically derived to adapt to local features without transmitting any overhead. Subsequently, at the coding unit level, a joint classifier decision tree structure can eliminate unnecessary iterations and control the risk of misprediction. In some embodiments, the UR frame block update mode can be available as an additional option at each leaf node of the QTBT.

[0066] In some embodiments, still referring to Figure 8, other syntax elements can be signaled at different levels of the bitstream. The UR frame block update can be enabled for an entire sequence by including an enable flag encoded in the Sequence Parameter Set (SPS). Additionally, the CTU flag can be encoded at the Coding Tree Unit (CTU) level to indicate whether any Coding Unit (CU) uses the UR frame block update mode. The CU flag can be encoded to indicate whether the current coding unit utilizes the UR frame block update mode. Although the above-described disclosed embodiments relate to the update of the UR frame, the above-described disclosed embodiments can alternatively or additionally be applied to other frames, images, including but not limited to long-term reference frames.

[0067] The subject matter described herein provides many technical advantages. For example, some implementations of the current subject matter can improve the bit rate of the bitstream by reducing the number of bits required to encode video. Such an improvement can be achieved by improving the UR frame, which in turn can improve prediction and reduce the residual. Additionally, in some implementations, the UR does not need to be updated as frequently, thereby reducing decoder computational requirements. In some implementations, the current subject matter does not increase the use of memory because the frame buffer can be utilized by other operations of the encoder and / or decoder. Some implementations of the current subject matter can provide for decoding blocks using the UR frame, which can include updating portions of the UR frame without having to update the entire UR frame. Such an approach can reduce complexity while increasing compression efficiency.

[0068] It should be noted that any one or more aspects and embodiments described herein can be conveniently implemented using digital electronic circuits, integrated circuits, application-specific integrated circuits (ASICs) designed specifically, field-programmable gate arrays (FPGAs), computer hardware, firmware, software, and / or combinations of the foregoing, as implemented and / or carried out in one or more machines programmed according to the teachings of this specification (e.g., one or more computing devices serving as user computing devices for electronic documents, one or more server devices such as document servers, etc.), which is obvious to those of ordinary skill in the computer field. These various aspects or features can include an implementation in one or more computer programs and / or software executable and / or interpretable on a programmable system, the programmable system including at least one programmable processor, which can be dedicated or general-purpose, and which is coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to send data and instructions to the storage system, at least one input device, and at least one output device. It is obvious to those of ordinary skill in the software field that a skilled programmer can readily prepare appropriate software code based on the teachings of this disclosure. The aspects and implementations using software and / or software modules discussed above can also include appropriate hardware for facilitating the implementation of the software and / or software modules with machine-executable instructions.

[0069] Such software can be a computer program product employing a machine-readable storage medium. A machine-readable storage medium can be any medium that can store and / or encode a sequence of instructions executable by a machine (e.g., a computing device) and that causes the machine to execute any of the methods and / or embodiments described herein. Examples of machine-readable storage media include, but are not limited to, magnetic disks, optical disks (e.g., CDs, CD-Rs, DVDs, DVD-Rs, etc.), magneto-optical disks, read-only memory "ROM" devices, random access memory "RAM" devices, magnetic cards, optical cards, solid-state storage devices, EPROMs, EEPROMs, programmable logic devices (PLDs), and / or any combination of the foregoing. The machine-readable media used herein are intended to include a single medium as well as a collection of physically separate media (e.g., a collection of optical disks, or one or more hard disk drives combined with computer memory). The machine-readable storage media used herein do not include transient forms of signal transmission.

[0070] Such software may also include information (e.g., data) carried as a data signal on a data carrier such as a carrier wave. For example, machine-executable information may be included as a data-bearing signal contained in a data carrier, where the signal encodes: a sequence of instructions or portions thereof to be executed by a machine (e.g., a computing device), and any associated information (e.g., data structures and data) that causes the machine to execute any one of the methods and / or embodiments described herein.

[0071] Examples of computing devices include, but are not limited to, e-reader devices, computer workstations, terminal computers, server computers, handheld devices (e.g., tablet computers, smart phones, etc.), network devices, network routers, network switches, bridges, any machine capable of executing a sequence of instructions specifying actions to be taken by it, and any combination of the foregoing. In one example, a computing device may include and / or be included in a kiosk.

[0072] Figure 9 FIG. shows a diagram of an embodiment of a computing device in an exemplary form of a computer system 900, in which a set of instructions can be executed to cause a control system to perform any one or more aspects and / or methods of the present disclosure. It is also contemplated that multiple computing devices may be utilized to implement a specifically configured instruction set for causing one or more of the devices to execute any one or more of the aspects and / or methods of the present disclosure. The computer system 900 includes a processor 904 and a memory 908 that communicate with each other and with other components via a bus 912. The bus 912 may include any one of a variety of types of bus structures (including, but not limited to, memory buses, memory controllers, peripheral buses, local buses, and any combination of the foregoing) using any one of a variety of bus architectures.

[0073] The memory 908 may include various components (e.g., machine-readable media), including but not limited to random access memory components, read-only components, and any combination of the foregoing. In one example, a basic input / output system 916 (BIOS) may be stored in the memory 908, which includes basic routines such as those that help transfer information between elements within the computer system 900 during startup. The memory 908 may also include instructions (e.g., software) 920 embodying any one or more of the aspects and / or methods of the present disclosure (e.g., stored on one or more machine-readable media). In another example, the memory 908 may further include any number of program modules, including but not limited to an operating system, one or more applications, other program modules, program data, and any combination of the foregoing.

[0074] The computer system 900 may also include a storage device 924. Examples of storage devices (e.g., storage device 924) include, but are not limited to, hard disk drives, disk drives, optical disc drives in combination with optical media, solid state storage devices, and any combination of the foregoing. The storage device 924 may be connected to the bus 912 via a suitable interface (not shown). Exemplary interfaces include, but are not limited to, SCSI, Advanced Technology Attachment (ATA), Serial ATA, Universal Serial Bus (USB), IEEE 1394 (FIREWIRE), and any combination of the foregoing. In one example, the storage device 924 (or one or more components thereof) may be removably connected to the computer system 900 (e.g., via an external port connector (not shown)). In particular, the storage device 924 and the associated machine-readable medium 928 may provide non-volatile and / or volatile storage of machine-readable instructions, data structures, program modules, and / or other data for the computer system 900. In one example, the software 920 may be stored, in whole or in part, within the machine-readable medium 928. In another example, the software 920 may be stored, in whole or in part, within the processor 904.

[0075] The computer system 900 may also include an input device 932. In one example, a user of the computer system 900 may input commands and / or other information into the computer system 900 via the input device 932. Examples of input devices 932 include, but are not limited to: alphanumeric input devices (e.g., keyboards), pointing devices, joysticks, gamepads, audio input devices (e.g., microphones, voice response systems, etc.), cursor control devices (e.g., mice), touchpads, optical scanners, video capture devices (e.g., still cameras, video cameras), touchscreens, and any combination of the foregoing. The input device 932 may be connected to the bus 912 via any one of a variety of interfaces (not shown), the variety of interfaces including, but not limited to, serial interfaces, parallel interfaces, game ports, USB interfaces, FireWire interfaces, direct interfaces to the bus 912, and any combination of the foregoing. The input device 932 may include a touchscreen interface, which may be part of or separate from the display 936, which will be discussed further below. The input device 932 may be used as a user selection device for selecting one or more graphical representations in a graphical interface as described above.

[0076] The user can also input commands and / or other information to the computer system 900 via a storage device 924 (e.g., a removable disk drive, a flash drive, etc.) and / or a network interface device 940. A network interface device such as the network interface device 940 can be used to connect the computer system 900 to one or more of various networks such as a network 944, and to one or more remote devices 948 connected to the network 944. Examples of network interface devices include, but are not limited to, network interface cards (e.g., mobile network interface cards, LAN cards), modems, and any combination thereof. Examples of networks include, but are not limited to, wide area networks (e.g., the Internet, enterprise networks), local area networks (e.g., networks associated with an office, a building, a campus, or other relatively small geographical spaces), telephone networks, data networks associated with telephone / voice providers (e.g., mobile communication provider data and / or voice networks), direct connections between two computing devices, and any combination of the foregoing. A network such as the network 944 can employ wired and / or wireless communication modes. Generally, any network topology can be used. Information (e.g., data, software 920, etc.) can be transmitted to and / or from the computer system 900 via the network interface device 940.

[0077] The computer system 900 can further include a video display adapter 952 for transmitting a displayable image to a display device such as a display device 936. Examples of display devices include, but are not limited to, liquid crystal displays (LCDs), cathode ray tubes (CRTs), plasma displays, light emitting diode (LED) displays, and any combination of the foregoing. The display adapter 952 and the display device 936 can be used in conjunction with the processor 904 to provide a graphical representation of various aspects of the present disclosure. In addition to the display device, the computer system 900 can include one or more other peripheral output devices, which include, but are not limited to, audio speakers, printers, and any combination of the foregoing. Such peripheral output devices can be connected to the bus 912 via a peripheral interface 956. Examples of peripheral interfaces include, but are not limited to, serial ports, USB connections, FireWire connections, parallel connections, and any combination of the foregoing.

[0078] Exemplary embodiments of the present invention have been described in detail above. Various modifications and additions can be made without departing from the gist and scope of the present invention. In order to provide various feature combinations in related new embodiments, the features of each of the above-described various embodiments can be appropriately combined with the features of other described embodiments. In addition, although many individual embodiments have been described above, the content described herein is merely an illustration of the application of the principles of the present invention. Further, although a particular method herein may be shown and / or described as being performed in a particular order, in the ordinary technology for implementing the embodiments disclosed herein, the order is highly variable. Accordingly, this description is meant as an example only and not to limit the scope of the present invention.

[0079] In the above description and claims, a list of elements or features may follow phrases such as "at least one" or "one or more". The term "and / or" may also appear in a list of two or more elements or features. Unless implicitly or explicitly contradicted by the context in which the phrase is used, such phrases are intended to mean any element or feature listed individually, or any combination of any listed element or feature with any other listed element or feature. For example, the phrases "at least one of A and B", "one or more of A and B", and "A and / or B" are each intended to mean "A alone, B alone, or both A and B". Similar interpretations apply to lists containing three or more items. For example, the phrases "at least one of A, B, and C", "one or more of A, B, and C", "A, B, and / or C" are each intended to mean "A alone, B alone, C alone, both A and B, both A and C, both B and C, or A, B, and C". Further, the term "based on" as used above and in the claims is intended to mean "at least partially based on", such that unrecited features or elements are also permitted.

[0080] The subject matter described herein can be embodied in a system, apparatus, method, and / or article according to desired configurations. The embodiments set forth in the foregoing description do not represent all embodiments consistent with the subject matter described herein. Rather, they are only some examples consistent with aspects related to the subject matter described. Although some variations have been described in detail above, other modifications or additions are possible. In particular, further features and / or variations can be provided in addition to the features and / or variations set forth herein. For example, the above-described embodiments can be directed to various combinations and sub-combinations of the disclosed features and / or combinations and sub-combinations of several other features disclosed above. Further, the logical flows described in the figures and / or herein do not necessarily require the particular order or sequence shown to achieve the desired result. Other embodiments are within the scope of the claims.

Claims

1. A method for encoding a video signal, comprising: Receiving a video signal; Generating an encoded bitstream, the encoded bitstream including signaling information, encoded pictures serving as reference pictures, a picture sequence including a first current encoded block and a second current encoded block, the signaling information including a sequence parameter set (SPS), the sequence parameter set indicating an enabled mode, wherein in the mode, a part of a stored reference picture can be changed, and wherein the encoded bitstream is further configured to be decoded by a decoder, the decoder receiving the bitstream and performing the following steps: Decoding the encoded picture and storing the picture as a reference picture; Changing a part of the reference picture by using the mode indicated as enabled in the SPS; Forming a predictor for the first current encoded block by using the changed part of the reference picture; Reconstructing the first current encoded block by adding residual pixels to the predictor; And Reconstructing the second current encoded block without using any changed part of the reference picture.

2. The method according to claim 1, wherein the second current encoded block is an intra-coded block.

3. The method according to claim 2, wherein the reference picture is a reference picture not output for display.

Images