System and method for window error estimation improvement in temporal filters

Abstract

Various implementations described herein include methods and systems for encoding video. In one aspect, a method includes receiving video data including a current block, where the current block has a size of N * M pixels, N and M being positive integers. The method includes: generating, for a current block, a prediction block larger than the current block, where the prediction block has a size of (N + W) x (M + W) pixels, W being a positive integer; the method includes determining a window error of a sample window of a sample of the current block using the generated prediction block.

Description

[0001] Related Applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 459,563, filed on April 14, 2023, entitled “Window Error Estimation Improvement in Temporal Filter,” and this application is a continuation of and claims priority to U.S. Patent Application No. 18 / 494,684, filed on October 25, 2023, entitled “Systems and Methods for Window Error Estimation Improvement in Temporal Filter.” Technical Field

[0003] The disclosed embodiments relate generally to video coding, including but not limited to systems and methods for improving window error estimation in temporal filters. Background Art

[0004] Digital video is supported by various electronic devices such as digital televisions, laptop or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video game consoles, smart phones, video teleconferencing devices, video streaming devices, etc. Electronic devices send and receive or otherwise transmit digital video data across communication networks, and / or store digital video data on storage devices. Due to the limited bandwidth capacity of communication networks and the limited memory resources of storage devices, video encoding can be used to compress video data according to one or more video encoding standards before transmitting or storing video data. Video encoding can be performed by hardware and / or software on electronic / client devices or servers that provide cloud services.

[0005] Video coding typically uses prediction methods (e.g., inter-frame prediction, intra-frame prediction, etc.) that exploit the redundancy inherent in video data. Video coding aims to compress video data into a form that uses a lower bit rate while avoiding or minimizing degradation of video quality. A variety of video codec standards have been developed. For example, High-Efficiency Video Coding (HEVC / H.265) is a video compression standard designed as part of the MPEG-H (Moving Picture Experts Group-H, MPEG-H) project. ITU-T (International Telecommunication Union-Telecommunication Standardization Sector, ITU-T) and ISO / IEC (International Organization for Standardization / International Electrotechnical Commission, ISO / IEC) released the HEVC / H.265 standard in 2013 (version 1), 2014 (version 2), 2015 (version 3), and 2016 (version 4). Versatile Video Coding (VVC / H.266) is a video compression standard intended as a successor to HEVC. The VVC / H.266 standard was released by ITU-T and ISO / IEC in 2020 (version 1) and 2022 (version 2). AOMedia Video 1 (Alliance for Open Media Video 1, AV1) is an open video coding format designed as an alternative to HEVC. On January 8, 2019, the verified version 1.0.0 with specification errata 1 was released. Summary of the invention

[0006] The present disclosure describes an improved window error estimation in a temporal filter. For example, the improved window error estimation takes into account the partitioning of a block to be temporally filtered. The temporal filtered samples in a filtered block are obtained by applying corresponding weights to different prediction samples. The corresponding weights applied to the corresponding prediction samples depend on both the window error and the block error associated with the corresponding prediction samples. When the block to be filtered does not contain a partition, additional neighboring samples are generated to provide enough samples for better window error estimation. When the block to be filtered contains a partition, error clipping is applied at the partition level rather than the block level to take into account the impact of different motion vectors at the partition boundaries. Additionally or alternatively, additional neighboring samples are generated for a specific partition to provide enough samples for better window error estimation.

[0007] According to some embodiments, a method of video encoding is provided. The method includes: (a) receiving video data including a current block, wherein the current block has a size of N×M pixels, N and M are positive integers; (b) generating a prediction block (e.g., an enlarged prediction block) larger than the current block for the current block, wherein the prediction block has a size of (N+W)x(M+W) pixels, W is a positive integer; (c) using the generated prediction block (e.g., the enlarged prediction block) to determine a window error of a sample window of samples of the current block.

[0008] According to some embodiments, a method of video encoding includes: (i) receiving video data including a current block, wherein the current block has two or more partitions and each of the two or more partitions corresponds to a different predictor; (ii) identifying a partition in which a sample in the two or more partitions is located; (iii) generating a sample window for the sample; (iv) performing error clipping on one or more window samples located outside the identified partition and outside a boundary of the current block; and (v) determining a window error of the sample window after performing error clipping.

[0009] According to some embodiments, a method of video encoding includes: (i) receiving video data including a current block, wherein the current block has two or more partitions and each of the two or more partitions corresponds to a different predictor; (ii) identifying a partition in which a sample of the two or more partitions is located, wherein the partition has a size of N×M pixels, N and M are positive integers; (iii) generating a prediction block (e.g., an enlarged prediction block) larger than the current partition for the partition, wherein the prediction block has a size of (N+W)x(M+W) pixels, W is a positive integer, and wherein the prediction block is generated using motion vector information of the partition; and (iv) determining a window error of a sample window of the sample using the generated prediction block (e.g., the enlarged prediction block).

[0010] According to some embodiments, a computing system is provided, such as a streaming system, a server system, a personal computer system or other electronic device. The computing system includes a control circuit system and a memory storing one or more groups of instructions. One or more groups of instructions include instructions for performing any of the methods described herein. In some embodiments, the computing system includes an encoder component and a decoder component (e.g., a transcoder).

[0011] According to some embodiments, a non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium stores one or more sets of instructions for execution by a computing system. The one or more sets of instructions include instructions for executing any of the methods described herein.

[0012] Thus, devices and systems utilizing methods for encoding and decoding video are disclosed. Such methods, devices, and systems may supplement or replace conventional methods, devices, and systems for video encoding / decoding. The features and advantages described in the specification are not necessarily all inclusive, and in particular, some additional features and advantages will be apparent to one of ordinary skill in the art in view of the drawings, description, and claims provided in the present disclosure. Furthermore, it should be noted that the language used in the specification is selected primarily for readability and instructional purposes and is not necessarily selected to describe or limit the subject matter described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order that the present disclosure may be understood in more detail, a more specific description may be made by reference to the features of various embodiments, some of which are illustrated in the accompanying drawings. However, the accompanying drawings illustrate only the relevant features of the present disclosure and are therefore not necessarily to be considered limiting, as the description may allow for other effective features that will be understood by those skilled in the art upon reading the present disclosure.

[0014] Figure 1 is a block diagram illustrating an example communication system in accordance with some implementations.

[0015] Figure 2A is a block diagram illustrating example elements of an encoder component according to some implementations.

[0016] Figure 2B is a block diagram illustrating example elements of a decoder component according to some implementations.

[0017] Figure 3 is a block diagram illustrating an example server system according to some implementations.

[0018] Figure 4An example temporal filtering process is shown in accordance with some implementations.

[0019] Figure 5A A prediction process according to some embodiments is shown.

[0020] Figure 5B A scenario for determining a window error associated with a block that does not include any partitions is shown in accordance with some embodiments.

[0021] Figure 5C An example scenario for determining a window error associated with a block comprising two or more partitions is shown in accordance with some embodiments.

[0022] Figure 5D Additional example scenarios for determining a window error associated with a block comprising two or more partitions are shown in accordance with some embodiments.

[0023] FIG. 6A to FIG. 6B is a flow chart illustrating an example method of encoding a video according to some implementations.

[0024] According to common practice, the various features shown in the drawings are not necessarily drawn to scale, and like reference numerals may be used to denote like features throughout the specification and drawings. DETAILED DESCRIPTION

[0025] The present disclosure describes a set of advanced video coding techniques for improving weight estimation of temporal filtering by considering partitions of blocks to be temporally filtered. Temporal filtered samples in a filtered block are obtained by applying corresponding weights to different prediction samples. The corresponding weights applied to the corresponding prediction samples depend on both the window error and the block error associated with the corresponding prediction samples. Not considering the partition scheme in the block may lead to inaccurate window errors. As described in more detail later, when the block to be filtered does not contain partitions, additional neighboring samples are generated to provide enough samples for better window error estimation. When the block to be filtered contains partitions, error clipping is applied at the partition level rather than the block level to take into account the impact of different motion vectors at the partition boundaries.

[0026] Example Systems and Devices

[0027] Figure 11 is a block diagram illustrating a communication system 100 according to some embodiments. The communication system 100 includes a source device 102 and a plurality of electronic devices 120 (e.g., electronic device 120-1 to electronic device 120-m) communicatively coupled to each other via one or more networks. In some embodiments, the communication system 100 is a streaming system, for example, for use with video-enabled applications such as video conferencing applications, digital TV applications, and media storage and / or distribution applications.

[0028] The source device 102 includes a video source 104 (e.g., a camera component or a media storage device) and an encoder component 106. In some embodiments, the video source 104 is a digital camera (e.g., configured to create an uncompressed video sample stream). The encoder component 106 generates one or more encoded video bitstreams based on the video stream. The video stream from the video source 104 can be a high data volume compared to the encoded video bitstream 108 generated by the encoder component 106. Because the encoded video bitstream 108 is a lower data volume (less data) than the video stream from the video source, the encoded video bitstream 108 requires less bandwidth to transmit and requires less storage space to store than the video stream from the video source 104. In some embodiments, the source device 102 does not include the encoder component 106 (e.g., configured to transmit uncompressed video data to the network 110).

[0029] One or more networks 110 represent any number of networks that transmit information between source device 102, server system 112, and / or electronic device 120, including, for example, wired (wired) and / or wireless communication networks. One or more networks 110 can exchange data in circuit switching channels and / or packet switching channels. Representative networks include telecommunication networks, local area networks, wide area networks, and / or the Internet.

[0030] One or more networks 110 include a server system 112 (e.g., a distributed / cloud computing system). In some embodiments, the server system 112 is a streaming server or includes a streaming server (e.g., configured to store and / or distribute video content such as an encoded video stream from a source device 102). The server system 112 includes a decoder component 114 (e.g., configured to encode and / or decode video data). In some embodiments, the decoder component 114 includes an encoder component and / or a decoder component. In various embodiments, the decoder component 114 is instantiated as hardware, software, or a combination thereof. In some embodiments, the decoder component 114 is configured to decode the encoded video bitstream 108 and re-encode the video data using different encoding standards and / or methods to generate encoded video data 116. In some embodiments, the server system 112 is configured to generate multiple video formats and / or encodings based on the encoded video bitstream 108. In some embodiments, the server system 112 is used as a Media-Aware Network Element (MANE). For example, the server system 112 can be configured to prune the encoded video bitstream 108 to tailor a potentially different bitstream for one or more of the electronic devices 120. In some implementations, the MANE is provided separately from the server system 112.

[0031] The electronic device 120-1 includes a decoder component 122 and a display 124. In some embodiments, the decoder component 122 is configured to decode the encoded video data 116 to generate an outgoing video stream that can be rendered on a display or other type of rendering device. In some embodiments, one or more of the electronic devices 120 do not include a display component (e.g., communicatively coupled to an external display device and / or include a media storage device). In some embodiments, the electronic device 120 is a streaming client. In some embodiments, the electronic device 120 is configured to access the server system 112 to obtain the encoded video data 116.

[0032] The source device and / or multiple electronic devices 120 are sometimes referred to as “end devices” or “user devices.” In some embodiments, one or more of the electronic devices 120 and / or the source device 102 are examples of server systems, personal computers, portable devices (e.g., smart phones, tablet computers, or laptop computers), wearable devices, video conferencing devices, and / or other types of electronic devices.

[0033] In an example operation of the communication system 100, the source device 102 transmits a coded video bitstream 108 to a server system 112. For example, the source device 102 may encode a picture stream captured by the source device. The server system 112 receives the coded video bitstream 108 and may decode and / or encode the coded video bitstream 108 using a decoder component 114. For example, the server system 112 may apply a coding that is more optimized for network transmission and / or storage to the video data. The server system 112 may transmit the coded video data 116 (e.g., one or more coded video bitstreams) to one or more of the electronic devices 120. Each electronic device 120 may decode the coded video data 116 and optionally display the video picture.

[0034] Figure 2A 1 is a block diagram illustrating example elements of an encoder component 106 according to some embodiments. The encoder component 106 receives a source video sequence from a video source 104. In some embodiments, the encoder component includes a receiver (e.g., a transceiver) component configured to receive the source video sequence. In some embodiments, the encoder component 106 receives the video sequence from a remote video source (e.g., a video source that is a component of a device different from the encoder component 106). The video source 104 can provide the source video sequence in the form of a digital video sample stream, which can have any suitable bit depth (e.g., 8 bits, 10 bits, or 12 bits), any color space (e.g., BT.601 Y CrCb or RGB), and any suitable sampling structure (e.g., Y CrCb 4:2:0 or Y CrCb 4:4:4). In some embodiments, the video source 104 is a storage device that stores previously captured / prepared videos. In some embodiments, the video source 104 is a camera that captures local image information as a video sequence. The video data may be provided as a plurality of individual pictures that are imparted with motion when viewed sequentially. The pictures themselves may be organized as a spatial array of pixels, wherein each pixel may include one or more samples, depending on the sampling structure, color space, etc. used. The relationship between pixels and samples may be readily understood by one of ordinary skill in the art. The following description focuses on samples.

[0035] The encoder component 106 is configured to encode and / or compress the pictures of the source video sequence into an encoded video sequence 216 in real time or under other time constraints required by the application. Implementing the appropriate encoding speed is a function of the controller 204. In some embodiments, the controller 204 controls other functional units as described below and is functionally coupled to the other functional units. The parameters set by the controller 204 may include rate control related parameters (e.g., picture skipping, quantizer and / or lambda value of rate distortion optimization techniques), picture size, picture group (Group Of Picture, GOP) layout, maximum motion vector search range, etc. Other functions of the controller 204 can be easily identified by those of ordinary skill in the art, as the functions may belong to the encoder component 106 optimized for a specific system design.

[0036] In some embodiments, the encoder component 106 is configured to operate in an encoding loop. In a simplified example, the encoding loop includes a source encoder 202 (e.g., responsible for creating symbols, such as a symbol stream, based on an input picture to be encoded and a reference picture) and a (local) decoder 210. The decoder 210 reconstructs the symbols to create sample data in a similar manner to the (remote) decoder (in the case where the compression between the symbols and the encoded video bitstream is lossless). The reconstructed sample stream (sample data) is input to the reference picture memory 208. Since the decoding of the symbol stream produces a bit-accurate result that is independent of the decoder location (local or remote), the contents of the reference picture memory 208 are also bit-accurate between the local encoder and the remote encoder. In this way, the reference picture samples interpreted by the prediction part of the encoder are the same sample values ​​that the decoder will interpret when using prediction during decoding. The principles of reference picture synchronization (and the drift that occurs when synchronization cannot be maintained, for example due to channel errors) are known to those of ordinary skill in the art.

[0037] The operation of decoder 210 may be combined with a remote decoder, for example, as follows Figure 2B The operation of the decoder 210 described in detail is the same. However, briefly referring to Figure 2B , since the symbols are available and the encoding of the symbols into an encoded video sequence by the entropy encoder 214 and the decoding of the symbols by the parser 254 can be lossless, the entropy decoding portion of the video decoder component 122 including the buffer memory 252 and the parser 254 may not be fully implemented in the local decoder 210.

[0038] In addition to parsing / entropy decoding, the decoder techniques described herein can exist in the corresponding encoder in the form of substantially the same functions. For this reason, the disclosed subject matter focuses on the decoder operation. The description of the encoder techniques can be simplified because the encoder techniques can be reciprocal with the decoder techniques.

[0039] As part of its operation, the source encoder 202 can perform motion compensated predictive coding, which predictively encodes an input frame with reference to one or more previously encoded frames from a video sequence that are designated as reference frames. In this manner, the encoding engine 212 encodes the differences between pixel blocks of an input frame and pixel blocks of a reference frame that can be selected as a prediction reference for the input frame. The controller 204 can manage the encoding operations of the source encoder 202, including, for example, setting parameters and subgroup parameters for encoding video data.

[0040] The decoder 210 may decode the encoded video data of a frame that may be designated as a reference frame based on the symbols created by the source encoder 202. The operation of the encoding engine 212 may advantageously be a lossy process. Figure 2A The reconstructed video sequence may be a copy of the source video sequence with some errors when decoded at a remote video decoder (not shown). Decoder 210 replicates the decoding process that may be performed on the reference frame by the remote video decoder and may cause the reconstructed reference frame to be stored in reference picture memory 208. In this manner, encoder component 106 locally stores a copy of the reconstructed reference frame that has common content (absent transmission errors) with the reconstructed reference frame to be obtained by the remote video decoder.

[0041] The predictor 206 may perform a prediction search for the encoding engine 212. That is, for a new frame to be encoded, the predictor 206 may search the reference picture memory 208 for sample data (as candidate reference pixel blocks) or specific metadata such as reference picture motion vectors, block shapes, etc. that may be used as appropriate prediction references for the new picture. The predictor 206 may operate on a pixel block by pixel block basis to find an appropriate prediction reference. As determined by the search results obtained by the predictor 206, the input picture may have prediction references taken from multiple reference pictures stored in the reference picture memory 208.

[0042] The outputs to all of the above-mentioned functional units may be subjected to entropy encoding in the entropy encoder 214. The entropy encoder 214 converts the symbols, as generated by the various functional units, into an encoded video sequence by losslessly compressing the symbols according to techniques known to those of ordinary skill in the art (e.g., Huffman encoding, variable length encoding, and / or arithmetic coding).

[0043] In some embodiments, the output of the entropy encoder 214 is coupled to a transmitter. The transmitter can be configured to buffer the encoded video sequence as created by the entropy encoder 214 in preparation for transmission via a communication channel 218, which can be a hardware / software link to a storage device that will store the encoded video data. The transmitter can be configured to merge the encoded video data from the source encoder 202 with other data to be transmitted, such as encoded audio data and / or an auxiliary data stream (source not shown). In some embodiments, the transmitter can transmit additional data along with the encoded video. The source encoder 202 can include such data as part of the encoded video sequence. The additional data can include time / space / SNR (Signal-to-Noise Ratio, SNR) enhancement layers, other forms of redundant data such as redundant pictures and slices, Supplementary Enhancement Information (SEI) messages, Visual Usability Information (VUI) parameter set fragments, etc.

[0044] The controller 204 can manage the operation of the encoder component 106. During encoding, the controller 204 can assign a certain encoded picture type to each encoded picture, which may affect the encoding technology applied to the corresponding picture. For example, a picture can be assigned as an intra picture (I picture), a predictive picture (P picture), or a bidirectional predictive picture (B picture). Intra pictures can be encoded and decoded without using any other frame in the sequence as a prediction source. Some video codecs allow different types of intra pictures, including, for example, independent decoder refresh (IDR) pictures. Those of ordinary skill in the art understand those variations of I pictures and their corresponding applications and features, and therefore are not repeated here. Predictive pictures can be encoded and decoded using inter-frame prediction or intra-frame prediction that predicts sample values ​​for each block using at most one motion vector and reference index. Bidirectional predictive pictures can be encoded and decoded using inter-frame prediction or intra-frame prediction that predicts sample values ​​for each block using at most two motion vectors and reference indices. Similarly, multi-predictive pictures can use more than two reference pictures and associated metadata for reconstruction of a single block.

[0045] The source picture may typically be spatially subdivided into blocks of samples (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples, respectively), and encoded block by block. The blocks may be predictively encoded with reference to other (already encoded) blocks, which are determined by the coding allocation applied to the corresponding picture of the block. For example, blocks of an I picture may be non-predictively encoded, or blocks of an I picture may be predictively encoded (spatial prediction or intra-frame prediction) with reference to already encoded blocks of the same picture. Blocks of pixels of a P picture may be non-predictively encoded via spatial prediction or via temporal prediction with reference to one previously encoded reference picture. Blocks of a B picture may be non-predictively encoded via spatial prediction or via temporal prediction with reference to one or two previously encoded reference pictures.

[0046] Video can be captured as multiple source pictures (video pictures) in a temporal sequence. Intra-picture prediction (often simplified to intra-prediction) exploits spatial correlations in a given picture, while inter-picture prediction exploits (temporal or other) correlations between pictures. In an example, a particular picture in encoding / decoding, which is referred to as the current picture, is divided into blocks. In the case where a block in the current picture is similar to a reference block in a reference picture that has been previously encoded in the video and is still buffered, the block in the current picture can be encoded by a vector called a motion vector. The motion vector points to a reference block in a reference picture, and in the case where multiple reference pictures are used, the motion vector may have a third dimension that identifies the reference picture.

[0047] The encoder component 106 may perform encoding operations according to a predetermined video encoding technique or standard, such as any described herein. In operation of the encoder component 106, the encoder component 106 may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in an input video sequence. Thus, the encoded video data may conform to a syntax specified by the video encoding technique or standard used.

[0048] Figure 2B is a block diagram illustrating example elements of decoder component 122 according to some implementations. Figure 2B The decoder component 122 in is coupled to the channel 218 and the display 124. In some embodiments, the decoder component 122 includes a transmitter coupled to the loop filter 256 and configured to transmit data to the display 124 (eg, via a wired connection or a wireless connection).

[0049] In some embodiments, the decoder component 122 includes a receiver coupled to the channel 218 and configured to receive data from the channel 218 (e.g., via a wired connection or a wireless connection). The receiver can be configured to receive one or more encoded video sequences decoded by the decoder component 122. In some embodiments, the decoding of each encoded video sequence is independent of other encoded video sequences. Each encoded video sequence can be received from the channel 218, which can be a hardware / software link to a storage device storing the encoded video data. The receiver can receive the encoded video data and other data, such as encoded audio data and / or auxiliary data streams, which can be forwarded to their respective use entities (not depicted). The receiver can separate the encoded video sequence from the other data. In some embodiments, the receiver receives additional (redundant) data along with the encoded video. The additional data can be included as part of the encoded video sequence. The additional data can be used by the decoder component 122 to decode the data and / or more accurately reconstruct the original video data. The additional data can be in the form of, for example, time, space or SNR enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0050] According to some embodiments, decoder component 122 includes a buffer memory 252, a parser 254 (sometimes also referred to as an entropy decoder), a scaler / inverse transform unit 258, an intra picture prediction unit 262, a motion compensated prediction unit 260, an aggregator 268, a loop filter unit 256, a reference picture memory 266, and a current picture memory 264. In some embodiments, decoder component 122 is implemented as an integrated circuit, a series of integrated circuits, and / or other electronic circuit systems. In some embodiments, decoder component 122 is at least partially implemented in software.

[0051] Buffer memory 252 is coupled between channel 218 and parser 254 (e.g., to combat network jitter). In some embodiments, buffer memory 252 is separate from decoder component 122. In some embodiments, a separate buffer memory is provided between the output of channel 218 and decoder component 122. In some embodiments, in addition to buffer memory 252 internal to decoder component 122 (e.g., which is configured to handle playout timing), a separate buffer memory is provided external to decoder component 122 (e.g., to combat network jitter). When receiving data from a store / forward device with sufficient bandwidth and controllability or from a synchronous network, buffer memory 252 may not be required or may be relatively small. In order to maximize the use of packet networks such as the Internet, buffer memory 252 may be required, which buffer memory 252 may be relatively large and may advantageously have an adaptive size and may be implemented at least in part in an operating system or similar element (not depicted) external to decoder component 122.

[0052] The parser 254 is configured to reconstruct the symbol 270 from the coded video sequence. The symbol may include, for example, information for managing the operation of the decoder component 122, and / or information for controlling a rendering device such as the display 124. The control information for the rendering device may be in the form of, for example, a supplemental enhancement information (SEI) message or a video usability information (VUI) parameter set fragment (not depicted). The parser 254 parses (entropy decodes) the coded video sequence. The encoding of the coded video sequence may be performed according to a video coding technique or standard, and may follow principles known to those skilled in the art, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser 254 may extract a subgroup parameter set for at least one of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroup may include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. Parser 254 may also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0053] Depending on the type of the coded video picture or portion (e.g., inter-picture and intra-picture, inter-block and intra-block) and other factors, the reconstruction of the symbol 270 may involve multiple different units. Which units are involved and how they are involved can be controlled by the parser 254 through subgroup control information parsed from the coded video sequence. For the sake of brevity, such subgroup control information flow between the parser 254 and the following multiple units is not depicted.

[0054] The decoder component 122 may be conceptually subdivided into a plurality of functional units, and in some implementations, these units closely interact with each other and may be at least partially integrated with each other. However, for the sake of brevity, the conceptual subdivision of the functional units is maintained herein.

[0055] The sealer / inverse transform unit 258 receives the quantized transform coefficients and control information (such as which transform to use, block size, quantization factor, and / or quantization scaling matrix) as symbols 270 from the parser 254. The sealer / inverse transform unit 258 may output a block including sample values, which may be input into the aggregator 268.

[0056] In some cases, the output samples of the sealer / inverse transform unit 258 belong to an intra-coded block; that is, a block that does not use predictive information from a previously reconstructed picture, but can use predictive information from a previously reconstructed portion of the current picture. Such predictive information can be provided by the intra-picture prediction unit 262. The intra-picture prediction unit 262 can use the surrounding reconstructed information obtained from the current (partially reconstructed) picture from the current picture memory 264 to generate a block of the same size and shape as the block under reconstruction. The aggregator 268 can add the prediction information that has been generated by the intra-picture prediction unit 262 to the output sample information as provided by the sealer / inverse transform unit 258 on a per-sample basis.

[0057] In other cases, the output samples of the scaler / inverse transform unit 258 belong to a block that is inter-coded and potentially motion compensated. In such a case, the motion compensated prediction unit 260 can access the reference picture memory 266 to obtain samples for prediction. After the obtained samples are motion compensated according to the symbols 270 belonging to the block, these samples can be added to the output of the scaler / inverse transform unit 258 (in this case, referred to as residual samples or residual signals) by the aggregator 268 to generate output sample information. The address within the reference picture memory 266 from which the motion compensated prediction unit 260 obtains the predicted samples can be controlled by a motion vector. The motion vector can be available to the motion compensated prediction unit 260 in the form of a symbol 270, which can have, for example, an X component, a Y component, and a reference picture component. Motion compensation can also include interpolation of sample values ​​such as those obtained from the reference picture memory 266 when using sub-sample accurate motion vectors, motion vector prediction mechanisms, etc.

[0058] The output samples of the aggregator 268 may be subjected to various loop filtering techniques in the loop filter unit 256. The video compression techniques may include in-loop filter techniques controlled by parameters included in the coded video bitstream and available to the loop filter unit 256 as symbols 270 from the parser 254, but the video compression techniques may also be responsive to meta information obtained during decoding of a coded picture or a previous (in decoding order) portion of a coded video sequence, as well as to previously reconstructed and loop filtered sample values. The output of the loop filter unit 256 may be a sample stream that may be output to a rendering device such as the display 124, as well as stored in the reference picture memory 266 for use in future inter-picture prediction.

[0059] Once reconstructed, certain coded pictures may be used as reference pictures for future prediction. Once a coded picture is reconstructed, and the coded picture has been identified as a reference picture (e.g., by parser 254), the current reference picture may become part of reference picture memory 266, and a new current picture memory may be reallocated before starting reconstruction of a subsequent coded picture.

[0060] The decoder component 122 may perform decoding operations according to a predetermined video compression technique that may be recorded in a standard such as any standard described herein. As specified in a video compression technology document or standard and particularly in a configuration file therein, the coded video sequence may conform to the syntax specified by the video compression technology or standard used in the sense that the coded video sequence follows the syntax of the video compression technology or standard. In addition, in order to conform to some video compression technologies or standards, the complexity of the coded video sequence may be within a range as defined by the hierarchy of the video compression technology or standard. In some cases, the hierarchy limits the maximum picture size, the maximum frame rate, the maximum reconstruction sample rate (measured in, for example, mega samples per second), the maximum reference picture size, etc. In some cases, the limits set by the hierarchy may be further defined by a Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management signaled in the coded video sequence.

[0061] Figure 3 is a block diagram illustrating a server system 112 according to some embodiments. The server system 112 includes a control circuit system 302, one or more network interfaces 304, a memory 314, a user interface 306, and one or more communication buses 312 for interconnecting these components. In some embodiments, the control circuit system 302 includes one or more processors (e.g., a CPU (Central Processing Unit, CPU), a GPU (Graphics Processing Unit, GPU) and / or a DPU (Data Processing Unit, DPU)). In some embodiments, the control circuit system includes one or more field programmable gate arrays (Field-Programmable Gate Array, FPGA), a hardware accelerator and / or one or more integrated circuits (e.g., an application-specific integrated circuit).

[0062] The network interface 304 can be configured to interface with one or more communication networks (e.g., wireless networks, wired networks, and / or optical networks). The communication networks can be local, wide area, metropolitan area network, vehicle and industrial, real-time, delay tolerant, etc. Examples of communication networks include: local area networks, such as Ethernet, wireless LAN (Local Area Network, LAN); cellular networks including GSM (Global System for Mobile Communications, GSM), 3G (the Third Generation, 3G), 4G (the Fourth Generation, 4G), 5G (the Fifth Generation, 5G), LTE (Long Term Evolution, LTE), etc.; TV wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV; vehicle and industrial networks including CANBus (Controller Area Network-BUS, CANBus), etc. Such communications can be one-way reception only (e.g., broadcast TV), one-way transmission only (e.g., CAN bus to certain CAN bus devices), or bidirectional (e.g., to other computer systems using local area digital networks or wide area digital networks). Such communications may include communications to one or more cloud computing networks.

[0063] The user interface 306 includes one or more output devices 308 and / or one or more input devices 310. The input device 310 may include one or more of the following: a keyboard, a mouse, a touch pad, a touch screen, a data glove, a joystick, a microphone, a scanner, a camera, etc. The output device 308 may include one or more of the following: an audio output device (e.g., a speaker), a visual output device (e.g., a display or a detector), etc.

[0064] The memory 314 may include high-speed random access memory (e.g., DRAM (Dynamic Random Access Memory, DRAM), SRAM (Static Random Access Memory, SRAM), DDRRAM (Double Data Rate Random Access Memory, DDR RAM) and / or other random access solid-state memory devices) and / or non-volatile memory (e.g., one or more magnetic disk storage devices, optical disk storage devices, flash memory devices and / or other non-volatile solid-state storage devices). The memory 314 optionally includes one or more storage devices remote from the control circuit system 302. The memory 314, or alternatively, the non-volatile solid-state memory device within the memory 314, includes a non-transitory computer-readable storage medium. In some embodiments, the memory 314 or the non-transitory computer-readable storage medium of the memory 314 stores the following programs, modules, instructions and data structures, or a subset or superset thereof:

[0065] Operating system 316, which includes procedures for handling various basic system services and for performing hardware-related tasks;

[0066] A network communications module 318 for connecting the server system 112 to other computing devices via one or more network interfaces 304 (eg, via a wired connection and / or a wireless connection);

[0067] A decoding module 320 for performing various functions related to encoding and / or decoding data, such as video data. In some embodiments, the decoding module 320 is an instance of the decoder component 114. The decoding module 320 includes, but is not limited to, one or more of the following:

[0068] o a decoding module 322 for performing various functions related to decoding encoded data, such as those previously described with respect to the decoder component 122; and

[0069] o an encoding module 340 for performing various functions related to encoding data, such as those previously described with respect to the encoder component 106; and

[0070] A picture memory 352 for storing pictures and picture data, for example for use with the decoding module 320. In some embodiments, the picture memory 352 includes one or more of the following: the reference picture memory 208, the buffer memory 252, the current picture memory 264, and the reference picture memory 266.

[0071] In some embodiments, the decoding module 322 includes a parsing module 324 (e.g., configured to perform the various functions previously described with respect to the parser 254), a transform module 326 (e.g., configured to perform the various functions previously described with respect to the scalar / inverse transform unit 258), a prediction module 328 (e.g., configured to perform the various functions previously described with respect to the motion compensated prediction unit 260 and / or the intra-picture prediction unit 262), and a filtering module 330 (e.g., configured to perform the various functions previously described with respect to the loop filter 256).

[0072] In some embodiments, the encoding module 340 includes a code module 342 (e.g., configured to perform various functions previously described with respect to the source encoder 202 and / or the encoding engine 212) and a prediction module 344 (e.g., configured to perform various functions previously described with respect to the predictor 206). In some embodiments, the decoding module 322 and / or the encoding module 340 includes Figure 3 For example, a shared prediction module is used by both the decoding module 322 and the encoding module 340.

[0073] Each module in the above-identified modules stored in the memory 314 corresponds to an instruction set for performing the functions described herein. The modules (e.g., instruction sets) identified above do not need to be implemented as separate software programs, processes, or modules, and therefore various subsets of these modules can be combined or otherwise rearranged in various embodiments. For example, the decoding module 320 optionally does not include a separate decoding module and encoding module, but uses the same set of modules to perform two function sets. In some embodiments, the memory 314 stores a subset of the modules and data structures identified above. In some embodiments, the memory 314 stores additional modules and data structures not described above, such as an audio processing module.

[0074] Although Figure 3 The server system 112 is shown according to some embodiments, but Figure 3 The present invention is intended more as a functional description of various features that may be present in one or more server systems rather than as a block diagram of the embodiments described herein. In practice, and as recognized by one of ordinary skill in the art, items shown separately may be combined and some items may be separated. For example, Figure 3Some of the items shown individually in the figure can be implemented on a single server, and a single item can be implemented by one or more servers. The actual number of servers used to implement the server system 112 and how the features are distributed among the servers will vary from implementation to implementation, and optionally, it depends in part on the amount of data traffic that the server system handles during peak usage time periods and during average usage time periods.

[0075] Example encoding processes and techniques

[0076] The encoding processes and techniques described below may be performed at the devices and systems described above (eg, source device 102, server system 112, and / or electronic device 120). Figure 4 An example temporal filtering process is shown in accordance with some implementations.

[0077] A temporal filter is a type of signal processing filter that operates on a time-varying signal to remove unwanted changes or noise that occur over time. Temporal filters can be used to selectively remove or attenuate signal components at specific frequencies in the time domain. For example, in video processing, temporal filtering can be used to reduce noise and improve video quality by removing unwanted changes in brightness or color over time. In video coding, the encoder can consider both distortion (e.g., error relative to the true value block) and bit rate (e.g., the cost of encoding the error) when minimizing the metric of rate*λ+distortion for each coded block. λ is a tuning factor that depends on the desired tradeoff between error (e.g., minimizing distortion and / or achieving the highest quality) and bit rate (e.g., desiring to use the minimum possible bit rate), which involves two competing objectives.

[0078] Figure 4 An example temporal filtering process using three reference pictures or frames (e.g., picture 404, current picture 402, and picture 406, alternatively also represented as picture (t-1), picture (t), and picture (t+1)) is shown. Current picture 402 represents the current frame to be filtered. When the next frame (e.g., the frame after the current frame in the video data) is only slightly different from the current frame (e.g., current picture 402), it is likely that the encoding of the blocks in the next frame (e.g., picture 406) will be skipped. In such a scenario, the metric to be minimized in the encoding process is rate*λ+distortion1+distortion2. For example, distortion1 is the error of the encoded block of the current picture 402 relative to the true value block (e.g., the original block) in picture 402, and distortion2 is the error of the encoded block of the current picture 402 relative to the true value block in picture 406 (e.g., because the encoding block of picture 406 has been skipped).

[0079] In some embodiments, rather than calculating the two distortions separately, the distortion between the coded block and the average of the original versions of the two pictures (e.g., the average of the original current picture 402 and the original picture 406) can be calculated. In some embodiments, the original versions of the two pictures correspond to frames in a video data stream (e.g., original video data). Therefore, using temporally filtered original pictures can improve coding efficiency (e.g., by minimizing the number of calculations to determine the corresponding distortions). In some embodiments, if the original frames are different, the filter strength drops rapidly (e.g., if the subsequent picture is significantly different from the first original frame, the subsequent original picture is lightly weighted and contributes less to the average / filtered frame) because the decoded sample values ​​of the first picture will be less likely to be reused for the second picture.

[0080] In some embodiments, temporal filtering involves three stages. The first stage of temporal filtering involves motion estimation (ME), since the coded blocks will likely not remain stationary in the sequence but have some motion (e.g., Figure 5A The motion vector MV0 in describes the motion of the current block 511 in the current picture 502 relative to the reference block 513 of the reference picture 504, and / or the motion vector MV1 describes the motion of the current block 511 in the current picture 502 relative to the reference block 515 of the reference picture 506). For a coding block, the motion estimation stage involves finding the best motion vector with the smallest error in a particular reference frame. To reduce complexity, the search range for the best motion vector is typically limited to a predefined distance relative to the motion vector (0, 0), which corresponds to its co-located block in the reference frame (e.g., a block that remains stationary between the current frame and the reference frame).

[0081] After finding the best motion vector in the motion estimation stage, the second stage of temporal filtering involves motion compensation (MC). Motion compensation takes as input a reference frame and the motion vector generated at the motion estimation (ME) stage. Motion compensation then applies an interpolation filter to the reference samples in the reference frame pointed to by the motion vector (e.g., from the current block) and generates a prediction block for the current block.

[0082] In the final stage of filtering, the prediction of the current block (e.g., obtained during the motion compensation (MC) stage) is assigned a weight. Figure 4 In , the blank squares in picture 404, current picture 402, and picture 406 may alternatively be considered to represent corresponding predicted blocks of these pictures based on motion vectors and reference frames associated with the corresponding pictures. In some embodiments, filtering is performed using prediction samples and estimated weights. Figure 44, the weights for the various blocks of picture 404 are shown in grid 410, the weights for the various blocks of current picture 402 are shown in grid 412, and the weights for the various blocks of picture 406 are shown in grid 414. The weighted predictions (e.g., by multiplying grid 410 with the corresponding prediction block of picture 404, by multiplying grid 412 with the corresponding prediction block of current picture 402, and by multiplying grid 414 with the corresponding prediction block of picture 406) are aggregated and divided by the total weight to produce a filtered block 416. In some embodiments, using an unfiltered "self frame" enables the current frame to contribute information to the final filtered block.

[0083] The final filtered samples (eg, in filter block 416) are denoted as P 经滤波的 And use Equation 1 to derive:

[0084]

[0085] Formula 1 - Filter Sample

[0086] Among them, w i is the prediction weight (e.g., w i represented by grid 410, grid 412 and grid 414) and P i are prediction samples (eg, from motion compensation (MC), represented by the empty squares in picture 404, current picture 402, and picture 406), and i indicates an index into a reference picture list.

[0087] In some embodiments, the weight w i Estimated based on the error between the predicted sample (e.g., obtained from motion compensation) and the original sample (e.g., the original picture frame from the original / original video data). In some embodiments, when the error is larger, the weight is correspondingly smaller. Similarly, in some embodiments, a larger motion vector indicates that the reference region is less reliable (e.g., further away), and a smaller weight is provided for the larger motion vector.

[0088] In some embodiments, temporal filtering is applied as a pre-encoding process and the filtered frames (e.g. Figure 4The filter block 416 in the decoded frame buffer replaces the frame to be encoded (e.g., the original frame from the video data). In some cases, the temporal filtering process can be decoupled from the encoder pipeline so that the temporal filter used in one codec can also be applied to other codecs. For example, after the encoding of a frame is completed, the encoder determines which reference frame stored in the decoded frame buffer needs to be replaced, and explicitly signals the update in the bitstream. In some embodiments, the decoded frame buffer stores different types of reference frames, including alternative reference frames (Alternate Reference Frame, ARF) that can be frames from the past or future. The ARF can be synthesized by the encoder, for example, by temporal filtering along the motion trajectory of multiple (e.g., continuous) original frames. Such a synthesized frame can remove (e.g., partially remove or substantially remove) the acquisition noise on each frame to potentially reduce the collective prediction error between several display frames, while encoding the common information of multiple original frames to optimize the overall rate-distortion performance.

[0089] Figure 5A A prediction process according to some embodiments is shown. As previously discussed, one option for predicting a current block 511 (e.g., CU) of a current picture 502 (e.g., having a size of w×h) is by using MV0 of a reference block 513 from a reference picture 504. A second option for predicting the current block 511 is by using MV1 of a reference block 515 from a reference picture 506.

[0090] Figure 5B 1 shows a scenario for determining a window error associated with a block that does not include any partitions according to some embodiments. For a block 508 that does not include any partitions, the block size corresponds to the number of samples in the entire block. For example, the block size is defined as N×M samples (e.g., in the motion estimation stage for the block 508). In some embodiments, Figure 5B As shown, N is the same as M, for example, the block size may be 32×32 samples, 64×64 samples, or other samples of size. In some embodiments, N is different from M, for example, the block size may be 32×64, 16×32, or other samples of size. The prediction weights (w i ). For each sample, two types of errors are estimated: window error and block error.

[0091] A window (e.g., a 5×5 window, or a window of different sizes) is defined for each sample so that the current sample is located at the center of the window. Figure 5B As shown, window 512 is centered on sample 510 at the upper left corner of block 508. Sample 510 is a boundary sample of block 508. As an example, Figure 5B Window 512 in has a size of W×W and includes 5×5 samples. The window may have other sizes (e.g., 3×3, 7×7, 15×15, or other sizes). Sample 516 is outside the boundary of block 508, but within window 512. For a boundary sample (e.g., sample 510) of a window of size W, the number of samples that are within the window but outside the block along a first dimension of the window (e.g., the width dimension) is (W-1) / 2. For example, where W is 5, there will be two samples along a particular row of the window (e.g., along the width dimension) that are outside the boundary of the block but within the window.

[0092] For such a case, in some embodiments, the prediction buffer 514 associated with the block 508 is expanded or enlarged to (N+(W-1))x(M+(W-1)) pixels to completely cover the samples within the window of the samples at the boundary of the block (e.g., sample 510), so that all samples used for the window error estimation of the boundary samples are collected within the window. In some embodiments, the prediction buffer 514 (also alternatively referred to as an "enlarged prediction buffer" or "enlarged prediction block" hereinafter) is generated at the motion compensation stage.

[0093] The window error of sample 510 includes the errors of all samples within a window 512 centered on sample 510. In some embodiments, the square of the absolute difference of each sample in block 508 (e.g., a 32×32 block or other sized block) is pre-calculated and aggregated in a window for a particular sample (e.g., a 5×5 window or other sized window). For example, the difference of each sample is the difference between the original sample and the predicted sample based on the reference frame and motion vector. In this way, computational complexity can be reduced because neighboring samples do not have to recalculate the window error of overlapping samples between their respective windows (e.g., their respective 5×5 windows). In some embodiments, the window error is the sum of absolute differences (SAD) or the sum of squared errors (SSE) or other metrics calculated based on the difference between the original sample and the predicted sample (e.g., predicted based on the motion vector and the associated reference frame). For blocks without partitions, the block error (eg, SSE or SAD) is calculated for each sample in the block (eg, 32x32 block) and normalized.

[0094] In some implementations, the value of sample 516 in prediction buffer 514 is populated with the corresponding value of the sample in the reference frame associated with block 508 while taking into account the motion vector associated with block 508. For example, if the motion vector associated with block 508 is (0, 0), the value of the sample to the left of sample 510 in the co-located block of sample 510 in the reference frame is placed in prediction buffer 514.

[0095] Because a portion of the samples in window 520 associated with sample 518 is outside the boundary of block 508 due to size W of window 520 , sample 518 is considered a boundary sample even though sample 518 is not a sample at the very edge of block 508 .

[0096] Figure 5C and Figure 5D A scenario for determining an error associated with a block including partitions according to some embodiments is shown. In some embodiments, each partition is further divided into sub-partitions (e.g., a 16×16 partition is further divided into four 8×8 partitions). When partitioning is enabled and N partitions (or sub-partitions) are formed (e.g., N=4 or some other number), N corresponding motion vectors are generated (one motion vector for each partition or sub-partition). For partitions (or sub-partitions) that include content with different motions, such a sub-partitioning scheme uses a corresponding motion vector for each partition or sub-partition (e.g., each 16×16 partition) to enable better prediction than using a single motion vector applied to the entire block (e.g., the entire 32×32 block).

[0097] For example, if partitioning is enabled, corresponding motion compensation is applied to each partition (e.g., each of the four 16×16 partitions). Figure 5C As shown, the predictions are then collected in buffers (eg, a 32x32 buffer with four 16x16 partitions) belonging to the corresponding predictors (eg, Predictor 0, Predictor 1, Predictor 2, and Predictor 3).

[0098] Figure 5C An example scenario for determining a window error associated with a block comprising two or more partitions according to some embodiments is shown. Figure 5C In FIG. 5 , a block 540 having a block size of N×N (e.g., 32×32, or other size) is divided into four partitions, each having a size of N / 2×N / 2 (e.g., 16×16, or other size). Instead of making the prediction buffer (e.g., Figure 5B The prediction buffer 514 in , or the enlarged prediction buffer) is used for the entire block 540, providing each partition Figure 5C1 . A prediction buffer 544 is schematically shown in FIG. 5 (e.g., an enlarged prediction buffer, an enlarged prediction buffer 544). The size of the enlarged prediction buffer depends on the window size. For a window having a size of W×W and a partition 542 having a size of N / 2×N / 2, the prediction buffer 544 has a size of (N / 2+(W-1))x(N / 2+(W-1)). Boundary samples in the partition corresponding to predictor 1 are shown as blank circles, while samples within the window but outside the partition are shown as shaded circles. Although only a single layer of boundary samples is shown, the number of layers of boundary samples depends on the size of the window (e.g., as indicated by Figure 5B Similarly, although only a single layer of shaded circles is shown, the number of samples in prediction buffer 544 depends on the size of the window. The manner in which the window is filled outside the boundaries of the partition is similar to that of reference 544. Figure 5B For example, the motion vector associated with partition 542 is used to fill in corresponding samples outside partition 542 but within a window of boundary samples from the reference frame. Having partitions in block 540 enables prediction results for samples in the block to be collected in corresponding partitions (e.g., four 16×16 partitions) rather than a single block buffer (e.g., a 32×32 block buffer, or an N×M buffer).

[0099] In addition to expanding the prediction buffer, another way to fill the window of samples outside the boundary of the partition is called "error clipping". In error clipping, the value of the corresponding sample on the boundary (e.g., the sample at the edge row or column of the partition) is copied, reused or duplicated as a sample within the window but outside the partition, which is adjacent to the corresponding boundary sample ( Figure 5C The samples to the left of sample 546 in share the same value as sample 546). Figure 5C , which corresponds to predictor 0. The other remaining samples are filled in a similar manner (e.g., the sample two columns to the left of sample 546 is duplicated with the sample value one column away to the left of sample 546). The rows above sample 546 are filled in a similar manner.

[0100] In this way, the values ​​of some boundary samples within the block, partition or sub-partition and their associated errors are copied or reused in samples outside the boundary of the block, partition or sub-partition within the window. As mentioned above, the associated error of the sample within each partition or block is the difference between the original sample and the predicted sample (e.g., predicted based on the motion vector associated with the partition and the reference frame corresponding to the partition).

[0101] Figure 5D548 shows an additional example scenario for determining a window error associated with a block comprising two or more partitions according to some embodiments. While window 548 shows error clipping performed at sub-partition boundaries that are also block boundaries, Figure 5D Examples of embodiments are shown in which error clipping is performed for samples not only at block boundaries but also at partition (or sub-partition) boundaries. In some embodiments, error clipping is applied only at boundaries of corresponding partitions that are also boundaries of blocks. In some embodiments, error clipping is applied both at boundaries of corresponding partitions that are also boundaries of blocks and at boundaries of corresponding partitions that are not boundaries of blocks. Applying error clipping operations at the partition or sub-partition level instead of the block level may improve window error estimation because different motion vectors (e.g., between partitions or sub-partitions) may generate samples with large differences at sub-partition boundaries.

[0102] For example, in Figure 5D , Predictor 1 may have a motion vector that is very different from the motion vector associated with Predictor 0. Instead of using the bottom left sample of the top right partition associated with Predictor 1 within window 550 for sample 552 in the top left partition corresponding to Predictor 0, error clipping is applied such that the value of sample 552 is copied to the sample to the right of sample 552 in window 550. In this manner, samples in window 550 will be more likely to share the same motion vector as samples within the top left partition (corresponding to Predictor 0).

[0103] For example, suppose (x, y) represents the coordinates of a sample within an N×M block, where x is contained in the range [0, N-1] and y is contained in the range [0, M-1]. Suppose there is a subpartition S with samples having coordinates (S x , S y ), where S x Included in the range [S x最小 , S x最大 ] x最小 and S x最大 Satisfy the constraint (0 <= S x最小 <=S x最大 <=N-1). Similarly, S y Included in the range [S y最小 , S y最大 ], and S y最小 and S y最大 Satisfy the constraint (0 <= S y最小 <=S y最大 <=M-1). Centered on the sample, a window of size (W×W) can be represented by the coordinates of the upper left corner, lower left corner, upper right corner, and lower right corner of the window. These coordinates of the window are (S x –W / 2, S y–W / 2)、(S x –W / 2, S y +W / 2)、(S x +W / 2,S y –W / 2)、(S x +W / 2,S y +W / 2).

[0104] The samples at the boundaries of the subpartitions have coordinates (S x , S y ): S x =S x最小 , S x =S x最大 , S y =S y最小 or S y =S y最大 .

[0105] According to S x Match S x最小 (For example, S x =S x最小 ), the samples are determined to be on the left (vertical) boundary of the sub-partition. The samples in the left half of the W×W window will be outside the current partition. The horizontal coordinates S of these samples xw Satisfaction x最小 –W / 2)<=S xw x最小 .

[0106] According to S y Match S y最小 (For example, S y =S y最小 ), the samples are determined to be on the upper (horizontal) boundary of the sub-partition. The samples in the upper half of the W×W window will be outside the current partition. The vertical coordinates S of these samples are yw Satisfaction y最小 –W / 2)<=S yw y最小 .

[0107] According to S x Match S x最大 (For example, S x =S x最大 ), the samples are determined to be on the right (vertical) boundary of the sub-partition. The samples in the right half of the W×W window will be outside the current partition. The horizontal coordinates S of these samples xw Satisfy S x最大 xw <=(S x最大 +W / 2).

[0108] ​​​According to S y Match S y最大 (For example, S y =S y最大 ), the samples are determined to be on the lower (horizontal) boundary of the sub-partition. The samples in the lower half of the W×W window will be outside the current partition. The vertical coordinates S of these samples are yw Satisfy S y最大 yw <=(S y最大 +W / 2).

[0109] As reference Figure 5C As described in the error clipping for window 548 in FIG, for samples of boundary samples that are within the W×W window but outside the current sub-partition, the errors of these samples are set equal to their corresponding boundary samples within the current partition. Assume that the error of the boundary samples is E(S x , S y ), where (S x , S y ) represents the coordinates of the boundary samples.

[0110] For samples on the left boundary (e.g., in S x Match S x最小 In the case of ), the left half of the samples within the W×W window of these boundary samples are outside the partition. In an embodiment using error clipping, the error E(S xw , S yw ) is set to E(S x最小 , S yw ), where S xw Satisfaction x最小 –W / 2)<=S xw x最小 , and S yw Satisfaction y最小 –W / 2)<=S yw <=(S y最大 +W / 2).

[0111] For samples on the upper boundary (e.g., in S y Match S y最小 In the case of , the upper half of the samples within the W×W window of these boundary samples are outside the partition. In an embodiment using error clipping, the error E(S xw , S yw ) is set to E(S xw , S y最小 ), where S yw Satisfaction y最小 –W / 2)<=S yw <S​​y最小 , and S xw Satisfaction x最小 –W / 2)<=Sx w <=(S x最大 +W / 2).

[0112] For samples on the right boundary (e.g., in S x Match S x最大 In the case of ), the right half of the samples within the W×W window of these boundary samples are outside the partition. In the implementation using error clipping, the error E(S xw , S yw ) is set to E(S x最大 , S yw ), where S xw Satisfy S x最大 xw <=(S x最大 +W / 2), and S yw Satisfaction y最小 –W / 2)<=S yw <=(S y最大 +W / 2).

[0113] For samples on the lower boundary (e.g., in S y Match S y最大 In the case of , the lower half of the samples within the W×W window of these boundary samples are outside the partition. In an embodiment using error clipping, the error E(S xw , S yw ) is set to E(S xw , S y最大 ), where S yw Satisfy S y最大 yw <=(S y最大 +W / 2), and S xw Satisfaction x最小 –W / 2)<=Sx w <=(S x最大 +W / 2).

[0114] In some embodiments, the sub-partitions are further divided, thereby generating recursive partitions or multi-layer partitions. Such divided sub-partitions have their own corresponding motion vectors. In some embodiments, the error clipping operation is applied at the minimum sub-partition level.

[0115] ​​In some embodiments, the corresponding window errors of different partitions of block 540 are determined in the same manner (e.g., by extending the prediction buffer, or by error clipping). In some embodiments, the corresponding window errors of different partitions of block 540 are determined in different manners (e.g., one or more partitions by extending the prediction buffer, and one or more partitions by error clipping).

[0116] In some embodiments, the prediction weights for a particular reference frame are estimated using the collective prediction (e.g., using four predictors or a number of predictors corresponding to the number of partitions or sub-partitions). For example, with four 16×16 partitions enabled, the SSE (or SAD) for each 16×16 partition is calculated and normalized. In this case, the block error is the same for samples in the same 16×16 partition, but different from samples in different partitions. The combined error is derived based on the window error and the block error as:

[0117]

[0118] Formula 2 - Combined Error

[0119] The block error provides a rough estimate of the samples in a particular partition. In contrast, the window error provides a more accurate error estimate in a local area close to the current sample. Therefore, a weight factor n is applied to the window error to make the final error estimate more biased towards the window error. In some embodiments, the weight factor n is 5 (e.g., or n can be any number greater than 1.5).

[0120] The combined error is also used to calculate the scaled_error, which is defined as:

[0121] Scaling_error = (combined_error * distance_factor / attenuation_factor)

[0122] Equation 3 - Scaling Error

[0123] Among them, the distance_factor is a parameter calculated based on the motion vector, and the decay_factor is a parameter used to reduce the error (e.g., thereby increasing the weight, as shown in the following formula). For example, when the motion vector is large, the distance it points to is far away from the current block. In such a scenario, a larger distance_factor indicates that the derived prediction may not be good, resulting in a smaller weight. On the other hand, the decay_factor is determined by the configured quantization parameter, the filter strength, and the noise level of the current frame. And the final weight is determined by:

[0124] Weight = e -缩放_误差

[0125] Formula 4 - Prediction weight

[0126] In general, the larger the scaled error, the smaller the final prediction weight. This weight is then used to calculate the predicted value using the formula described previously. To obtain the final filtered samples. In some embodiments, the final filtered samples are signaled by the encoder (e.g., as an ARF synthesized by the encoder) and stored in the decoded frame buffer until the encoder explicitly signals in the bitstream that the synthesized ARF stored in the decoded frame buffer will be replaced.

[0127] Fig. 6A 6 is a flow chart illustrating a method 600 of encoding a video according to some embodiments. The method 600 may be performed at a computing system (e.g., server system 112, source device 102, or electronic device 120) having a control circuit system and a memory storing instructions for execution by the control circuit system. In some embodiments, the method 600 is performed by executing instructions stored in a memory (e.g., memory 314) of the computing system.

[0128] The system receives (602) video data including a current block. The current block has a size of N×M pixels, where N and M are positive integers (e.g., Figure 5B The system generates (604) a prediction block larger than the current block (e.g., an enlarged prediction block, such as Figure 5B The prediction block has a size of (N+W)x(M+W) pixels, where W is a positive integer. The system uses the generated prediction block to determine (606) samples of the current block (e.g., Figure 5B 508 in the sample window 510) (e.g., Figure 5B 512). For example, the window error and block error are estimated for each sample. The window error may include the sum of squared errors (SSE) calculated for the sample window and normalized to the number of samples within the sample window. If the window size of each sample is limited to W×W (W>0), the prediction buffer is enlarged to (N+(W-1))x(M+(W-1)) or (N+V)x(M+V), where V=W-1, so that all necessary samples required for window error estimation of boundary samples are collected. A prediction block larger than the current block can be generated at the motion compensation stage and is generated using motion vector information and a reference frame associated with the current block.

[0129] Figure 6B6 is a flow chart illustrating a method 650 of encoding a video according to some embodiments. The method 650 may be performed at a computing system (e.g., server system 112, source device 102, or electronic device 120) having a control circuit system and a memory storing instructions for execution by the control circuit system. In some embodiments, the method 650 is performed by executing instructions stored in a memory (e.g., memory 314) of the computing system.

[0130] The system receives (652) video data including a current block, wherein the current block has two or more partitions, and each of the two or more partitions corresponds to a different predictor (e.g., Figure 5C and Figure 5D The system identifies (654) samples in two or more partitions (e.g., with Figure 5C and Figure 5D The system generates (656) a sample window (e.g., Figure 5C Window 548 and Figure 5D The system performs (658) error clipping (e.g., Figure 5C The error clipping in the window 548 and Figure 5D The system determines (660) a window error for the sample window.

[0131] Although Fig. 6A and Figure 6B Multiple logical stages are shown in a particular order, but stages that are not order-dependent may be reordered, and other stages may be combined or split. Some reordering or other groupings not specifically mentioned are obvious to one of ordinary skill in the art, so the ordering and groupings presented herein are not exhaustive. In addition, it should be appreciated that the stages may be implemented in hardware, firmware, software, or any combination thereof.

[0132] In some embodiments, according to determining that the current block has two or more partitions: the system identifies the partition in which the sample in the two or more partitions is located, and for the sample (e.g., Figure 5CThe system also performs error clipping on one or more window samples located outside the identified partition. The window error is determined after error clipping is performed. Window samples are samples within a W×W sample window centered on the (current) sample, for example, if a partitioning scheme is enabled within an N×M block (e.g., a 32×32 block is divided into four 16×16 sub-blocks), error clipping is performed on samples at the partition boundary. Error clipping includes copying sample error values ​​of samples at the partition for use with window samples outside the partition. In some embodiments, instead of performing error clipping on one or more window samples located outside the identified partition, a prediction sub-block larger than the current sub-block (e.g., an enlarged prediction sub-block) is generated for the identified partition using a motion vector associated with the identified partition and a reference frame associated with the identified partition.

[0133] In some embodiments, the two or more partitions include two or more partition levels, and wherein the identified partition is a minimum partition level. For example, the sub-partition can be further divided, thereby generating a recursive partition or a multi-layer partition. The divided sub-partition has its own motion vector, and the error clipping operation is applied at the minimum sub-partition level.

[0134] In some embodiments, determining the window error comprises deriving a sum of squared errors (SSE) of a sample window. In some embodiments, determining the window error comprises deriving a sum of absolute differences (SAD) of a sample window. For example, the window error can be SAD, SSE, or other metrics calculated based on the difference between the original sample and the predicted sample. In some embodiments, the window error of the current block is determined as part of applying a temporal filter to the current block. For example, the temporal filter replaces the current block with a filter block for subsequent encoding processing.

[0135] In some embodiments, the system uses the determined window error to derive prediction weights for the current block. For example, the window error of the sample is used (e.g., in combination with the block error) to derive weights for sample prediction of the sample. In some embodiments, the prediction weights correspond to predictions made using a corresponding reference frame of the current block; and the system uses the prediction weights to generate a filtered current block (e.g., Figure 4 ); and the system signals the filtered current block in the video bitstream.

[0136] In some implementations, the system receives a block including a current block (e.g., Figure 5A Block 511 and / or Figure 5B508 in the current block), wherein the current block has two or more partitions, and each of the two or more partitions corresponds to a different predictor (eg, the partition corresponds to Figure 5C and Figure 5D The system identifies the partitions in which the samples in the two or more partitions are located, and the system generates a sample window for the samples. The system generates a sample window for one or more window samples (e.g., Figure 5C Window 548 and / or Figure 5D After performing error clipping, the system determines the window error of the sample window.

[0137] In some embodiments, the current block has a size of N×M pixels. The window error is determined using a prediction block of the current block, the generated prediction block has a size of (N+W)×(M+W) pixels, and N, M, and W are positive integers.

[0138] In some embodiments, performing error clipping includes copying the value of a first sample at a partition boundary to a second sample located outside the partition boundary. For example, the error value of a sample at a left boundary is copied to a sample to the left of the left boundary, the error value of a sample at a right boundary is copied to a sample to the right of the right boundary, the error value of a sample at an upper boundary is copied to a sample above the upper boundary, and the error value of a sample at a lower boundary is copied to a sample below the lower boundary. For example, an error is calculated for a first sample, and the calculated error is copied for a second sample.

[0139] In some embodiments, the prediction weights correspond to predictions made using a corresponding reference frame of the current block; and the system generates a filtered current block using the prediction weights; and the system signals the filtered current block in a video bitstream.

[0140] In some implementations, the system receives a block including a current block (e.g., Figure 5A Block 511 and / or Figure 5B 508 in the current block), wherein the current block has two or more partitions, and each of the two or more partitions corresponds to a different predictor (eg, the partition corresponds to Figure 5C and Figure 5D The system identifies a partition in which samples in two or more partitions are located, wherein the partition has a size of N×M pixels, where N and M are positive integers. The system generates a prediction block (e.g., an enlarged prediction block, Figure 5CThe system uses the generated prediction block to determine a window error for a sample window of samples. Turning now to some example embodiments.

[0141] (A1) In one aspect, some embodiments include a method of video encoding (e.g., method 600). In some embodiments, the method is performed at a computing system (e.g., server system 112, source device 102, or electronic device 120) having a control circuit system and a memory. In some embodiments, the method is performed at an encoding module (e.g., encoding module 320). In some embodiments, the method is performed at a source encoding component (e.g., source encoder 202), an encoding engine (e.g., encoding engine 212), and / or an entropy encoder (e.g., entropy encoder 214). The method includes: (i) receiving video data including a current block, wherein the current block has a size of N×M pixels, N and M are positive integers (e.g., Figure 5B (ii) generating a prediction block larger than the current block for the current block (eg, an enlarged prediction block, Figure 5B ), wherein the generated prediction block has a size of (N+W)x(M+W) pixels, W being a positive integer; (iii) using the generated prediction block to determine samples of the current block (e.g., Figure 5B The sample window (e.g., Figure 5B Window error of window 512).

[0142] (A2) In some embodiments according to A1, the method further includes: based on determining that the current block has two or more partitions: (i) identifying the partition in which the sample in the two or more partitions is located; (ii) generating a sample window for the sample; and (iii) performing error clipping for one or more window samples located outside the identified partition, wherein the window error is determined after performing the error clipping.

[0143] (A3) In some embodiments according to A2, the two or more partitions include two or more partition levels, and wherein the identified partition is a minimum partition level.

[0144] (A4) In some embodiments of any one of A1 to A3, determining the window error includes deriving a sum of squared errors (SSE) of the sample window.

[0145] (A5) In some embodiments of any one of A1 to A4, determining the window error comprises deriving a sum of absolute differences (SAD) of the sample window.

[0146] (A6) In some embodiments of any one of A1 to A5, the window error of the current block is determined as part of applying a temporal filter to the current block.

[0147] (A7) In some embodiments according to any one of A1 to A6, the method includes deriving a prediction weight for the current block using the determined window error.

[0148] (A8) In some embodiments according to A7, the prediction weight corresponds to a prediction made using a corresponding reference frame of the current block; and the method includes: (i) generating a filtered current block (e.g., Figure 4 416 in the filtering block); and (ii) signaling the filtered current block in the video bitstream.

[0149] (A9) In some embodiments of any one of A1 to A8, the current block has a size of 32×32 pixels.

[0150] (B1) In one aspect, some embodiments include a method of video encoding (e.g., method 650). In some embodiments, the method is performed at a computing system (e.g., server system 112, source device 102, or electronic device 120) having a control circuit system and a memory. In some embodiments, the method is performed at an encoding module (e.g., encoding module 320). In some embodiments, the method is performed at a source encoding component (e.g., source encoder 202), an encoding engine (e.g., encoding engine 212), and / or an entropy encoder (e.g., entropy encoder 214). The method includes: (i) receiving video data including a current block, wherein the current block has two or more partitions, and each of the two or more partitions corresponds to a different predictor (e.g., Figure 5C and Figure 5D (ii) identifying samples in the two or more partitions (e.g., Figure 5C and Figure 5D (iii) generating a sample window (e.g., Figure 5C Window 548 and Figure 5D (iv) performing error clipping (eg, Figure 5CThe error clipping in the window 548 and Figure 5D (v) determining a window error for the sample window after performing error clipping.

[0151] (B2) In some embodiments according to B1, the current block has a size of N×M pixels, and a prediction block of the current block that is larger than the current block is used to determine the window error, wherein the prediction block has a size of (N+W)x(M+W) pixels, where N, M, and W are positive integers.

[0152] (B3) In some embodiments according to B1 or B2, performing error clipping includes copying the value of a first sample at a partition boundary to a second sample located outside the partition boundary.

[0153] (B4) In some embodiments according to any one of B1 to B3, error clipping is performed on one or more window samples located outside the current block.

[0154] (B5) In some embodiments according to any one of B1 to B4, the method includes deriving a prediction weight for the current block using the determined window error.

[0155] (B6) In some embodiments according to any one of B1 to B5, the prediction weight corresponds to a prediction made using a corresponding reference frame of the current block; and the method includes: generating a filtered current block using the prediction weight; and signaling the filtered current block in a video bitstream.

[0156] (B7) In some embodiments according to any one of B1 to B6, the current block has a size of 32×32 pixels.

[0157] (C1) In one aspect, some embodiments include a method of video encoding. In some embodiments, the method is performed at a computing system (e.g., server system 112, source device 102, or electronic device 120) having a control circuit system and a memory. In some embodiments, the method is performed at an encoding module (e.g., encoding module 320). In some embodiments, the method is performed at a source encoding component (e.g., source encoder 202), an encoding engine (e.g., encoding engine 212), and / or an entropy encoder (e.g., entropy encoder 214). The method includes: (i) receiving video data including a current block, wherein the current block has two or more partitions, and each of the two or more partitions corresponds to a different predictor; (ii) identifying a partition in which samples of the two or more partitions are located, wherein the partition has a size of N×M pixels, N and M are positive integers; (iii) generating a prediction block (e.g., an enlarged prediction block) larger than the current block for the partition, wherein the generated prediction block has a size of (N+W)x(M+W) pixels, W is a positive integer, and wherein the prediction block is generated using motion vector information of the partition; and (iv) using the generated prediction block to determine a window error of a sample window of the sample.

[0158] (C2) In some embodiments according to C1, the two or more partitions include two or more partition levels, and wherein the identified partition is a minimum partition level.

[0159] (C3) In some embodiments according to any one of C1 to C2, determining the window error includes deriving a sum of squared errors (SSE) of the sample window.

[0160] (C4) In some embodiments of any one of C1 to C3, determining the window error includes deriving a sum of absolute differences (SAD) of the sample window.

[0161] (C5) In some embodiments according to any one of C1 to C4, the window error of the current block is determined as part of applying a temporal filter to the current block.

[0162] (C6) In some embodiments according to any one of C1 to C5, the method includes deriving a prediction weight for the current block using the determined window error.

[0163] (C7) In some embodiments according to C6, the prediction weight corresponds to a prediction made using a corresponding reference frame of the current block; and the method includes: (i) generating a filtered current block using the prediction weight; and (ii) signaling the filtered current block in a video bitstream.

[0164] (C8) In some embodiments according to any one of C1 to C7, the current block has a size of 32×32 pixels.

[0165] On the other hand, some embodiments include a computing system (e.g., server system 112) that includes a control circuit system (e.g., control circuit system 302) and a memory (e.g., memory 314) coupled to the control circuit system, the memory storing one or more sets of instructions configured to be executed by the control circuit system, the one or more sets of instructions including instructions for performing any of the methods described herein (e.g., A1 to A9, B1 to B7, and C1 to C8 above).

[0166] On the other hand, some embodiments include a non-transitory computer-readable storage medium storing one or more sets of instructions for execution by a control circuit system of a computing system, wherein the one or more sets of instructions include instructions for performing any of the methods described herein (e.g., A1 to A9, B1 to B7, and C1 to C8 above).

[0167] It should be understood that, although the terms "first", "second" etc. can be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. The terms used herein are only for the purpose of describing a specific embodiment, and are not intended to limit the claims. As used in the description of implementation and the appended claims, unless the context clearly indicates otherwise, the singular forms "one (a)", "one (an)" and "the" are also intended to include plural forms. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items in the associated listed items. It will also be understood that, when used in this specification, the term "includes" and / or "comprising" specifies the existence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the existence or addition of one or more other features, integers, steps, operations, elements, parts and / or its groups.

[0168] As used herein, the term “if” may be interpreted as “when the precondition is true” or “after the precondition is true” or “in response to determining that the precondition is true” or “in accordance with determining that the precondition is true” or “in response to detecting that the precondition is true”, depending on the context. Similarly, the phrase “if it is determined that [the precondition is true]” or “if [the precondition is true]” or “when [the precondition is true]” may be interpreted as “after determining that the precondition is true” or “in response to determining that the precondition is true” or “in accordance with determining that the precondition is true” “after detecting that the precondition is true” or “in response to detecting that the precondition is true”, depending on the context.

[0169] For purposes of illustration, the foregoing description has been described with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the claims to the precise forms disclosed. Many modifications and variations are possible in light of the above teachings. The embodiments were chosen and described in order to best illustrate the principles of operation and practical application, thereby enabling others skilled in the art to implement.

Claims

1. A method of video encoding performed at a computing system having a memory and one or more processors, the method comprising: Receiving video data including a current block, wherein the current block has a size of N×M pixels, N and M are positive integers; generating a prediction block larger than the current block for the current block, wherein the prediction block has a size of (N+W)x(M+W) pixels, where W is a positive integer; and The generated prediction block is used to determine a window error of a sample window of samples of the current block.

2. The method according to claim 1, further comprising: According to determining that the current block has two or more partitions: identifying a partition of the two or more partitions in which the sample is located; generating the sample window for the sample; as well as Error clipping is performed on one or more window samples located outside the identified partition, wherein the window error is determined after performing the error clipping.

3. The method according to claim 2, wherein: The two or more partitions include two or more partition levels, and wherein the identified partition is a minimum partition level.

4. The method according to claim 1, wherein: Determining the window error includes deriving a sum of squared errors (SSE) for the sample window.

5. The method according to claim 1, wherein: Determining the window error includes deriving a sum of absolute differences (SAD) for the sample window.

6. The method according to claim 1, wherein: The window error of the current block is determined as part of applying a temporal filter to the current block.

7. The method of claim 1, further comprising deriving a prediction weight for the current block using the determined window error.

8. The method according to claim 7, wherein: The prediction weight corresponds to a prediction performed using a corresponding reference frame of the current block; and The method further comprises: generating a filtered current block using the prediction weights; as well as The filtered current block is signaled in a video bitstream.

9. The method according to claim 1, wherein: The current block has a size of 32×32 pixels.

10. A computing system comprising: Control circuit system; Memory; as well as one or more sets of instructions stored in the memory and configured for execution by the control circuitry, the one or more sets of instructions comprising instructions for: Receiving video data including a current block, wherein the current block has a size of N×M pixels, N and M are positive integers; generating a prediction block larger than the current block for the current block, wherein the prediction block has a size of (N+W)x(M+W) pixels, where W is a positive integer; and The generated prediction block is used to determine a window error of a sample window of samples of the current block.

11. The computing system of claim 10, wherein: The one or more sets of instructions also include instructions for: based on determining that the current block has two or more partitions: identifying a partition of the two or more partitions in which the sample is located; generating the sample window for the sample; as well as Error clipping is performed on one or more window samples located outside the identified partition, wherein the window error is determined after performing the error clipping.

12. The computing system of claim 11, wherein: The two or more partitions include two or more partition levels, and wherein the identified partition is a minimum partition level.

13. The computing system of claim 10, wherein: Determining the window error includes deriving a sum of squared errors (SSE) or a sum of absolute differences (SAD) for the sample window.

14. The computing system of claim 10, wherein: The one or more sets of instructions also include instructions for deriving a prediction weight for the current block using the determined window error.

15. The computing system of claim 10, wherein: The current block has a size of 32×32 pixels.

16. A non-transitory computer-readable storage medium storing one or more sets of instructions configured for execution by a computing device having control circuitry and memory, the one or more sets of instructions comprising instructions for: receiving video data including a current block, wherein: The current block has a size of N×M pixels, N and M are positive integers; generating a prediction block larger than the current block for the current block, wherein the prediction block has a size of (N+W)x(M+W) pixels, where W is a positive integer; and The generated prediction block is used to determine a window error of a sample window of samples of the current block.

17. The non-transitory computer readable storage medium of claim 16, wherein: The one or more sets of instructions also include instructions for: based on determining that the current block has two or more partitions: identifying a partition of the two or more partitions in which the sample is located; generating the sample window for the sample; as well as Error clipping is performed on one or more window samples located outside the identified partition, wherein the window error is determined after performing the error clipping.

18. The non-transitory computer readable storage medium of claim 17, wherein: The two or more partitions include two or more partition levels, and wherein the identified partition is a minimum partition level.

19. The non-transitory computer readable storage medium of claim 16, wherein: Determining the window error includes deriving a sum of squared errors (SSE) or a sum of absolute differences (SAD) for the sample window.

20. The non-transitory computer readable storage medium of claim 16, wherein: The one or more sets of instructions also include instructions for deriving a prediction weight for the current block using the determined window error.

Images