Hardware-friendly block partitioning combination decision in video coding

Abstract

The present disclosure relates to a hardware-friendly block partitioning combination decision in video coding. To effectively reduce complexity and enable hardware-friendly coding, a technique can be implemented to strategically select a partitioning combination from a set of one or more predetermined partitioning combinations based on a sharpness value of a CTU and optionally based on a temporal variation value of the CTU. In selecting the partitioning combination, appropriate thresholding can be applied to the sharpness value and / or the temporal variation value. The one or more threshold values used can be adapted to the QP used for coding. In some cases, the one or more threshold values used can also be adapted to the sharpness value. Selecting the partitioning combination from the set can reduce the computations associated with finding the optimal block partitioning and corresponding coding tools for the CUs in the CTU.

Description

Background Technology

[0001] Video compression is a technology used to make video files smaller and easier to transmit over the internet. Different methods and algorithms exist for video compression, each with varying performance and trade-offs. Video compression involves encoding and decoding. Encoding is the process of transforming (uncompressed) video data into a compressed format. Decoding is the process of recovering the video data from the compressed format. An encoder-decoder system is called a codec. Attached Figure Description

[0002] The embodiments will be readily understood through the following detailed description taken in conjunction with the accompanying drawings. For ease of description, similar reference numerals designate similar structural elements. In the accompanying figures, embodiments are illustrated by way of example rather than limitation.

[0003] Figure 1 An encoding system and several decoding systems according to some embodiments of the present disclosure are shown.

[0004] Figure 2 An exemplary encoder according to some embodiments of the present disclosure is shown to encode video frames and output an encoded bitstream.

[0005] Figure 3 An exemplary decoder according to some embodiments of the present disclosure is shown, which decodes an encoded bitstream and outputs a decoded video.

[0006] Figure 4 The division according to some embodiments of this disclosure is shown.

[0007] Figure 5 Intra-prediction modes according to some embodiments of this disclosure are shown.

[0008] Figure 6 The partitioning and combination decision and coding according to some embodiments of this disclosure are illustrated.

[0009] Figure 7 This is a flowchart illustrating a method for encoding a coding tree unit (CTU) according to some embodiments of the present disclosure.

[0010] Figure 8 This is a flowchart illustrating a method for determining one or more thresholds based on a quantization parameter (QP) and a sharpness value, according to some embodiments of the present disclosure.

[0011] Figure 9 This is a flowchart illustrating a method for selecting a partitioning combination according to some embodiments of the present disclosure.

[0012] Figure 10This is a flowchart illustrating a method for selecting a partitioning combination according to some embodiments of the present disclosure.

[0013] Figure 11 This is a flowchart illustrating a method for selecting a partitioning combination according to some embodiments of the present disclosure.

[0014] Figure 12 A block diagram of an exemplary computing device according to some embodiments of the present disclosure is depicted. Detailed Implementation

[0015] Overview

[0016] Video encoding, or video compression, is the process of storing, transmitting, and playing back compressed video data. Video compression can involve taking a large amount of raw video data and applying one or more compression techniques to reduce the amount of data required to represent the video while maintaining an acceptable level of visual quality. In some cases, video compression can provide efficient storage and transmission of video content over bandwidth-constrained networks.

[0017] Video comprises one or more video frames or a (temporal) sequence of frames. A frame with a larger frame index or associated with a later timestamp can be considered a frame in the forward direction relative to the current frame. A frame with a smaller frame index or associated with a previous timestamp can be considered a frame in the backward direction relative to the current frame. A frame can include an image or a single still image. A frame can have millions of pixels. For example, a frame of uncompressed 4K video can have a resolution of 3840 × 2160 pixels. Pixels can have luma / luminance and chroma / chrominance values. The terms "frame" and "image" can be used interchangeably.

[0018] There are several frame types for image types. I-frames (or Intra-frames) are the least compressible and do not depend on other frames for decoding. I-frames can include frames showing scene changes. I-frames can be reference frames for one or more other frames. P-frames can depend on data from previous frames for decoding and can be more compressible than I-frames. P-frames can be reference frames for one or more other frames. B-frames can depend on data from previous and forward frames for decoding and can be more compressible than I-frames and P-frames. B-frames can reference two or more frames, such as a future frame and a past frame. Other frame types can include reference B-frames and non-reference B-frames. Reference B-frames can act as a reference to another frame. Non-reference B-frames are not used as a reference to any frame. Reference B-frames are stored in the decoded image buffer, while non-reference B-frames do not need to be stored in the decoded image buffer. P-frames and B-frames can be referred to as inter-frames. The order or coding hierarchy of I-frames, P-frames, and B-frames can be called a group of pictures (GOP). In some cases, a frame can be an instantaneous decoder refresh (IDR) frame within a GOP. An IDR frame indicates that frames following an IDR frame cannot reference any frames preceding it. Therefore, an IDR frame signals to the decoder that it can clear the decoded picture buffer. Each IDR frame can be an I-frame, but an I-frame may or may not be an IDR frame. A closed GOP can begin with an IDR frame. A slice can be a spatially independent region of a frame that is encoded separately from any other region in the same frame.

[0019] Some recently released codecs can achieve a subjective quality improvement of approximately 50% at similar bit rates, offering a significant technical advantage over earlier codecs. However, the encoding and decoding complexity of these codecs has increased significantly. This increase in encoding and decoding complexity is due to the addition of new block partitioning and encoding tools. During encoding (e.g. Figure 2As shown, a frame can be divided into one or more blocks, which are used for block-based video compression. The way a larger block is divided into smaller blocks is called a partition or partitioning pattern. A block can include a square or non-square / rectangular region of a frame. Various video compression techniques may use different terms for blocks or different partitioning structures to create blocks. In some video compression techniques, a frame can be divided into CTUs. In some techniques, a CTU may be called a superblock or macroblock. A CTU can be 32×32 pixels, 64×64 pixels, 128×128 pixels, or larger. A CTU, for example, can be recursively divided into coding units (CUs) or smaller blocks (for the luma and chroma components, respectively) according to a tree structure. CUs can have sizes of 128×128, 64×64 pixels, 32×32 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, 8×8 pixels, 8×4 pixels, or 4×4 pixels. Each leaf node of this partitioning structure becomes a CU. These CUs serve as units for applying prediction (intra-frame prediction or inter-frame prediction) and transform coding.

[0020] Figure 4 The division of CTU to CU according to some embodiments of this disclosure is illustrated. For example... Figure 4 As seen, the codec allows flexible quadtree (QT) partitioning with nested multi-type tree (QT+MTT) partitioning. This allows each CTU to be recursively partitioned down to the minimum size using quadtree partitioning, after which further partitioning can occur using binary tree (BT) and ternary tree (TT) structures. Quadtree partitions divide a block into four squares of the same size. Binary tree partitions divide a block into two equal parts horizontally or vertically. Ternary tree partitions divide a block into three parts, with the middle part being twice the size of the other two parts. Recursive partitioning of CTUs greatly increases adaptability to partitioning across various content types. Starting with a 128×128 CTU, the codec can first apply QT partitioning down to the minimum allowed size (often 4×4 or 8×8), after which further BT and TT partitioning can be applied. Each partitioning decision creates new nodes in the partition tree, and each of these nodes has multiple potential partitioning options. Mathematical analysis shows that the number of possible partitioning combinations grows exponentially with the CTU size. There are approximately billions or even trillions of possible partitioning patterns for a 128×128 CTU.

[0021] Figure 5Intra-prediction modes according to some embodiments of this disclosure are illustrated. In a codec, 95 intra-prediction modes may be supported as possible coding tools for a CU. A given CU can be intra-predicted using one of these 95 possible intra-prediction modes. These 95 intra-prediction modes may include 65 (conventional) angle modes, 1 DC mode, 1 planar mode, and 28 wide-angle modes for non-square CUs.

[0022] To find the optimal partitioning and coding tools (e.g., intra-prediction modes) for CTU, the encoder calculates the bit rate and distortion for many feasible combinations of block partitioning and corresponding coding tools. Such computation leads to a significant and drastic increase in complexity. In some complexity comparisons between reference codes of two different codecs, the intra-coding runtime of this codec is more than 20 times slower than the earlier codec, and the inter-coding runtime is more than 7 times slower.

[0023] Reducing encoding complexity allows encoders to encode video faster and enables their use in more applications. Furthermore, reduced complexity makes the encoding process easier to execute within the hardware circuitry used to implement a video encoding pipeline. Several heuristics based on early skipping can be implemented to reduce complexity. However, these early skipping methods are designed for software encoding and involve a significant amount of on-the-fly decision-making. The complexity remains the same for worst-case scenarios, making hardware pipeline design potentially difficult. Consequently, early skipping methods may offer only a very small performance improvement for hardware-based encoding. While machine learning methods can analyze block statistics and reduce the number of partitioning candidates, these methods are still software-oriented. From a hardware perspective, there is no reduction in the number of gates or the worst-case complexity.

[0024] To more effectively reduce complexity and enable hardware-friendly coding, a technique can be implemented to strategically select a partition combination from a set of one or more predetermined partition combinations based on the sharpness value and optional temporal variation value of the CTU. A partition combination may include one or more predetermined candidate partitions. The sharpness value and / or temporal variation value can be extracted during pre-analysis. They can represent the temporal and spatial variability of the CTU. When selecting a partition combination, appropriate thresholding can be applied to the sharpness value and / or temporal variation value. The one or more thresholds used can be adapted to the QP used for coding. In some cases, the one or more thresholds used can be adapted to the sharpness value. Selecting a partition combination from this set reduces the computational cost associated with finding the optimal block partition and corresponding coding tool for the CU in the CTU.

[0025] In some embodiments, to select a partitioning combination, a threshold may be compared with a sharpness value determined for the CTU. In some embodiments, to select a partitioning combination, a threshold may be compared with a temporal variation value determined for the CTU. One or more sharpness values ​​determined for the CTU may be used to compare with one or more thresholds to select a partitioning combination for the CTU from a set of one or more (fixed) predetermined partitioning combinations. One or more temporal variation values ​​determined for the CTU may be used to compare with one or more thresholds to select a partitioning combination for the CTU from a set of one or more (fixed) predetermined partitioning combinations. The one or more thresholds may include one or more thresholds associated with inter-frame prediction partitioning and one or more thresholds associated with intra-frame prediction partitioning.

[0026] In some embodiments, the encoder may perform rate-distortion optimization on one or more predetermined candidate partitions of the selected partition combination. Ultimately, the encoder may select a partition from the one or more predetermined candidate partitions of the partition combination for encoding the CTU based on the rate-distortion optimization. Encoding complexity is significantly reduced by cutting down possible partitions and corresponding encoding tools through the partition combination decision process. In other words, fewer partitions and encoding tools are evaluated by the encoder.

[0027] Despite reduced coding complexity, partitioning decisions can still be made while maintaining quality gain. Sharpness and / or temporal variation values ​​can serve as effective metrics for selecting the optimal partitioning combination, and can be further evaluated by the encoder. Furthermore, the thresholds for QP and sharpness values ​​can be modified to adapt partitioning decisions to the content and encoding process. A partitioning combination will consist of a (fixed and predetermined) reduced set of possible partitions for the CTU. Therefore, dedicated hardware video encoding pipelines can be designed and implemented to support partitioning combination encoding with predetermined / fixed partitions, with a very small increase in the number of gates.

[0028] In some experiments involving the coding of video segments with different resolutions and characteristics, the partitioning combined decision technique can achieve a significant reduction in complexity along with very small or negligible quality effects.

[0029] The adaptive partitioning and combination decision-making techniques described and illustrated in this paper can be applied to various codecs, such as AVC (Advanced Video Coding), HEVC (High Efficiency Video Coding), AV1 (AOMedia Video 1), AV2 (AOMedia Video 2), VVC (Versatile Video Coding), and VP9. AVC, also known as "ITU-T H.264," was approved in 2003 and last revised on August 13, 2024. HEVC, also known as "ITU-T H.265," was approved in 2013 and last revised on April 3, 2024. AV1 is a video coding codec designed for video transmission over the Internet. Version 1.1.1 of the "AV1 Bitstream and Decoding Process Specification" (with errata) was last modified in 2019. AV2 is under development. VVC, also known as "ITU-T H.266", was finalized in 2020 and last revised on September 29, 2023. VP9 is an open video codec that became available on June 17, 2013, and last revised in October 2023.

[0030] Video compression

[0031] Figure 1 An encoding system 130 and one or more decoding systems 150 according to some embodiments of the present disclosure are shown. 1…D .

[0032] Encoding system 130 can Figure 12The encoding system 130 can be implemented on computing device 1200. Encoding system 130 can be implemented in the cloud or data center. Encoding system 130 can be implemented on a device used to capture video. Encoding system 130 can be implemented on a standalone computing system. Encoding system 130 can perform the encoding process in video compression. Encoding system 130 can receive a sequence of video (e.g., uncompressed video, original video, raw video, etc.) including video frames 104. Video frames 104 can include image frames or images that constitute the video. The video can have a frame rate, or frames per second (FPS), which defines the number of frames per second of the video. The higher the FPS, the more realistic and smooth the video looks. Typically, for a natural, realistic viewing experience for a human viewer, the FPS is greater than 24 frames per second. Examples of videos can include TV series, movies, short films, short videos (e.g., less than 15 seconds long), videos capturing gaming experiences, computer screen content, video conferencing content, live event broadcast content, sports content, surveillance video, videos shot using mobile computing devices (e.g., smartphones), etc. In some cases, videos may include a mix or combination of different types of videos.

[0033] The encoding system 130 may include an encoder 102 that receives video frames 104 and encodes the video frames 104 into an encoded bitstream 180. Figure 2 An exemplary implementation of encoder 102 is shown.

[0034] The encoded bitstream 180 can be compressed, meaning it can be smaller than video frame 104. The encoded bitstream 180 can include a series of bits (e.g., 0s and 1s). The encoded bitstream 180 can have header information, payload information, and trailer information, which can be encoded in the bitstream as bits. The header information can provide information about one or more of the following: the format of the encoded bitstream 180, the encoding process implemented in encoder 102, the parameters of encoder 102, and metadata of the encoded bitstream 180. For example, the header information can include one or more of the following: resolution information, frame rate, aspect ratio, color space, etc. The payload information can include data representing the content of video frame 104, such as sample frames, symbols, syntax elements, etc. For example, the payload information can include bits encoded for one or more of the following: the motion predictor, transform coefficients, prediction mode, and quantization level of video frame 104. The trailer information can indicate the end of the encoded bitstream 180. Trailing information may include other information, such as checksums, error-correcting codes, and signatures. The format of the encoded bitstream 180 can vary depending on the specifications of the encoding and decoding processes (i.e., the codec).

[0035] The encoded bitstream 180 may include data packets in which encoded video data and signaling information can be packaged. An exemplary format is the Open Bitstream Unit (OBU), used for AV1 encoded bitstreams. An OBU may include a header and a payload. The header may include information about the OBU, such as information indicating the type of OBU. Examples of OBU types may include: sequence header OBU, frame header OBU, metadata OBU, temporal delimiter OBU, and tile group OBU. The payload in the OBU may carry quantized transform coefficients and syntax elements, which can be used in the decoder to correctly decode the encoded video data to regenerate video frames.

[0036] The encoded bitstream 180 can be transmitted via network 140 to one or more decoding systems 150. 1…D Network 140 can be the Internet. Network 140 can include one or more of the following: cellular data network, wireless data network, wired data network, wired Internet network, fiber optic network, satellite Internet network, etc.

[0037] D decoding systems 150 are shown. 1…D Decoding system 150 1…D At least one of them can be Figure 12 Implemented on computing device 1200. System 150 1…D Examples can include personal computers, mobile computing devices, gaming devices, augmented reality devices, mixed reality devices, virtual reality devices, televisions, etc. Decoding system 150 1…D Each of these can perform the decoding process in video compression. Decoding system 150 1…D Each of these can include a decoder (e.g., decoder 1…D 162). 1…D ) and one or more display devices (e.g., display device 1…D 164) 1…D ). Figure 3 An exemplary implementation of a decoder (e.g., decoder 1 1621) is shown in the figure.

[0038] For example, decoding system 1 1501 may include decoder 1 1621 and display device 1 1641. Decoder 1 1621 can implement the decoding process of video compression. Decoder 1 1621 can receive encoded bitstream 180 and generate decoded video 1681. Decoded video 1681 may include a series of video frames, which may be a version or reconstructed version of video frame 104 encoded by encoding system 130. Display device 1 1641 can output decoded video 1681 for display to one or more human viewers or users of decoding system 1 1501.

[0039] For example, decoding system 2 1502 may include decoder 2 1622 and display device 2 1642. Decoder 2 1622 can implement the decoding process of video compression. Decoder 2 1622 can receive encoded bitstream 180 and generate decoded video 1682. Decoded video 1682 may include a series of video frames, which may be a version or reconstructed version of video frame 104 encoded by encoding system 130. Display device 2 1642 can output decoded video 1682 for display to one or more human viewers or users of decoding system 2 1502.

[0040] For example, the decoding system D 150 D It can include decoder D 162 D and display device D 164 D Decoder D 162 D This enables the decoding process for video compression. Decoder D 162 D It can receive 180 encoded bitstreams and produce 168 decoded video. D 168 decoded videos D It may include a series of video frames, which may be a version or reconstructed version of video frame 104 encoded by encoding system 130. Display device D 164 D It can output decoded video 168 D Used to decode the D150 system D Displayed to one or more human viewers or users.

[0041] Video encoder

[0042] Figure 2An encoder 102 according to some embodiments of the present disclosure is shown to encode video frames 104 and output an encoded bitstream. The encoder 102 may include one or more signal processing operations and data processing operations, including inter-frame and intra-frame prediction, transform, quantization, in-loop filtering, and entropy coding. The encoder 102 may include a reconstruction loop involving inverse quantization and inverse transform to ensure that the decoder sees the same reference blocks and frames. The encoder 102 may receive video frames 104 and encode them into an encoded bitstream 180. The encoder 102 may include one or more of the following: partitioning 206, transform and quantization 214, inverse transform and inverse quantization 218, in-loop filter 228, motion estimation 234, inter-frame prediction 236, intra-frame prediction 238, and entropy coding 216.

[0043] In some embodiments, video frame 104 may be processed by partition combination decision 294 to select a partition combination for evaluation by encoder 102 from a set of predetermined partition combinations. The partition combination may include one or more predetermined candidate partitions. Partition 206 may partition video frame 104 according to one or more predetermined candidate partitions. Various encoding tools for one or more predetermined candidate partitions may be evaluated by encoder 102.

[0044] In some embodiments, video frame 104 may be processed by preprocessing 290 before the encoder 102 applies the encoding process. Preprocessing 290 and encoder 102 can form a sequence as follows: Figure 1The encoding system 130 is shown. Preprocessing 290 can analyze video frame 104 to determine image statistics, which can be used to inform one or more encoding processes to be performed by one or more components in encoder 102. Preprocessing 290 can determine information that can be used for quantization (QP) adaptation, scene transition detection, and frame type adaptation. Preprocessing 290 can determine a recommended frame type for each frame. Preprocessing 290 can apply motion-compensated temporal filtering (MCTF) to denoise video frame 104. The filtered version of video frame 104 with MCTF applied can be provided to encoder 102 as an input video frame (instead of video frame 104), for example, to partition 206. MCTF can include motion estimation analysis operations and bilateral filtering operations. MCTF can attenuate random image components in a motion-aware manner to improve encoding efficiency. MCTF can operate on 8×8 pixel or 16×16 pixel blocks. MCTF can operate on luma and chroma values ​​separately. MCTF can be applied in three dimensions (e.g., two spatial directions and one temporal direction). MCTF can generate noise estimates for individual blocks. In some embodiments, one or more operations of preprocessing 290 can be implemented as software instructions executed by a processor. In some embodiments, one or more operations of preprocessing 290 can be implemented using computational circuitry designed to perform the one or more operations in hardware.

[0045] Partition 206 divides a frame of video frame 104 (or a filtered version of video frame 104 from preprocessing 290) into pixel blocks. Different codecs may allow different variable ranges of block sizes. In one codec, a frame can be divided by partition 206 into blocks of various sizes, such as 128×128, 64×64 pixels, 32×32, or 16×16 pixels. Large blocks may be referred to as superblocks, macroblocks, or CTUs. Partition 206 can further subdivide each large block using a multi-way partition tree structure. In some cases, a large block can be further recursively subdivided by partition 206 using a multi-way partition tree structure (e.g., down to smaller blocks, such as 4×4 size blocks). In one codec, a frame can be divided by partition 206 into CTUs of size 128×128 pixels. Partition 206 can use a QT partition structure to divide a CTU into four CUs. Partition 206 can further recursively subdivide the CUs using a QT partition structure. Partitioning 206 can use various tree structures (such as QT, BT, or TT structures, etc.) Figure 4(As shown) (Further subdividing the CU). The smallest CU can have a size of 4×4 pixels. In this paper, the CU can be referred to as a block. Subdivision 206 can output initial samples 208, for example, as pixel blocks.

[0046] In some cases, one or more operations in partition 206 can be implemented in intra-frame prediction 238 and / or inter-frame prediction 236.

[0047] Intra-frame prediction 238 can predict samples of blocks from previously encoded spatially adjacent / reference blocks in the same frame. Intra-frame prediction 238 can receive reconstructed prediction samples 226 (from previously encoded spatially adjacent blocks in the same frame). Reconstructed prediction samples 226 can be generated by adder 222 based on reconstructed prediction residual 224 and prediction sample 212. Intra-frame prediction 238 can determine a suitable predictor for predicting samples from previously encoded spatially adjacent / reference blocks in the same frame (thus making an intra-frame prediction decision). Intra-frame prediction 238 can generate prediction sample 212, which is generated using the suitable predictor. Intra-frame prediction 238 can output or identify the adjacent / reference blocks and predictors used to generate prediction sample 212. The identified adjacent / reference blocks and predictors can be encoded in the encoded bitstream 180 so that the decoder can reconstruct blocks using the same adjacent / reference blocks and predictors. In one codec, intra-prediction 238 can support a variety of different predictors, such as 56 different predictors. In another codec, intra-prediction 238 can support a variety of different predictors, such as 95 different predictors. Some predictors (e.g., orientation predictors) can capture different spatial redundancies in directional textures. The pixel values ​​of a block can be predicted using the orientation predictor in intra-prediction 238 by extrapolating the pixel values ​​of adjacent / reference blocks along a certain direction. Intra-prediction 238 in different codecs can support different sets of predictors to take advantage of different spatial patterns within the same frame. Examples of predictors can include DC, planar, paeth, smooth, smooth vertical, smooth horizontal, recursive filtering patterns, chroma-from-luma, intra-block copying (IBC), color palette or palette coding, multi-reference rows, intra-sub-partition, matrix-based intra-prediction (matrix coefficients can be defined using offline training of a neural network), angle prediction, wide-angle prediction, cross-component linear models, template matching, etc. IBC works by copying reference blocks within the same frame to predict the current block. Palette coding or palette patterns work by using a color palette with a small number of colors (e.g., 2-8 colors) and encoding the current block using the index of the color palette. In some cases, intra-prediction 238 can perform block prediction, where predicted blocks can be generated using vectors from reconstructed neighboring / reference blocks within the same frame. Optionally, some type of interpolation filter can be applied to the predicted blocks to blend the pixels of the predicted blocks. In intra-frame prediction 238, a vector compensation process can be used to predict the pixel values ​​of a block by shifting the neighboring / reference block (within the same frame) according to the vector (and optionally applying an interpolation filter to the neighboring / reference block) to generate prediction samples 212.Intra-prediction 238 can output or identify the vector applied to generate prediction samples 212. In some codecs, intra-prediction 238 can encode (1) the residual vector generated from the applied vector and the vector prediction candidates, and (2) information about the vector prediction candidates, rather than encoding the applied vector itself. Intra-prediction 238 can output or identify the type of interpolation filter applied to generate prediction samples 212.

[0048] Motion estimation 234 and inter-frame prediction 236 can predict samples of a block from samples of a previously encoded frame (e.g., a reference frame in the decoded image buffer 232). Motion estimation 234 and inter-frame prediction 236 can perform operations to make inter-frame prediction decisions. Motion estimation 234 can perform motion analysis and determine motion information for the current frame. Motion estimation 234 can determine the motion field of the current frame. The motion field can include motion vectors for the blocks in the current frame. Motion estimation 234 can determine the average magnitude of the motion vectors in the current frame. Motion estimation 234 can determine motion information that indicates how much motion is present in the current frame (e.g., large motion, very dynamic motion, small / slight motion, very static).

[0049] Motion estimation 234 and inter-frame prediction 236 can perform motion compensation, which may involve identifying suitable reference blocks and suitable motion predictors (or motion vector predictors) for a block, and optionally applying interpolation filters to the reference block. Motion estimation 234 can receive initial samples 208 from partition 206. Motion estimation 234 can receive samples (e.g., samples from previously encoded frames or reference frames) from decoded image buffer 232. Motion estimation 234 can use several reference frames to determine one or more suitable motion predictors. The motion predictor may include reference blocks and motion vectors that can be applied to generate motion-compensated blocks or predicted blocks. The motion predictor may include motion vectors that capture the motion of blocks between frames in the video. Motion estimation 234 can output or identify one or more reference frames and one or more suitable motion predictors. Inter-frame prediction 236 can apply one or more suitable motion predictors determined in motion estimation 234 and one or more reference frames to generate predicted samples 212. The identified reference frames and motion predictors can be encoded in an encoded bitstream 180 so that the decoder can reconstruct the block using the same reference frames and motion predictors. In one codec, motion estimation 234 can implement a single-reference-frame prediction mode, where a single reference frame with a corresponding motion predictor is used for inter-frame prediction 236. Motion estimation 234 can also implement a composite reference-frame prediction mode, where two reference frames with two corresponding motion predictors are used for inter-frame prediction 236. In one codec, motion estimation 234 can implement techniques for searching and identifying good reference frames that can generate the most effective motion predictor. The techniques in motion estimation 234 can include searching for good reference frame candidates both spatially (within the same frame) and temporally (within previously encoded frames). The techniques in motion estimation 234 can include searching deep spatial neighborhoods to find a spatial candidate pool. The techniques in motion estimation 234 can include using a temporal motion field estimation mechanism to generate a temporal candidate pool. The techniques in motion estimation 234 can use a motion field estimation process. After the temporal and spatial candidates can be sorted, and a suitable motion predictor can be determined, the inter-frame prediction 236 can support a variety of different motion predictors in one codec.Examples of predictors may include geometric motion vectors (complex, nonlinear motion), warp motion compensation (affine transformations capturing non-translational object motion), overlapping block motion compensation, advanced composite prediction (composite wedge prediction, differential modulation mask prediction, frame distance-based composite prediction, and composite inter-frame-intra-frame prediction), dynamic spatial and temporal motion vector references, affine motion compensation (capturing higher-order motions such as rotation, scaling, and shearing), adaptive motion vector resolution modes, geometric partitioning modes, bidirectional optical flow, prediction refinement with optical flow, weighted bidirectional prediction, extended merging prediction, etc. Optionally, some type of interpolation filter may be applied to the prediction block to blend the pixels of the prediction block. The pixel values ​​of the block can be predicted using the motion predictor / vector determined during motion compensation in motion estimation 234 and inter-frame prediction 236, and interpolation filters may be selectively applied. In some cases, inter-frame prediction 236 may perform motion compensation, where the prediction block can be generated from the reconstructed reference block of the reference frame using the motion predictor / vector. Inter-frame prediction 236 may output or identify the motion predictor / vector applied to generate prediction samples 212. In some codecs, inter-frame prediction 236 can encode (1) the residual vectors generated from the applied vectors and vector prediction candidates, and (2) information identifying the vector prediction candidates, rather than encoding the applied vectors themselves. Inter-frame prediction 236 can output or identify the type of interpolation filter applied to generate prediction samples 212.

[0050] Mode selection 230 can be notified by a component (e.g., motion estimation 234) to determine whether inter-frame prediction 236 or intra-frame prediction 238 is more efficient for encoding a block (thus making an encoding decision). Inter-frame prediction 236 can output prediction samples 212 for the prediction block. Inter-frame prediction 236 can output a selected predictor and a selected interpolation filter (if applicable), which can be used to generate the prediction block. Intra-frame prediction 238 can output prediction samples 212 for the prediction block. Intra-frame prediction 238 can output a selected predictor and a selected interpolation filter (if applicable), which can be used to generate the prediction block. Regardless of the mode, prediction residual 210 can be generated by subtractor 220 by subtracting prediction samples 212 from initial samples 208. In some cases, prediction residual 210 may include residual vectors from inter-frame prediction 236 and / or intra-frame prediction 238.

[0051] Transform and quantization 214 can receive prediction residual 210. Prediction residual 210 can be generated by subtractor 220, which takes initial sample 208 and subtracts prediction sample 212 to output prediction residual 210. Prediction residual 210 can be referred to as the prediction error of intra-frame prediction 238 and inter-frame prediction 236 (e.g., the error between initial sample and prediction sample 212). The prediction error has a smaller numerical range than the initial sample and can be encoded in the encoded bitstream 180 with fewer bits. Transform and quantization 214 can include one or more of transform and quantization. Transformation can include converting prediction residual 210 from the spatial domain to the frequency domain. Transformation can include applying one or more transform kernels. Examples of transform kernels can include horizontal and vertical forms of the Discrete Cosine Transform (DCT), the Asymmetric Discrete Sine Transform (ADST), the Inverted ADST, and the Identity Transform (IDTX), multiple transform selection, low-frequency non-separable transform, sub-block transform, non-square transform, DCT-VIII, Discrete Sine Transform VII (DST-VII), Discrete Wavelet Transform (DWT), etc. The transform can convert the prediction residuals 2^10 into transform coefficients. Quantization can quantize the transformed coefficients, for example, by reducing the precision of the transform coefficients. Quantization can include using a quantization matrix (e.g., linear and nonlinear quantization matrices) with quantization parameters or quantization step sizes. Elements in the quantization matrix can be larger for higher frequency bands and smaller for lower frequency bands, meaning that higher frequency coefficients are quantized more coarsely and lower frequency coefficients are quantized more finely. Quantization can include dividing each transform coefficient by the corresponding element in the quantization matrix (e.g., the quantization parameter) and rounding to the nearest integer. Effectively, the quantization matrix can implement different QPs for different frequency bands and chromaticity planes, and spatial prediction can be used. For each frame, a suitable quantization matrix can be selected and signaled, and encoded in the encoded bitstream 180. Transform and quantization 214 can output the quantized transform coefficients and syntax elements 278, which indicate the encoding mode and parameters used in the encoding process implemented in encoder 102.

[0052] In this article, QP refers to a parameter in video coding that controls the degree of compression by determining how much detail is preserved or discarded during the encoding process. QP is directly related to the quantization step size. A larger step size can result in greater loss of information but a smaller file size. A smaller step size can result in better information preservation but a larger file size. QP can have values ​​from 0 to 51. Lower QP values ​​(ranging from 0 to 20) result in minimal compression, preserving more detail and quality in the video, but they also result in larger file sizes. A medium range of QP values ​​(between 21 and 35) strikes a balance between video quality and file size, providing moderate compression suitable for most streaming applications where a balance between quality and bandwidth usage is required. Higher QP values ​​(from 36 to 51) apply more compression, resulting in significant quality loss, but they significantly reduce file size, which is useful for low-bandwidth scenarios. The exact range and impact of QP values ​​can vary depending on the specific coding standard and encoder settings. QP values ​​directly affect how DCT coefficients are divided and rounded in Transform & Quantization 214. Larger QP values ​​result in more aggressive rounding, effectively removing high-frequency details that are less perceptible to human vision. This parameter is used in the encoder's rate-distortion optimization process, allowing the encoder to balance visual quality with bandwidth constraints. Modern encoders can dynamically adjust both frame-level and macroblock-level QP values ​​to optimize compression based on scene complexity and motion. In some cases, incremental QP or QP offsets are used to adjust the base QP. Incremental QP (or QP offset) is a mechanism in video coding that allows relative adjustments to the base QP value used for a specific coding unit or frame type. These offsets enable transform and quantization to apply different levels of compression to different parts of the video stream, optimizing the balance between quality and bit rate. For example, since B-frames are less critical to overall quality, they typically use higher QP values ​​(positive increments) than I-frames, while visually important regions can receive negative incremental QPs to retain more detail. In many encoders, incremental QP can be configured for various structural elements, such as stripe type, layered coding layers, or regions of specific interest within a frame. This granular control in quantization helps achieve better subjective quality by allocating more bits to visually important content while maintaining effective compression for less noticeable areas.

[0053] In some embodiments, the QP used by the transform and quantization 214 is determined by the preprocessing 290. The preprocessing 290 may generate one or more quantization parameters used by the transform and quantization 214.

[0054] Inverse transform and dequantization 218 can apply the inverse operation of the operations performed in transform and quantization 214 to generate a reconstruction prediction residual 224 as part of the reconstruction path to generate a decoded image buffer 232 for encoder 102. Inverse transform and dequantization 218 can receive quantized transform coefficients and syntax elements 278. Inverse transform and dequantization 218 can perform one or more dequantization operations, such as applying a dequantization matrix, to obtain unquantized / initial transform coefficients. Inverse transform and dequantization 218 can perform one or more inverse transform operations, such as inverse transform (e.g., inverse DCT, inverse DWT, etc.), to obtain the reconstruction prediction residual 224. A reconstruction path is provided in encoder 102 to generate reference blocks and frames stored in decoded image buffer 232. The reference blocks and frames can be matched with blocks and frames to be generated in decoder. The reference blocks and frames are used as reference blocks and frames by motion estimation 234, inter-frame prediction 236, and intra-frame prediction 238.

[0055] Loop filter 228 can perform filtering to smooth artifacts introduced by encoding processes in encoder 102 (e.g., partitioning 206 and transform and quantization 214). Loop filter 228 can receive reconstructed prediction samples 226 from adder 222 and output frames to decoded image buffer 232. Examples of loop filters can include constrained low-pass filters, directional deringing filters, edge-guided conditional replacement filters, loop recovery filters, Wiener filters, self-guided recovery filters, constrained directional enhancement filters (CDEF), LMCS filters, sample adaptive offset (SAO) filters, adaptive loop filters (ALF), cross-component ALF, low-pass filters, deblocking filters, etc. For example, applying a deblocking filter across the boundary between two blocks can resolve block artifacts caused by the Gibbs phenomenon. In some embodiments, loop filter 228 can acquire data from a frame buffer containing reconstructed prediction samples 226 of individual blocks of a video frame. Loop filter 228 can determine whether to apply a loop filter. Loop filter 228 can determine one or more suitable filters to achieve good visual quality and / or appropriately remove artifacts introduced by the encoding process in encoder 102. Loop filter 228 can determine the type of loop filter to apply across the boundary between two blocks. Loop filter 228 can determine one or more strengths (e.g., filter coefficients) of the loop filter applied across the boundary between two blocks based on the reconstructed prediction samples 226 of the two blocks. In some cases, loop filter 228 can consider the desired bit rate when determining one or more suitable filters. In some cases, loop filter 228 can consider a specified QP when determining one or more suitable filters. Loop filter 228 can apply one or more (suitable) filters across the boundary separating two blocks. After applying the one or more (suitable) filters, loop filter 228 can write the (filtered) reconstructed samples to the frame buffer (e.g., decoded image buffer 232).

[0056] Entropy coding 216 can receive quantized transform coefficients and syntax elements 278 (referred to as symbols in this document) and perform entropy coding. Entropy coding 216 can generate and output an encoded bitstream 180. Entropy coding 216 can utilize statistical redundancy and apply lossless algorithms to encode symbols and produce a compressed bitstream, such as the encoded bitstream 180. Entropy coding 216 can implement several versions of arithmetic coding. Different versions may have different advantages and disadvantages. In one codec, entropy coding 216 can implement (symbol-to-symbol) adaptive multi-symbol arithmetic coding. In another codec, entropy coding 216 can implement a context-based adaptive binary arithmetic encoder (CABAC). Binary arithmetic coding differs from multi-symbol arithmetic coding. Binary arithmetic coding can only encode one bit at a time, such as a binary value with either 0 or 1. Binary arithmetic coding can first convert each symbol to a binary representation (e.g., using a fixed number of bits per symbol). Processing only binary values ​​of 0 or 1 simplifies computation and reduces complexity. Binary arithmetic coding assigns a probability to each binary value (e.g., the probability that a bit has a binary value of 0 and the probability that a bit has a binary value of 1). Multi-symbol arithmetic coding encodes an alphabet with at least two or three symbol values ​​and assigns a probability to each symbol value in that alphabet. Multi-symbol arithmetic coding can encode more bits at a time, which can result in fewer operations for encoding the same amount of data. Multi-symbol arithmetic coding can require more computation and storage (because probability estimates may be updated for every element in the alphabet). Maintaining and updating probabilities for each possible symbol value in multi-symbol arithmetic coding (e.g., cumulative probability estimates) can be more complex (e.g., complexity grows with the size of the alphabet). Multi-symbol arithmetic coding should not be confused with binary arithmetic coding because these two different entropy coding processes are implemented differently and can result in different encoded bitstreams for the same set of quantized transform coefficients and syntax elements 278.

[0057] Video Decoding

[0058] Figure 3A decoder 1 1621 according to some embodiments of the present disclosure is illustrated for decoding an encoded bitstream and outputting decoded video. Decoder 1 1621 may include one or more signal processing operations and data processing operations, including entropy decoding, inverse transform, inverse quantization, inter-frame and intra-frame prediction, loop filtering, etc. Decoder 1 1621 may have signal and data processing operations mirroring those performed in the encoder. Decoder 1 1621 may apply signal and data processing operations emitted as signals in the encoded bitstream 180 to reconstruct the video. Decoder 1 1621 may receive the encoded bitstream 180 and generate and output a decoded video 1681 having multiple video frames. The decoded video 1681 may be provided to one or more display devices for display to one or more human viewers. Decoder 1 1621 may include one or more of entropy decoding 302, inverse transform and inverse quantization 218, loop filtering 228, inter-frame prediction 236, and intra-frame prediction 238. Some functions have been previously described and used in encoders, such as... Figure 2 Encoder 102.

[0059] Entropy decoder 302 can decode the encoded bitstream 180 and output the symbols encoded in the encoded bitstream 180. These symbols may include quantized transform coefficients and syntax elements 278. Entropy decoder 302 can reconstruct these symbols from the encoded bitstream 180.

[0060] The inverse transform and dequantization 218 can receive the quantized transform coefficients and syntax elements 278, and perform the operations performed in the encoder. The inverse transform and dequantization 218 can output the reconstructed prediction residual 224. The adder 222 can receive the reconstructed prediction residual 224 and the prediction sample 212, and generate the reconstructed prediction sample 226. The inverse transform and dequantization 218 can output the syntax element 278, which has signaling information for notifying / instructing / controlling the operations in the decoder 1 1621 (such as mode selection 230, intra-frame prediction 238, inter-frame prediction 236, and loop filter 228).

[0061] Depending on the prediction mode signaled in the encoded bitstream 180 (e.g., as transform coefficients in quantization and syntax elements in syntax element 278), intra-frame prediction 238 or inter-frame prediction 236 can be applied to generate prediction samples 212.

[0062] Adder 222 can add the predicted sample 212 of the decoded reference block to the reconstructed prediction residual 224 to produce the reconstructed prediction sample 226 of the reconstructed block. For intra-frame prediction 238, the decoded reference block can be in the same frame as the block being decoded or reconstructed. For inter-frame prediction 236, the decoded reference block can be in a different (reference) frame in the decoded image buffer 232.

[0063] Intra-frame prediction 238 can determine a reconstructed vector based on the residual vector and selected vector predictor candidates. Intra-frame prediction 238 can apply a reconstructed predictor or vector (e.g., based on predictor information provided by a signal) to a reconstructed block that can be generated using a decoded reference block from the same frame. Intra-frame prediction 238 can apply an appropriate interpolation filter type (e.g., based on interpolation filter information provided by a signal) to the reconstructed block to generate prediction samples 212.

[0064] Inter-frame prediction 236 can determine reconstructed vectors based on residual vectors and selected vector predictor candidates. Inter-frame prediction 236 can apply a reconstructed predictor or vector (e.g., based on predictor information signaled) to a reconstructed block generated using decoded reference blocks from different frames of the decoded image buffer 232. Inter-frame prediction 236 can apply an appropriate interpolation filter type (e.g., based on interpolation filter information signaled) to the reconstructed block to generate predicted samples 212.

[0065] Loop filter 228 can receive reconstructed prediction sample 226. Loop filter 228 can apply the reconstructed prediction sample 226 to one or more filters signaled in the encoded bitstream 180. Loop filter 228 can output decoded video 1681.

[0066] Hardware-friendly block partitioning decision

[0067] Figure 6 The partitioning combination decision 294 and encoder 102 are illustrated according to some embodiments of the present disclosure. Figure 6 One or more components shown can perform Figure 7 One or more operations of the method 700 used for encoding CTU.

[0068] The partitioning combination decision 294 can receive video frames 104. The partitioning combination decision 294 can receive a CTU (Category Unit) of one video frame from the video frames 104. The CTU can be a 128×128 pixel block, or have a size determined by the encoder 102. For this CTU, the partitioning combination decision 294 can extract information about the CTU, such as one or more spatial sharpness features and temporal variation features. Based on the information about the CTU and by applying appropriate thresholding, the partitioning combination decision 294 can output a partitioning combination selected from a set of predetermined / fixed partitioning combinations. The selected partitioning combination can be provided to the encoder 102 to control the encoding process implemented by the encoder 102.

[0069] The partitioning and combination decision 294 may include sharpness calculation 602. Sharpness calculation 602 can determine the sharpness value of the CTU in a frame of video frame 104 for encoding by encoder 102. Sharpness calculation 602 can perform... Figure 7 The estimated spatial sharpness in 702.

[0070] In some embodiments, sharpness calculation 602 can determine the block sharpness value of one or more (32×32) blocks of the CTU. A CTU with a size of 128×128 can have 16 32×32 blocks. A CTU with a size of 64×64 can have 4 32×32 blocks. Sharpness calculation 602 can determine the block sharpness value of one 32×32 block of the CTU. Sharpness calculation 602 can determine the block sharpness value of each 32×32 block of the CTU. The block sharpness value S of one 32×32 block of the CTU. i It can be calculated as follows:

[0071]

[0072] N is a constant weighting factor. P(x, y) represents the pixel value at position (x, y) in a 32×32 block. The block sharpness value quantifies the difference between two adjacent pixels in a 32×32 block (e.g., |P(x, y) - P(x, y+1)| and |(P(x, y) - P(x+1, y)|). In one implementation, N = 1024. The CTU's (spatial) sharpness value S... final It is the maximum block sharpness value among all 32×32 blocks in CTU:

[0073]

[0074] In some embodiments, sharpness calculation 602 can calculate one or more sharpness values ​​for one or more blocks of a coding tree unit. Sharpness calculation 602 can select the maximum sharpness value from the one or more sharpness values ​​as the sharpness value of the coding tree unit.

[0075] In some cases, the CTU sharpness value can be the minimum value among the block sharpness values. In some cases, the CTU sharpness value can be the mean, mode, or median value among the block sharpness values. In some cases, the CTU sharpness value can be an eigenvalue among the block sharpness values.

[0076] The sharpness value for one or more blocks can be provided to the thresholding and partitioning combination selection 608. The sharpness value for CTU can be provided to the thresholding and partitioning combination selection 608. The sharpness value for CTU can be provided to the adaptive thresholding calculation 606.

[0077] The partitioning combination decision 294 may include adaptive threshold calculation 606. Adaptive threshold calculation 606 may receive QP 620. Adaptive threshold calculation 606 may determine one or more thresholds based on QP 620. Adaptive threshold calculation 606 may determine one or more thresholds based on frame type 622. Adaptive threshold calculation 606 may determine one or more thresholds associated with inter-frame prediction partitioning, such as a threshold associated with inter-frame prediction partitioning of size 128×128, a threshold associated with inter-frame prediction partitioning of size 64×64, and a threshold associated with inter-frame prediction partitioning of size 32×32. Adaptive threshold calculation 606 may determine one or more thresholds associated with intra-frame prediction partitioning, such as a threshold associated with intra-frame prediction partitioning of size 64×64 and a threshold associated with intra-frame prediction partitioning of size 32×32. Various thresholds may be used as decision points in a decision logic / tree to select a partitioning combination from a set of fixed / predetermined partitioning combinations. Exemplary operation of adaptive threshold calculation 606 is described in... Figure 8 As shown in the diagram. Adaptive threshold calculation 606 can be performed. Figure 7 The determination of QP and sharpness adaptive threshold in 704.

[0078] The partitioning decision 294 may include time variation calculation 604. Time variation calculation 604 can determine the time variation value of the CTU. In response to frame type 622 indicating that the frame is not an intra-frame frame or a frame to be predicted inter-frame, time variation calculation 604 can determine the time variation value.

[0079] refer to Figure 7 In step 706, a check can be performed on frame type 622 to determine whether frame type 622 is an intra frame. If frame type 622 is not an intra frame, method 700 can proceed to step 710, where time change calculation 604 can determine the time change value of the CTU. If frame type 622 is, conversely, an intra frame, method 700 can proceed to step 708.

[0080] Reference Figure 6The temporal variation calculation 604 can determine the motion prediction cost of one or more (32×32) blocks of the CTU. The motion prediction cost is the estimated motion compensation cost of the block. A CTU of size 128×128 can have 16 32×32 blocks. A CTU of size 64×64 can have 4 32×32 blocks. The temporal variation calculation 604 can determine the motion prediction cost of one 32×32 block of the CTU. The temporal variation calculation 604 can determine the motion prediction cost of each 32×32 block of the CTU. The temporal variation calculation 604 performs a motion search on a downsampled or reduced version of the frame. In some embodiments, the frame is downsampled by a factor of 4, and therefore one 32×32 block of the CTU is downsampled to an 8×8 block. The temporal variation calculation 604 can perform an 8×8 block motion search to find a reference with the minimum inter-frame prediction cost for each 8×8 block (a downsampled version of one 32×32 block of the CTU). The minimum inter-frame prediction cost is used as the motion prediction cost V for a 32×32 block (or downsampled 8×8 block) of CTU. i For the time variation value V of CTU final It is the maximum value among all motion prediction values ​​in the CTU (32×32 blocks or downsampled 8×8 blocks):

[0081]

[0082] The maximum motion prediction value among all 8×8 blocks in the current downsampled CTU is used as the time variation value.

[0083] In some cases, the time variation value of CTU can be the minimum value among the motion prediction cost values. In some cases, the time variation value of CTU can be the mean, mode, or median value among the motion prediction cost values. In some cases, the time variation value of CTU can be an eigenvalue among the motion prediction cost values.

[0084] The time variation calculation 604 can calculate one or more motion prediction values ​​for one or more blocks of a coding tree unit. The time variation calculation 604 can select the maximum motion prediction value from these one or more motion prediction values ​​as the time variation value of the coding tree unit.

[0085] The one or more motion prediction values ​​can be provided to the thresholding and partitioning combination selection 608. The time variation value of CTU can be provided to the thresholding and partitioning combination selection 608.

[0086] In response to frame type 622 being intra-predictive, one or more block sharpness values ​​of one or more 32×32 blocks of the CTU, along with the CTU's sharpness value, can be used by thresholding and partition combination selection 608 to select an intra-partition combination from a set of fixed / predetermined partition combinations. Thresholding and partition combination selection 608 can compare one or more block sharpness values ​​of one or more 32×32 blocks of the CTU and the CTU's sharpness value with one or more thresholds. An exemplary operation of thresholding and partition combination selection 608 in selecting an intra-partition combination is described in... Figure 9 As shown in the diagram. Thresholding and partitioning combination selection 608 can be performed in Figure 7 The 708 selection of partitioning combinations is based on threshold and spatial sharpness.

[0087] In some embodiments, the thresholding and partitioning combination selection 608 selects a partitioning combination from a set of one or more predetermined partitioning combinations based on one or more thresholds and sharpness values. In some embodiments, the thresholding and partitioning combination selection 608 selects a partitioning combination from a set of one or more predetermined partitioning combinations based on one or more thresholds, one or more block sharpness values, and sharpness values. The partitioning combination has or includes one or more predetermined candidate partitions.

[0088] In response to frame type 622 being inter-predictive (not intra-predictive), the motion prediction values ​​of one or more 32×32 blocks of the CTU and the temporal change value of the CTU can be used by thresholding and partition combination selection 608 to select an inter-split combination from a set of fixed / predetermined partition combinations. Thresholding and partition combination selection 608 can compare the motion prediction values ​​of one or more 32×32 blocks of the CTU and the temporal change value of the CTU with one or more thresholds. In some embodiments, the block sharpness values ​​of one or more 32×32 blocks of the CTU and / or the sharpness value of the CTU can be used by thresholding and partition combination selection 608 to determine whether to skip intra-prediction. Thresholding and partition combination selection 608 can compare the block sharpness values ​​of one or more 32×32 blocks of the CTU and / or the sharpness value of the CTU with one or more thresholds. Exemplary operations of thresholding and partition combination selection 608 in selecting inter-split combinations and whether to skip intra-prediction are described in... Figure 10 As shown in the diagram. Thresholding and partitioning combination selection 608 can be performed in Figure 7 The 712 is based on a combination of threshold, time variation, and (optionally) spatial sharpness selection for partitioning.

[0089] In some embodiments, the thresholding and segmentation combination selection 608 selects a segmentation combination from the group by further selecting the segmentation combination based on a time-varying value. In some embodiments, the thresholding and segmentation combination selection 608 selects a segmentation combination from the group by further selecting the segmentation combination based on one or more motion prediction values ​​and time-varying values. One or more values ​​associated with time-varying values ​​and one or more values ​​associated with spatial sharpness are used in combination to select inter-frame segmentation combinations and intra-frame segmentation combinations.

[0090] After thresholding and partition combination selection 608 determines the partition combination, encoder 102 encodes the CTU according to the candidate partition selected from the partition combination.

[0091] To select candidate partitions, rate-distortion optimization (RDO) 680 in encoder 102 can perform RDO on one or more predetermined candidate partitions of the selected partition combination. RDO 680 can select a partition from one or more predetermined candidate partitions of the partition combination for CTU encoding based on rate-distortion optimization. RDO 680 can also select one or more encoding tools from one or more candidate encoding tools for CTU encoding based on rate-distortion optimization. RDO 680 can perform... Figure 7 RDO in 714.

[0092] In some embodiments, the RDO 680 can balance the trade-off between compression efficiency and video quality. The RDO 680 can select optimal encoding parameters to minimize distortion (quality loss) at a given bitrate, or conversely, to achieve the best possible quality for a given bitrate. The RDO 680 evaluates different encoding options, such as different CTU partitions and possible encoding tools, and calculates their impact on both bitrate and the resulting video quality. The RDO 680 can select parameters that provide the best trade-off between these two factors, ensuring that the encoded video maintains high visual fidelity while keeping the file size manageable.

[0093] In modern image and video coding standards, without RDO 680 in encoder 102, many high-level features (such as block partitioning decisions) may not be economical. The equations used in RDO 680 can be based on the Lagrange multiplier method. The objective is to minimize the cost function, which combines bit rate (bit cost) and distortion (quality loss). The distortion metric D is based on the sum of squared differences and / or mean squared error, and the bit rate R includes all bits associated with the decision, including header (information), motion (information), side information, and transform data. By using P to represent the parameter vector to be decided, the optimal solution is obtained by solving the following unconstrained Lagrange formula:

[0094]

[0095] λ controls rate-distortion tradeoffs. In some embodiments, λ is a function of QP 620 and frame type 622:

[0096] λ=f(QP,frame type )

[0097] QP and / or sharpness adaptive threshold

[0098] Figure 8 This is a flowchart illustrating a method 800 for determining one or more thresholds based on QP and (optionally) sharpness values, according to some embodiments of the present disclosure. Method 800 may be derived from... Figure 6 The adaptive threshold calculation is performed in 606.

[0099] In 802, the threshold T is initialized:

[0100] T=(λ+c)*W

[0101] c is a constant offset, and W is a constant weighting parameter. In other words, the initial threshold can be based on... Figure 6 The threshold T is determined using λ, which is used in the RDO680. As discussed earlier, λ is a function of QP 620. The initial threshold T is determined based on QP.

[0102] In 804, a check is performed to determine the sharpness value S for CTU. final Is it less than the fixed / predetermined value B?

[0103] In 806, in response to determining S final <B(804), the initial threshold is adjusted based on the sharpness value. The adjusted threshold T s The following can be calculated:

[0104]

[0105] If S final ≥B, the initial threshold is directly used as the adjusted threshold.

[0106] In 808, one or more thresholds associated with inter-frame prediction partitioning are determined based on an initial threshold or an adjusted threshold.

[0107] In some embodiments, the threshold T associated with the inter-frame prediction partitioning of a partition size of 128×128 128inter It can be determined as follows:

[0108] T 128inter =T s / 2

[0109] In some embodiments, the threshold T associated with the inter-frame prediction partitioning of a partition size of 64×64 64inter It can be determined as follows:

[0110] T 64inter =T s

[0111] In some embodiments, the threshold T associated with the inter-frame prediction partitioning of a partition size of 32×32 32inter It can be determined as follows:

[0112] T 32inter =2*T s

[0113] In 810, one or more thresholds associated with inter-frame prediction partitioning are determined based on an initial threshold or an adjusted threshold.

[0114] In some embodiments, the threshold T associated with intra-frame prediction partitioning of partition size 64×64 64intra It can be determined as follows:

[0115] T 64intra =d*T 64inter =d*T s

[0116] d is a constant weighting parameter.

[0117] In some embodiments, the threshold T associated with intra-frame prediction partitioning of partition size 32×32 32intra It can be determined as follows:

[0118] T 32intra =d*T 32inter =d*2T s

[0119] Select intra-frame partitioning combination

[0120] Figure 9 This is a flowchart illustrating a method 900 for selecting a partition combination according to some embodiments of the present disclosure. Method 900 may begin at point A and end at point B. Method 900 may be... Figure 6 The thresholding and partitioning combination selection is performed in step 608. Method 900 can be performed to select intra-predictive partitioning combinations for intra-frames.

[0121] Method 900 may include comparing a sharpness value (or one or more block sharpness values) with one or more thresholds and selecting a partitioning combination based on the comparison. A set of fixed / predefined intra-prediction partitioning combinations may include:

[0122] • Intra-frame combination 1: 8×8, 8×4, 4×8, and 4×4 are disabled; others are allowed.

[0123] • Intra-frame combination 2: 64×64, 8×4, 4×8, and 4×4 are disabled; others are allowed.

[0124] • Intraframe Combination 3: 32×32 and below are allowed.

[0125] In 902, a check is performed to determine whether the sharpness value for CTU is less than a threshold associated with an intra-prediction partition of partition size 64×64. final <T 64intra If yes, proceed from method 900 to 904. If no, proceed from method 900 to 906.

[0126] In 904, intra-frame partitioning combination 1 is selected.

[0127] In 906, the partitioning at 64×64 for intra-frame prediction is skipped.

[0128] In 908, an inspection is performed to determine whether the block sharpness value of the 32×32 block in the CTU is less than a threshold associated with the intra-prediction partitioning of the partition size 32×32. i <T 32intra If yes, proceed from method 900 to 910. If no, proceed from method 900 to 912.

[0129] In 910, intra-frame partitioning combination 2 was selected.

[0130] In 912, intra-frame partitioning combination 3 was selected.

[0131] Select inter-frame partitioning combination

[0132] Figure 10 This is a flowchart illustrating a method for selecting a partition combination according to some embodiments of the present disclosure. Method 1000 may begin at C and end at D. Method 1000 may be... Figure 6 The thresholding and partitioning combination selection is performed in step 608. Method 1000 can be performed to select inter-prediction partitioning combinations and / or intra-prediction partitioning combinations for non-intra-frame or inter-frame frames.

[0133] Method 1000 may include comparing time values ​​(or one or more motion prediction values) with one or more thresholds, for example, in response to the frame type being an inter-frame prediction frame, and selecting a partitioning combination based on the comparison. Method 1000 may also include comparing time variation values ​​with thresholds. A set of fixed / predefined inter-frame prediction partitioning combinations may include:

[0134] • Inter-frame combination 1: 128×128 is enabled; 16×8, 8×16, 16×4, 4×16, 8×8, 8×4 and 4×8 are disabled; others are allowed.

[0135] • Inter-frame combination 2: 128×128, 8×8, 8×4, and 4×8 are disabled; others are allowed.

[0136] • Inter-frame combination 3: 128×128, 64×64, 8×4, 4×8, and 4×4 are disabled; others are allowed.

[0137] • Inter-frame combination 4: 32×32 and below are allowed.

[0138] In 1002, a check is performed to determine whether the time variation value for CTU is less than a threshold associated with the inter-frame prediction partition of partition size 128×128, V. final <T 128inter If yes, method 1000 proceeds to 1004. If no, method 1000 proceeds to 1012.

[0139] In 1004, inter-frame partitioning combination 1 is selected.

[0140] In 1006, a check is performed to determine whether the sharpness value for CTU is greater than (or equal to) a threshold associated with intra-prediction partitioning of partition size 64×64. final ≥T 64intra If yes, proceed from method 1000 to 1008. If no, proceed from method 1000 to... Figure 9 904.

[0141] In 1008, intra-frame prediction is skipped for CTU.

[0142] In 1012, a check is performed to determine whether the temporal variation value for CTU is less than a threshold associated with the inter-frame prediction partition of a partition size of 64×64, V. final <T 64inter If yes, proceed from method 1000 to 1014. If no, proceed from method 1000 to 1016.

[0143] In 1014, inter-frame partitioning combination 2 was selected.

[0144] In 1016, the divisions for inter-frame prediction at 128×128 and 64×64 are skipped.

[0145] In 1018, a check is performed to determine whether the motion prediction cost of a 32×32 block in the CTU is less than a threshold associated with the inter-frame prediction partition of the partition size 32×32, V. i <T 32interIf yes, proceed from method 1000 to 1020. If no, proceed from method 1000 to 1022.

[0146] In 1020, inter-frame partitioning combination 3 was selected.

[0147] In 1022, inter-frame partitioning combination 4 was selected.

[0148] exist Figure 9-10 The partitioning combinations shown are illustrative, and it is anticipated that other partitioning combinations can be predetermined / fixed.

[0149] Methods for selecting optimal partitioning combinations

[0150] Figure 11 This is a flowchart illustrating a method for selecting a partition combination according to some embodiments of the present disclosure. Method 1100 may be performed by one or more components of the partition combination decision 294 in the figure and / or one or more components of the encoder 102.

[0151] In 1102, the partitioning combination decision 294 can determine the sharpness value of the CTU in the frame of the video encoded by encoder 102.

[0152] In 1104, the partitioning combination decision 294 can determine one or more thresholds based on QP.

[0153] In 1106, the partition combination decision 294 can select a partition combination from a set of one or more predetermined partition combinations based on one or more thresholds and sharpness values. The partition combination may include one or more predetermined candidate partitions.

[0154] In 1108, encoder 102 can encode CTU according to candidate partitions selected from the partition combination.

[0155] Exemplary computing device

[0156] Figure 12 This is a block diagram of an apparatus or system (e.g., an exemplary computing device 1200) according to some embodiments of the present disclosure. One or more computing devices 1200 may be used to implement the functions shown in the figures and described herein. Figure 12 Many of the components shown may be included in computing device 1200, but any one or more of these components may be omitted or copied to suit the application. In some embodiments, some or all of the components included in computing device 1200 may be attached to one or more motherboards. In some embodiments, some or all of these components are fabricated on a single system-on-a-chip (SoC) die. Additionally, in various embodiments, computing device 1200 may not include... Figure 12The computing device 1200 may include one or more components shown in the diagram, and may include interface circuitry for coupling to one or more components. For example, the computing device 1200 may not include a display device 1206, but may include display device interface circuitry (e.g., connectors and driver circuitry) to which the display device 1206 may be coupled. In another set of examples, the computing device 1200 may not include an audio input device 1218 or an audio output device 1208, but may include audio input or output device interface circuitry (e.g., connectors and support circuitry) to which the audio input device 1218 or the audio output device 1208 may be coupled.

[0157] Computing device 1200 may include processing device 1202 (e.g., one or more processing devices, one or more processing devices of the same type, or one or more processing devices of different types). Processing device 1202 may include processing circuitry or electronic circuitry that processes electronic data from data storage elements (e.g., registers, memories, resistors, capacitors, qubit units) to transform the electronic data into other electronic data that can be stored in registers and / or memories. Examples of processing device 1202 may include CPUs, GPUs, quantum processors, machine learning processors, artificial intelligence processors, neural network processors, artificial intelligence accelerators, application-specific integrated circuits (ASICs), analog signal processors, analog computers, microprocessors, digital signal processors, field-programmable gate arrays (FPGAs), tensor processing units (TPUs), data processing units (DPUs), etc.

[0158] Computing device 1200 may include memory 1204, which may itself include one or more memory devices, such as volatile memory (e.g., DRAM), non-volatile memory (e.g., read-only memory (ROM)), high-bandwidth memory (HBM), flash memory, solid-state memory, and / or hard disk drives. Memory 1204 includes one or more non-transitory computer-readable storage media. In some embodiments, memory 1204 may include memory that shares a die with processing device 1202.

[0159] In some embodiments, memory 1204 includes one or more non-transitory computer-readable media storing executable instructions to perform the operations described herein, such as Figure 1-3 and Figure 6The operations illustrated in methods 700, 800, 900, 1000, and 1100. In some embodiments, memory 1204 includes one or more non-transitory computer-readable media storing executable instructions to perform one or more operations of encoder 102. In some embodiments, memory 1204 includes one or more non-transitory computer-readable media storing executable instructions to perform one or more operations of partitioning combination decision 294. In some embodiments, memory 1204 includes one or more non-transitory computer-readable media storing executable instructions to perform one or more operations of RDO 680. In some embodiments, memory 1204 includes one or more non-transitory computer-readable media storing executable instructions to perform one or more operations of encoder 102. The instructions stored in memory 1204 can be executed by processing device 1202.

[0160] In some embodiments, memory 1204 may store data, such as data structures, binary data, bits, metadata, files, blobs, etc., as shown in the figures and described herein. Memory 1204 may include one or more non-transitory computer-readable media storing one or more of the following: input frames to the encoder (e.g., video frame 104), intermediate data structures computed by the encoder, bitstreams generated by the encoder (e.g., encoded bitstream 180), bitstreams received by the decoder (e.g., encoded bitstream 180), intermediate data structures computed by the decoder, and reconstructed frames generated by the decoder. Memory 1204 may include one or more non-transitory computer-readable media storing one or more of the following: data received and / or generated by partitioning combination decision 294. Memory 1204 may include one or more non-transitory computer-readable media storing one or more of the following: data received and / or generated by partitioning combination decision 294. Figure 11 Method 1100 receives and / or generates data.

[0161] In some embodiments, computing device 1200 may include communication device 1212 (e.g., one or more communication devices). For example, communication device 1212 may be configured to manage wired and / or wireless communication for the transmission of data from and to computing device 1200. The term “wireless” and its derivatives may be used to describe circuits, devices, systems, methods, technologies, communication channels, etc., that transmit data over a non-solid medium by using modulated electromagnetic radiation. This term does not imply that the associated device does not contain any wires, although they may not in some embodiments. Communication device 1212 may implement any of a number of wireless standards or protocols, including but not limited to Institute of Electrical and Electronics Engineers (IEEE) standards, including Wi-Fi (IEEE 1202.10 family), IEEE 1202.16 standards (e.g., IEEE 1202.16-2005 amendment), Long Term Evolution (LTE) projects and any amendments, updates, and / or revisions thereof (e.g., Advanced LTE project, Ultra Mobile Broadband (UMB) project (also known as “3GPP2”), etc.). IEEE 1202.16 compliant Broadband Wireless Access (BWA) networks are commonly referred to as WiMAX networks. The acronym stands for Worldwide Interoperability for Microwave Access, a certification mark for products that have passed compliance and interoperability testing with the IEEE 1202.16 standard. Communication equipment 1212 can operate according to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE networks. Communication equipment 1212 can also operate according to Enhanced Data Rate GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). Communication device 1212 can operate according to Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolved Data Optimization (EV-DO) and its derivatives, as well as any other wireless protocol designated as 3G, 4G, 5G, and above. In other embodiments, communication device 1212 can operate according to other wireless protocols. Computing device 1200 may include antenna 1222 to facilitate wireless communication and / or receive other wireless communications (such as radio frequency transmissions). Computing device 1200 may include receiving circuitry and / or transmitting circuitry. In some embodiments, communication device 1212 can manage wired communication, such as electrical, optical, or any other suitable communication protocol (e.g., Ethernet). As described above, communication device 1212 may include multiple communication chips.For example, the first communication device 1212 may be dedicated to short-range wireless communication, such as Wi-Fi or Bluetooth, while the second communication device 1212 may be dedicated to long-range wireless communication, such as Global Positioning System (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some embodiments, the first communication device 1212 may be dedicated to wireless communication, while the second communication device 1212 may be dedicated to wired communication.

[0162] The computing device 1200 may include a power supply / power circuit 1214. The power supply / power circuit 1214 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of the computing device 1200 to a power source (e.g., DC power, AC power, etc.) that is separate from the computing device 1200.

[0163] The computing device 1200 may include a display device 1206 (or a corresponding interface circuit as discussed above). For example, the display device 1206 may include any visual indicator, such as a head-up display, computer monitor, projector, touch screen display, liquid crystal display (LCD), light-emitting diode display, or flat panel display.

[0164] The computing device 1200 may include an audio output device 1208 (or a corresponding interface circuit as discussed above). For example, the audio output device 1208 may include any device that generates an audible indicator, such as a speaker, headphones, or earbuds.

[0165] The computing device 1200 may include an audio input device 1218 (or a corresponding interface circuit as discussed above). The audio input device 1218 may include any device that generates a signal representing sound, such as a microphone, microphone array, or digital musical instrument (e.g., a musical instrument with a Musical Instrument Digital Interface (MIDI) output).

[0166] The computing device 1200 may include a GPS device 1216 (or a corresponding interface circuit as discussed above). The GPS device 1216 may communicate with a satellite-based system and may receive the location of the computing device 1200, as is known in the art.

[0167] Computing device 1200 may include sensor 1230 (or one or more sensors). Computing device 1200 may include the corresponding interface circuitry discussed above. Sensor 1230 can sense physical phenomena and convert them into electrical signals that can be processed by, for example, processing device 1202. Examples of sensor 1230 may include: capacitive sensors, inductive sensors, resistive sensors, electromagnetic field sensors, light sensors, cameras, imagers, microphones, pressure sensors, temperature sensors, vibration sensors, accelerometers, gyroscopes, strain sensors, humidity sensors, distance sensors, ranging sensors, time-of-flight sensors, pH sensors, particle sensors, air quality sensors, chemical sensors, gas sensors, biosensors, ultrasonic sensors, scanners, etc.

[0168] The computing device 1200 may include another output device 1210 (or a corresponding interface circuit as discussed above). Examples of other output devices 1210 may include audio codecs, video codecs, printers, wired or wireless transmitters for providing information to other devices, haptic output devices, gas output devices, vibration output devices, lighting output devices, home automation controllers, or additional storage devices.

[0169] The computing device 1200 may include another input device 1220 (or a corresponding interface circuit as discussed above). Examples of other input devices 1220 may include an accelerometer, gyroscope, compass, image capture device, keyboard, cursor control device (such as a mouse), stylus, touchpad, barcode reader, quick-response (QR) code reader, any sensor, or radio frequency identification (RFID) reader.

[0170] The computing device 1200 can have any desired form factor, such as a handheld or mobile computer system (e.g., mobile phone, smartphone, mobile internet device, music player, tablet, laptop, netbook, personal digital assistant (PDA), ultra-portable personal computer, remote control, wearable device, head-mounted device, glasses, footwear, electronic clothing, etc.), desktop computer system, server or other networked computing component, printer, scanner, monitor, set-top box, entertainment control unit, vehicle control unit, digital camera, digital video recorder, Internet of Things device, or wearable computer system. In some embodiments, the computing device 1200 can be any other electronic device that processes data.

[0171] Select Example

[0172] Example 1 provides a method comprising: determining a sharpness value of a coding tree unit in a frame of a video to be encoded; determining one or more thresholds based on quantization parameters; selecting a partition combination from a set of one or more predetermined partition combinations based on the one or more thresholds and the sharpness value, the partition combination including one or more predetermined candidate partitions; and encoding the coding tree unit according to the candidate partition selected from the partition combination.

[0173] Example 2 provides the method of Example 1, wherein determining the sharpness value of the coding tree unit includes calculating one or more sharpness values ​​for one or more blocks of the coding tree unit; and selecting the maximum sharpness value from the one or more sharpness values ​​as the sharpness value of the coding tree unit.

[0174] Example 3 provides the method of Example 1 or 2, further comprising determining the time change value of the coding tree unit; wherein selecting the partition combination from the group includes further selecting the partition combination based on the time change value.

[0175] Example 4 provides the method described in Example 3, wherein determining the time change value includes calculating one or more motion prediction generation values ​​for one or more blocks of the coding tree unit; and selecting the maximum motion prediction generation value from the one or more motion prediction generation values ​​as the time change value of the coding tree unit.

[0176] Example 5 provides a method according to any one of Examples 1-4, wherein determining the one or more thresholds includes determining an initial threshold based on the quantization parameter; and determining the one or more thresholds based on the initial threshold.

[0177] Example 6 provides the method described in Example 5, wherein determining the one or more thresholds further includes determining an adjusted threshold based on the sharpness value; wherein determining the one or more thresholds includes determining the one or more thresholds based on the adjusted threshold.

[0178] Example 7 provides a method according to any one of Examples 1-6, wherein selecting the partition combination from the group includes comparing the sharpness value with one or more thresholds; and selecting the partition combination based on the comparison.

[0179] Example 8 provides a method according to any one of Examples 3-7, wherein selecting the partition combination from the group includes comparing the time change value with one or more thresholds; and selecting the partition combination based on the comparison.

[0180] Example 9 provides a method as described in any one of Examples 1-8, wherein the one or more thresholds include one or more of the following: one or more thresholds associated with inter-frame prediction partitioning, and one or more thresholds associated with intra-frame prediction partitioning.

[0181] Example 10 provides a method as described in any one of Examples 1-9, further comprising performing rate-distortion optimization of the one or more predetermined candidate partitions of the partition combination; and selecting, based on the rate-distortion optimization, a partition for encoding the coding tree unit from the one or more predetermined candidate partitions of the partition combination.

[0182] Example 11 provides one or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to: determine a sharpness value of a coding tree unit in a frame of a video to be encoded; determine one or more thresholds based on quantization parameters; select a partition combination from one or more predetermined partition combinations based on the one or more thresholds and the sharpness value, the partition combination including one or more predetermined candidate partitions; and encode the coding tree unit according to the candidate partition selected from the partition combination.

[0183] Example 12 provides one or more non-transitory computer-readable media as described in Example 11, wherein determining the sharpness value of the coding tree unit includes calculating one or more sharpness values ​​for one or more blocks of the coding tree unit; and selecting the maximum sharpness value from the one or more sharpness values ​​as the sharpness value of the coding tree unit.

[0184] Example 13 provides one or more non-transitory computer-readable media as described in Example 11 or 12, wherein the instructions further cause the one or more processors to: determine a time-varying value of the coding tree unit; wherein selecting the partition combination from the group includes further selecting the partition combination based on the time-varying value.

[0185] Example 14 provides one or more non-transitory computer-readable media as described in Example 13, wherein determining the time change value includes calculating one or more motion prediction generation values ​​for one or more blocks of the coding tree unit; and selecting the maximum motion prediction generation value from the one or more motion prediction generation values ​​as the time change value of the coding tree unit.

[0186] Example 15 provides one or more non-transitory computer-readable media as described in any one of Examples 11-14, wherein determining the one or more thresholds includes determining an initial threshold based on the quantization parameter; and determining the one or more thresholds based on the initial threshold.

[0187] Example 16 provides one or more non-transitory computer-readable media as described in Example 15, wherein determining the one or more thresholds further includes determining an adjusted threshold based on the sharpness value; wherein determining the one or more thresholds includes determining the one or more thresholds based on the adjusted threshold.

[0188] Example 17 provides one or more non-transitory computer-readable media as described in any one of Examples 11-16, wherein selecting the partition combination from the group includes comparing the sharpness value with the one or more thresholds; and selecting the partition combination based on the comparison.

[0189] Example 18 provides one or more non-transitory computer-readable media as described in any one of Examples 13-17, wherein selecting the partition combination from the group includes comparing the time-varying value with the one or more thresholds; and selecting the partition combination based on the comparison.

[0190] Example 19 provides one or more non-transitory computer-readable media as described in any one of Examples 11-18, wherein the one or more thresholds include one or more of: one or more thresholds associated with inter-frame prediction partitioning, and one or more thresholds associated with intra-frame prediction partitioning.

[0191] Example 20 provides one or more non-transitory computer-readable media as described in any one of Examples 11-19, wherein the instructions further cause the one or more processors to: perform rate-distortion optimization of the one or more predetermined candidate partitions of the partition combination; and, based on the rate-distortion optimization, select a partition from the one or more predetermined candidate partitions of the partition combination for encoding the coding tree unit.

[0192] Example 21 provides an apparatus including one or more processors; and one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to: determine a sharpness value of a coding tree unit in a frame of a video to be encoded; determine one or more thresholds based on quantization parameters; select a partition combination from one or more predetermined partition combinations based on the one or more thresholds and the sharpness value, the partition combination including one or more predetermined candidate partitions; and encode the coding tree unit according to the candidate partition selected from the partition combination.

[0193] Example 22 provides the apparatus of Example 21, wherein determining the sharpness value of the coding tree unit includes calculating one or more sharpness values ​​for one or more blocks of the coding tree unit; and selecting the maximum sharpness value from the one or more sharpness values ​​as the sharpness value of the coding tree unit.

[0194] Example 23 provides the apparatus described in Example 21 or 22, wherein the instructions further cause the one or more processors to: determine a time-varying value of the coding tree unit; wherein selecting the partition combination from the group includes further selecting the partition combination based on the time-varying value.

[0195] Example 24 provides the apparatus described in Example 23, wherein determining the time change value includes calculating one or more motion prediction generation values ​​for one or more blocks of the coding tree unit; and selecting the maximum motion prediction generation value from the one or more motion prediction generation values ​​as the time change value of the coding tree unit.

[0196] Example 25 provides an apparatus according to any one of Examples 21-24, wherein determining the one or more thresholds includes determining an initial threshold based on the quantization parameter; and determining the one or more thresholds based on the initial threshold.

[0197] Example 26 provides the apparatus of Example 25, wherein determining the one or more thresholds further includes determining an adjusted threshold based on the sharpness value; wherein determining the one or more thresholds includes determining the one or more thresholds based on the adjusted threshold.

[0198] Example 27 provides an apparatus as described in any one of Examples 21-26, wherein selecting the partition combination from the group includes comparing the sharpness value with one or more thresholds; and selecting the partition combination based on the comparison.

[0199] Example 28 provides an apparatus as described in any one of Examples 23-27, wherein selecting the partition combination from the group includes comparing the time change value with one or more thresholds; and selecting the partition combination based on the comparison.

[0200] Example 29 provides an apparatus as described in any one of Examples 21-28, wherein the one or more thresholds include one or more of: one or more thresholds associated with inter-frame prediction partitioning, and one or more thresholds associated with intra-frame prediction partitioning.

[0201] Example 30 provides an apparatus as described in any one of Examples 21-29, wherein the instructions further cause the one or more processors to: perform rate-distortion optimization of the one or more predetermined candidate partitions of the partition combination; and, based on the rate-distortion optimization, select a partition from the one or more predetermined candidate partitions of the partition combination for encoding the coding tree unit.

[0202] Example A provides a computer program product including instructions that, when executed by a processor, cause the processor to perform the method described in any one of Examples 1-10.

[0203] Example B provides an apparatus including means for performing the method described in any one of Examples 1-10.

[0204] Example C provides a partitioning and combination decision as described and illustrated in this document.

[0205] Example E provides the encoder and partitioning combination decision as described and illustrated in this document.

[0206] Example F provides an apparatus including a computing circuit for performing the method of any one of Examples 1-10.

[0207] Deformation and other considerations

[0208] although Figure 1-3 The operations of the example methods shown in Figures 6-11 and described with reference to these figures are illustrated as occurring once for each operation and in a specific order; however, it will be recognized that some operations can be performed in any suitable order and repeated as needed. Furthermore, one or more operations can be performed in parallel. Figure 1-3 The operations shown in 6-11 can be combined, or may include more or less detail than described.

[0209] The foregoing description of the implementations shown in this disclosure, including those described in the abstract, is not intended to be exhaustive or to limit this disclosure to its precise form. While specific implementations and examples of this disclosure have been described herein for illustrative purposes, various equivalent modifications are possible within the scope of this disclosure, as will be recognized by those skilled in the art. These modifications can be made to this disclosure based on the detailed description above.

[0210] For illustrative purposes, specific numerical values, materials, and configurations have been described to provide a thorough understanding of exemplary implementations. However, it will be apparent to those skilled in the art that this disclosure may be practiced without specific details, and / or may be practiced using only some of the described aspects. In other instances, well-known features have been omitted or simplified so as not to obscure the exemplary implementations.

[0211] Furthermore, reference is made to the accompanying drawings, which form part of this document, in which illustrative embodiments that can be practiced are shown. It is to be understood that other embodiments may be utilized, and structural or logical changes may be made without departing from the scope of this disclosure. Therefore, the following detailed description should not be construed as restrictive.

[0212] In a manner most conducive to understanding the disclosed subject matter, various operations can be described sequentially as a plurality of discrete actions or operations. However, the order of description should not be construed as implying that these operations are necessarily sequentially dependent. In particular, these operations may not be performed in the order presented. The described operations may be performed in an order different from that described in the embodiments. In additional embodiments, various additional operations may be performed, or the described operations may be omitted.

[0213] For the purposes of this disclosure, the phrase "A or B" or the phrase "A and / or B" means (A), (B), or (A and B). For the purposes of this disclosure, the phrase "A, B, or C" or the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). The term "between," when used with respect to a measurement range, includes the endpoints of the measurement range.

[0214] For the purposes of this disclosure, "A is less than or equal to a first threshold" is equivalent to "A is less than a second threshold," provided that the first and second thresholds are set in a manner such that for any value of A, both statements produce the same logical result. For the purposes of this disclosure, "B is greater than a first threshold" is equivalent to "B is greater than or equal to a second threshold," provided that the first and second thresholds are set in a manner such that for any value of B, both statements produce the same logical result.

[0215] The phrases “in an embodiment” or “inembodiments” used in the description can each refer to one or more of the same or different embodiments. As used with respect to embodiments of this disclosure, the terms “comprising,” “including,” “having,” etc., are synonymous. This disclosure may use perspective-based descriptions such as “above,” “below,” “top,” “bottom,” and “side” to interpret various features of the drawings, but these terms are for ease of discussion only and do not indicate desired or required directions. The drawings are not necessarily drawn to scale. Unless otherwise stated, the use of ordinal adjectives such as “first,” “second,” and “third,” etc., to describe common objects only indicates that different instances of similar objects are referred to, and is not intended to imply that the objects so described must be in a given order in time, space, sequence, or any other way.

[0216] In the following detailed description, various aspects of the exemplary implementations will be described using terminology commonly employed by those skilled in the art, in order to convey the substance of their work to others skilled in the art.

[0217] The terms “substantially,” “close,” “approximately,” “near,” and “about” generally refer to a target value as described herein or known in the art within + / - 20%. Similarly, terms indicating the orientation of various elements, such as “coplanar,” “perpendicular,” “orthogonal,” “parallel,” or any other angle between elements, generally refer to a target value as described herein or known in the art within + / - 5-20%.

[0218] Furthermore, the terms “comprise,” “comprising,” “include,” “including,” “have,” “having,” or any other variations thereof are intended to cover non-exclusive inclusion. For example, a method, process, or apparatus that includes a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such method, process, or apparatus. Additionally, the term “or” refers to an inclusive “or,” not an exclusive “or.”

[0219] The systems, methods, and apparatuses disclosed herein each have several innovative aspects, and none of them alone is solely responsible for all the desired properties disclosed herein. Details of one or more implementations of the subject matter described herein are set forth in the specification and accompanying drawings.

Claims

1. A method, characterized in that, include: Determine the sharpness value of the coding tree unit in the frame of the video to be encoded; One or more thresholds are determined based on quantization parameters; Based on the one or more thresholds and the sharpness value, a partitioning combination is selected from a set of one or more predetermined partitioning combinations, the partitioning combination including one or more predetermined candidate partitions; as well as The coding tree unit is encoded according to the candidate partitions selected from the partition combinations.

2. The method as described in claim 1, characterized in that, The determination of the sharpness value of the coding tree unit includes: Calculate one or more sharpness values ​​for one or more blocks of the coding tree unit; and The maximum sharpness value is selected from the one or more sharpness values ​​as the sharpness value of the coding tree unit.

3. The method as described in claim 1 or 2, characterized in that, Also includes: Determine the time variation value of the coding tree unit; Selecting the partition combination from the group includes further selecting the partition combination based on the time change value.

4. The method as described in claim 3, characterized in that, Determining the time change value includes: Calculate one or more motion prediction values ​​for one or more blocks of the coding tree unit; and The maximum motion prediction value is selected from the one or more motion prediction values ​​as the time change value of the coding tree unit.

5. The method as described in claim 1 or 2, characterized in that, Determining the one or more thresholds includes: The initial threshold is determined based on the quantization parameters; and The one or more thresholds are determined based on the initial threshold.

6. The method as described in claim 5, characterized in that, Determining the one or more thresholds further includes: The adjusted threshold is determined based on the sharpness value; Determining the one or more thresholds includes determining the one or more thresholds based on the adjusted thresholds.

7. The method as described in claim 1 or 2, characterized in that, Selecting the partition combination from the group includes: Compare the sharpness value with one or more thresholds; and The partitioning combination is selected based on the comparison.

8. The method as described in claim 3, characterized in that, Selecting the partition combination from the group includes: Compare the time change value with one or more thresholds; and The partitioning combination is selected based on the comparison.

9. The method as described in claim 1 or 2, characterized in that, The thresholds mentioned therein include one or more of the following: one or more thresholds associated with inter-frame prediction partitioning, and one or more thresholds associated with intra-frame prediction partitioning.

10. The method as described in claim 1 or 2, characterized in that, Also includes: Perform rate-distortion optimization on the one or more predetermined candidate partitions of the partition combination; as well as Based on the rate-distortion optimization, a partition for encoding the coding tree unit is selected from the one or more predetermined candidate partitions of the partition combination.

11. One or more non-transitory computer-readable media, characterized in that, Its stored instructions, when executed by one or more processors, cause the one or more processors to: Determine the sharpness value of the coding tree unit in the frame of the video to be encoded; One or more thresholds are determined based on quantization parameters; Based on the one or more thresholds and the sharpness value, a partitioning combination is selected from a set of one or more predetermined partitioning combinations, the partitioning combination including one or more predetermined candidate partitions; as well as The coding tree unit is encoded according to the candidate partitions selected from the partition combinations.

12. The one or more non-transitory computer-readable media as claimed in claim 11, characterized in that, The determination of the sharpness value of the coding tree unit includes: Calculate one or more sharpness values ​​for one or more blocks of the coding tree unit; and The maximum sharpness value is selected from the one or more sharpness values ​​as the sharpness value of the coding tree unit.

13. One or more non-transitory computer-readable media as claimed in claim 11 or 12, characterized in that, The instructions further cause the one or more processors to: Determine the time variation value of the coding tree unit; Selecting the partition combination from the group includes further selecting the partition combination based on the time change value.

14. The one or more non-transitory computer-readable media as claimed in claim 13, characterized in that, Determining the time change value includes: Calculate one or more motion prediction values ​​for one or more blocks of the coding tree unit; and The maximum motion prediction value is selected from the one or more motion prediction values ​​as the time change value of the coding tree unit.

15. One or more non-transitory computer-readable media as claimed in claim 11 or 12, characterized in that, Determining the one or more thresholds includes: The initial threshold is determined based on the quantization parameters; and The one or more thresholds are determined based on the initial threshold.

16. One or more non-transitory computer-readable media as claimed in claim 15, characterized in that, Determining the one or more thresholds further includes: The adjusted threshold is determined based on the sharpness value; Determining the one or more thresholds includes determining the one or more thresholds based on the adjusted thresholds.

17. One or more non-transitory computer-readable media as claimed in claim 11 or 12, characterized in that, Selecting the partition combination from the group includes: Compare the sharpness value with one or more thresholds; and The partitioning combination is selected based on the comparison.

18. One or more non-transitory computer-readable media as claimed in claim 13, characterized in that, Selecting the partition combination from the group includes: Compare the time change value with one or more thresholds; and The partitioning combination is selected based on the comparison.

19. One or more non-transitory computer-readable media as claimed in claim 11 or 12, characterized in that, The thresholds mentioned therein include one or more of the following: one or more thresholds associated with inter-frame prediction partitioning, and one or more thresholds associated with intra-frame prediction partitioning.

20. One or more non-transitory computer-readable media as claimed in claim 11 or 12, characterized in that, The instructions further cause the one or more processors to: Perform rate-distortion optimization of the one or more predetermined candidate partitions of the partition combination; and Based on the rate-distortion optimization, a partition for encoding the coding tree unit is selected from the one or more predetermined candidate partitions of the partition combination.

21. An apparatus, characterized in that, include: One or more processors; as well as One or more non-transitory computer-readable media storing instructions, which, when executed by one or more processors, cause the one or more processors to: Determine the sharpness value of the coding tree unit in the frame of the video to be encoded; One or more thresholds are determined based on quantization parameters; Based on the one or more thresholds and the sharpness value, a partitioning combination is selected from a set of one or more predetermined partitioning combinations, the partitioning combination including one or more predetermined candidate partitions; as well as The coding tree unit is encoded according to the candidate partitions selected from the partition combinations.

22. The apparatus as claimed in claim 21, characterized in that, The determination of the sharpness value of the coding tree unit includes: Calculate one or more sharpness values ​​for one or more blocks of the coding tree unit; and The maximum sharpness value is selected from the one or more sharpness values ​​as the sharpness value of the coding tree unit.

23. The apparatus as claimed in claim 21 or 22, characterized in that, The instructions further cause the one or more processors to: Determine the time variation value of the coding tree unit; Selecting the partition combination from the group includes further selecting the partition combination based on the time change value.

24. The apparatus as claimed in claim 23, characterized in that, Determining the time change value includes: Calculate one or more motion prediction values ​​for one or more blocks of the coding tree unit; and The maximum motion prediction value is selected from the one or more motion prediction values ​​as the time change value of the coding tree unit.

25. The apparatus as claimed in claim 21 or 22, characterized in that, Determining the one or more thresholds includes: The initial threshold is determined based on the quantization parameters; and The one or more thresholds are determined based on the initial threshold.

Images