Method, computer equipment, device and storage medium for reordering reference frames

Abstract

The present disclosure relates to adaptively reordering reference frames for video coding techniques. Template matching (TM) can be used to reorder reference frames or reference frame pairs for each block by comparing the difference between the template of the current block and the template of the reference block of motion information about spatial reference motion information (or spatial motion vector) and / or temporal reference motion information (or temporal motion vector). The difference between the template of the current block and the template of the reference block can be calculated for each spatial reference motion information and / or temporal reference motion information, and the difference is marked as a score value of the associated reference frame or reference frame pair. The available reference frames or reference frame pairs are sorted based on the score value.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on and claims the benefit of priority to U.S. non-provisional application No. 17 / 932,333, filed on September 15, 2022, entitled “METHOD AND APPARATUS FOR ADAPTIVE REORDERING FOR REFERENCE FRAMES,” which claims the benefit of priority to U.S. Provisional Application No. 63 / 247,088, filed on September 22, 2021, entitled “METHOD AND APPARATUSFOR ADAPTIVE REORDERING FOR REFERENCE FRAMES,” each of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure describes a set of advanced video coding techniques, and more particularly relates to a method, a computer device, an apparatus, and a non-transitory computer-readable storage medium for reordering reference frames for each block by Template Matching (TM). Background Art

[0004] This background description is provided herein for the purpose of generally presenting the context of the present disclosure. To the extent that the work is described in this background section, the work of the presently named inventors and aspects that may not otherwise be qualified as prior art at the time of filing this application are neither explicitly nor implicitly admitted to be prior art against the present disclosure.

[0005] Video encoding and decoding can be performed using inter-picture prediction with motion compensation. Uncompressed digital video may include a series of pictures, each of which has a spatial dimension of, for example, 1920×1080 luminance samples and all associated or subsampled chrominance samples. The series of pictures may have a fixed or variable picture rate (alternatively referred to as a frame rate), such as 60 pictures per second or 60 frames per second. Uncompressed video has specific bit rate requirements for streaming or data processing. For example, a video with a pixel resolution of 1920×1080, a frame rate of 60 frames per second, and a chrominance subsampling of 4:2:0 at 8 bits per pixel per color channel requires a bandwidth of nearly 1.5 Gbit / s. One hour of such video requires more than 600 gigabytes of storage space.

[0006] One purpose of video encoding and decoding can be to reduce redundancy in an uncompressed input video signal by compression. Compression can help reduce the bandwidth and / or storage space requirements mentioned above, in some cases by two orders of magnitude or more. Both lossless compression and lossy compression and their combinations can be used. Lossless compression refers to a technique that can reconstruct an exact copy of the original signal from the compressed original signal via a decoding process. Lossy compression refers to an encoding / decoding process in which the original video information is not fully retained during encoding and is not fully restored during decoding. When lossy compression is used, the reconstructed signal may be different from the original signal, but the distortion between the original signal and the reconstructed signal is made small enough so that the reconstructed signal is useful for the intended application even if there is some information loss. In the case of video, lossy compression is widely used in many applications. The amount of tolerable distortion depends on the application. For example, users of certain consumer video streaming applications may tolerate higher distortion than users of movie or television broadcast applications. The compression ratio that can be achieved by a specific encoding algorithm can be selected or adjusted to reflect various distortion tolerances: higher tolerable distortion generally allows encoding algorithms that produce higher losses and higher compression ratios.

[0007] Video encoders and decoders may utilize techniques from several broad categories and steps including, for example, motion compensation, Fourier transforms, quantization, and entropy coding.

[0008] Video codec techniques may include techniques known as intra-coding. In intra-coding, sample values ​​are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, pictures are spatially subdivided into sample blocks. When all sample blocks are encoded in intra-mode, the picture may be referred to as an intra-picture. Intra-pictures and their derivatives such as independent decoder refresh pictures may be used to reset decoder states, and may therefore be used as the first picture in an encoded video bitstream and video session or as a still image. The samples of the intra-predicted blocks may then be transformed into the frequency domain, and the transform coefficients thus generated may be quantized before entropy coding. Intra-prediction represents a technique for minimizing sample values ​​in the pre-transform domain. In some cases, the smaller the DC value after the transform and the smaller the AC coefficient, the fewer bits are required to represent the block after entropy coding at a given quantization step size.

[0009] With the current design of the signaling method for reference frames, the order of reference frames in reference frame list 0 and reference frame list 1 is fixed for all blocks within a frame, regardless of the reference frames in neighboring blocks. Therefore, the spatial and / or temporal correlation of the reference frame selection is not exploited. Summary of the invention

[0010] According to an embodiment, a method for reordering reference frames for each block by template matching (TM) is provided, characterized in that the method includes: comparing a template of a current block with a template of a reference block for motion information, wherein the motion information includes spatial reference motion information or temporal reference motion information; calculating a difference between the template of the current block and the template of the reference block; determining a score value of an associated reference frame based on the calculated difference; and reordering the reference frames based on the determined score.

[0011] According to an embodiment, a computer device is provided, characterized in that the computer device includes: one or more computer-readable non-transitory storage media configured to store computer program codes; and one or more computer processors configured to access the computer program codes and execute the above method for reordering reference frames for each block by template matching (TM) as instructed by the computer program codes.

[0012] According to an embodiment, a device for decoding a video bitstream is provided, characterized in that the device comprises: a memory storing instructions; and a processor communicating with the memory, wherein when the processor executes the instructions, the processor is configured to cause the device to perform the above method for reordering reference frames for each block by template matching (TM).

[0013] According to an embodiment, a non-transitory computer-readable storage medium is provided, which stores instructions, characterized in that when the instructions are executed by a processor, the instructions are configured to cause the processor to perform the above method for reordering reference frames for each block through template matching (TM).

[0014] The present invention provides a method, a computer device, an apparatus, and a non-transitory computer-readable storage medium for reordering reference frames for each block by template matching (TM), providing a technique for adaptively reordering reference frames, wherein spatial and / or temporal correlation of reference frame selection is utilized to improve coding efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Additional features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which:

[0016] Figure 1A A schematic diagram of an exemplary subset of intra prediction direction modes is shown.

[0017] Figure 1B A diagram of exemplary intra prediction directions is shown.

[0018] Figure 2 A schematic diagram of a current block and its surrounding spatial merging candidates for motion vector prediction in one example is shown.

[0019] Figure 3 A schematic illustration showing a simplified block diagram of a communication system according to an example embodiment is shown.

[0020] Figure 4 A schematic illustration of a simplified block diagram of a communication system according to another example embodiment is shown.

[0021] Figure 5 A schematic illustration of a simplified block diagram of a video decoder according to an example embodiment is shown.

[0022] Figure 6 A schematic illustration of a simplified block diagram of a video encoder according to an example embodiment is shown.

[0023] Figure 7 A block diagram of a video encoder according to another example embodiment is shown.

[0024] Figure 8 A block diagram of a video decoder according to another example embodiment is shown.

[0025] Fig. 9 A scheme for coding block segmentation according to an example implementation of the present disclosure is shown;

[0026] Fig.10 Another scheme of coding block segmentation according to an example implementation of the present disclosure is shown;

[0027] Fig.11 Another scheme of coding block segmentation according to an example implementation of the present disclosure is shown;

[0028] Fig.12 An example of partitioning a basic block into coding blocks according to an example partitioning scheme is shown;

[0029] Fig.13 An example three-way segmentation scheme is shown;

[0030] Fig.14 An example quadtree binary tree coding block partitioning scheme is shown;

[0031] Fig.15 A scheme for splitting a coding block into a plurality of transform blocks and a coding order of the transform blocks according to an example implementation of the present disclosure is shown;

[0032] Fig.16Another scheme for splitting a coding block into multiple transform blocks and the encoding order of the transform blocks according to an example implementation of the present disclosure is shown;

[0033] Fig.17 Another scheme for splitting a coding block into a plurality of transform blocks according to an example implementation of the present disclosure is shown;

[0034] Fig.18 An example segmentation tree for block segmentation is shown;

[0035] Fig.19 An example partitioning and tree for a Quad-Tree plus Binary Tree (QTBT) structure is shown;

[0036] Fig. 20 An example ternary tree partition is shown;

[0037] Fig.21 An example template matching (TM) on a search area around an initial motion vector (MV) is shown;

[0038] Fig. 22 An example of decoder-side motion vector refinement is shown;

[0039] Fig.23 An example spatial motion vector search pattern is shown.

[0040] Fig.24 A flowchart illustrating a method according to an example implementation of the present disclosure; and

[0041] Fig.25 A schematic diagram of a computer system according to an example implementation of the present disclosure is shown. DETAILED DESCRIPTION

[0042] Throughout the specification and claims, terms may have nuanced meanings that are suggested or implied in the context beyond the explicitly stated meaning. The phrases "in one embodiment" or "in some embodiments" as used herein do not necessarily refer to the same embodiment, and the phrases "in another embodiment" or "in other embodiments" as used herein do not necessarily refer to different embodiments. Similarly, the phrases "in one implementation" or "in some implementations" as used herein do not necessarily refer to the same implementation, and the phrases "in another implementation" or "in other implementations" as used herein do not necessarily refer to different implementations. For example, it is intended that the claimed subject matter includes combinations of all or part of the exemplary embodiments / implementations.

[0043] Typically, terms can be understood at least in part based on usage in context. For example, as used herein, terms such as "and", "or" or "and / or" can include a variety of meanings depending at least in part on the context in which such terms are used. Typically, if "or" is used for an association list such as A, B or C, "or" is intended to represent A, B and C, which are used here in an inclusive sense, and A, B or C, which are used here in an exclusive sense. In addition, at least in part depending on the context, as used herein, the terms "one or more" or "at least one" can be used to describe any feature, structure or characteristic in a singular sense, or can be used to describe a combination of features, structures or characteristics in a plural sense. Similarly, terms such as "one", "an" or "the" can also be understood to convey singular usage or to convey plural usage, which depends at least in part on the context. In addition, the term "based on" or "determined by..." can be understood to not necessarily be intended to convey an exclusive set of factors, and can alternatively allow the presence of additional factors that are not necessarily explicitly described, which also depends at least in part on the context.

[0044] Conventional intra-frame coding, such as that known from, for example, MPEG (Moving Picture Experts Group, MPEG)-2 generation coding techniques, does not use intra-frame prediction. However, some newer video compression techniques include techniques that attempt to encode / decode blocks based on surrounding sample data and / or metadata obtained during encoding and / or decoding of data blocks that are spatially adjacent and precede the data block being intra-coded or decoded in decoding order. Such techniques are hereinafter referred to as "intra-frame prediction" techniques. Note that, in at least some cases, intra-frame prediction uses reference data only from the current picture being reconstructed, and does not use reference data from other reference pictures.

[0045] There can be many different forms of intra-frame prediction. When more than one such technique can be used in a given video coding technique, the techniques used can be referred to as intra-frame prediction modes. One or more intra-frame prediction modes can be provided in a particular codec. In some cases, a mode can have sub-modes and / or can be associated with various parameters, and the mode / sub-mode information and intra-frame coding parameters for a video block can be encoded separately or included together in a mode codeword. Which codeword is used for a given mode, sub-mode and / or parameter combination may have an impact on the coding efficiency gain through intra-frame prediction, and therefore the entropy coding technique used to convert the codeword into a bitstream.

[0046] Certain modes of intra prediction were introduced with H.264, refined in H.265, and further refined in newer coding techniques such as the Joint Exploration Model (JEM), Versatile Video Coding (VVC), and Benchmark set (BMS). Typically, for intra prediction, the predictor block can be formed using neighboring sample values ​​that have become available. For example, the available values ​​of a particular set of neighboring samples along a particular direction and / or line can be copied into the predictor block. A reference to the direction in use can be encoded in the bitstream, or the reference to the direction in use can be predicted by itself.

[0047] Reference Figure 1A , depicted at the bottom right is a subset of nine predictor directions specified in the 33 possible intra-frame predictor directions of H.265 (corresponding to the 33 angular modes of the 35 intra-frame modes specified in H.265). The point (101) where the arrows converge represents the sample being predicted. The arrows represent the direction in which the sample at 101 is predicted using neighboring samples. For example, arrow (102) indicates that sample (101) is predicted based on one or more neighboring samples at a 45 degree angle to the upper right and horizontal direction. Similarly, arrow (103) indicates that sample (101) is predicted based on one or more neighboring samples at a 22.5 degree angle to the horizontal direction to the lower left of sample (101).

[0048] Still refer to Figure 1A , depicted at the top left is a square block (104) of 4×4 samples (indicated by bold dashed lines). The square block (104) includes 16 samples, each of which is labeled with "S", its position in the Y dimension (e.g., row index), and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (from the top) and the first sample in the X dimension (from the left). Similarly, sample S44 is the fourth sample in block (104) in both the Y and X dimensions. Since the size of the block is 4×4 samples, S44 is at the bottom right. Also shown are example reference samples, which follow a similar numbering scheme. The reference samples are labeled with R, their Y position (e.g., row index) and X position (column index) relative to the block (104). In both H.264 and H.265, prediction samples that are adjacent to the block being reconstructed are used.

[0049] The intra picture prediction of block 104 may start by copying reference sample values ​​from neighboring samples according to a signaled prediction direction. For example, assume that the encoded video bitstream includes signaling indicating, for this block 104, the prediction direction of arrow (102) - i.e., the samples are predicted according to one or more prediction samples to the upper right at a 45 degree angle to the horizontal direction. In such a case, samples S41, S32, S23, and S14 are predicted according to the same reference sample R05. Then, sample S44 is predicted according to reference sample R08.

[0050] In some cases, the values ​​of multiple reference samples may be combined, such as by interpolation, in order to compute the reference sample; especially if the directions do not divide evenly by 45 degrees.

[0051] As video coding technology continues to develop, the number of possible directions is also increasing. For example, in H.264 (2003), there are nine different directions that can be used for intra-frame prediction. In H.265 (2013), this increased to 33, and at the time of this disclosure, JEM / VVC / BMS can support up to 65 directions. Experimental studies have been conducted to help identify the most appropriate intra-frame prediction directions, and certain techniques in entropy coding can be used to encode those most appropriate directions with a small number of bits, thereby accepting a specific bit penalty for the direction. In addition, the direction itself can sometimes be predicted based on the neighboring directions used in the intra-frame prediction of decoded neighboring blocks.

[0052] Figure 1B A schematic diagram (180) is shown depicting 65 intra prediction directions according to JEM to illustrate that the number of prediction directions increases in various encoding techniques evolving over time.

[0053] The manner in which bits representing intra-prediction directions are mapped to prediction directions in the coded video bitstream may vary from one video coding technique to another; and may range, for example, from simple direct mappings of prediction directions to intra-prediction modes, to codewords, to complex adaptive schemes involving most probable modes, and the like. In all cases, however, there may be some directions used for leading prediction that are statistically less likely to appear in the video content than some other directions. Since the goal of video compression is to reduce redundancy, in a well-designed video coding technique, those less likely directions may be represented by a larger number of bits than more likely directions.

[0054] Inter-picture prediction or inter-prediction may be based on motion compensation. In motion compensation, sample data from a previously reconstructed picture or a portion thereof (reference picture) may be used to predict a newly reconstructed picture or picture portion (e.g., block) after being spatially shifted in the direction indicated by a motion vector (hereinafter referred to as MV). In some cases, the reference picture may be the same as the picture in the current reconstruction. The MV may have two dimensions, X and Y, or three dimensions, where the third dimension is an indication of the reference picture used (similar to the time dimension).

[0055] In some video compression techniques, the current MV applicable to a particular region of sample data can be predicted based on other MVs, for example, based on other MVs related to other regions of sample data that are spatially adjacent to the region being reconstructed and that precede the current MV in the decoding order. This can significantly reduce the total amount of data required to encode the MV by eliminating redundancy in the related MVs, thereby improving compression efficiency. MV prediction can work effectively, for example, because when encoding an input video signal obtained from a camera (called natural video), there is a statistical possibility that a larger area than the area applicable to a single MV moves in a similar direction in the video sequence, and therefore in some cases similar motion vectors derived from MVs of adjacent regions can be used to predict the larger area. This causes the actual MV for a given region to be similar or identical to the MV predicted from the surrounding MVs. After entropy coding, such an MV can then be represented with fewer bits than would be used if the MV was encoded directly rather than predicted from adjacent MVs. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., MV) derived from the original signal (i.e., sample stream). In other cases, the MV prediction itself may be lossy, for example due to round-off errors when computing the predictor from several surrounding MVs.

[0056] Various MV prediction mechanisms are described in H.265 / HEVC (High Efficiency Video Coding, HEVC) (ITU-T (International Telecommunication Union-Telecommunication Standardization Sector, ITU-T) H.265 Recommendation, "High Efficiency Video Coding", December 2016). Among the multiple MV prediction mechanisms specified by H.265, the following is a technique referred to as "spatial merging" hereinafter.

[0057] Specifically, refer to Figure 2, the current block (201) includes samples discovered by the encoder during the motion search process that can be predicted from a previous block of the same size that has been spatially shifted. Instead of encoding the MV directly, the MV associated with any of the five surrounding samples represented by A0, A1 and B0, B1, B2 (corresponding to 202 to 206, respectively) can be derived from metadata associated with one or more reference pictures, for example, the MV is derived from the most recent (in decoding order) reference picture. In H.265, MV prediction can use a predictor from the same reference picture used by a neighboring block.

[0058] AOMedia Video 1 (AV1) is an open video coding format designed for video transmission over the Internet. AOMedia Video 1 was developed as a successor to VP9 by building on the VP9 code base and combining additional technologies. The AV1 bitstream specification includes reference video codecs such as H.265 or High Efficiency Video Coding (HEVC) standards or Versatile Video Coding (VVC).

[0059] Figure 3 A simplified block diagram of a communication system (300) according to an embodiment of the present disclosure is shown. The communication system (300) includes a plurality of terminal devices that can communicate with each other via, for example, a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). Figure 3 In the example, a first pair of terminal devices (310) and (320) can perform unidirectional transmission of data. For example, the terminal device (310) can encode video data (e.g., video data of a video picture stream captured by the terminal device (310)) for transmission to another terminal device (320) via a network (350). The encoded video data can be transmitted in the form of one or more encoded video bitstreams. The terminal device (320) can receive the encoded video data from the network (350), decode the encoded video data to recover the video picture, and display the video picture based on the recovered video data. Unidirectional data transmission may be implemented in a media service application or the like.

[0060] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) that perform bidirectional transmission of encoded video data, which bidirectional transmission may be implemented, for example, during a video conferencing application. For the bidirectional transmission of data, in the example, each of the terminal devices (330) and (340) can encode video data (e.g., video data of a video picture stream captured by the terminal device) for transmission to the other of the terminal devices (330) and (340) via the network (350). Each of the terminal devices (330) and (340) can also receive the encoded video data transmitted by the other of the terminal devices (330) and (340), and can decode the encoded video data to restore the video picture, and can display the video picture at an accessible display device based on the restored video data.

[0061] exist Figure 3 In the example, terminal devices (310), (320), (330) and (340) can be implemented as servers, personal computers and smart phones, but the applicability of the basic principles of the present disclosure may not be so limited. Implementations of the present disclosure may be implemented in desktop computers, laptop computers, tablet computers, media players, wearable computers, dedicated video conferencing equipment, etc. Network (350) represents any number or type of network that transmits encoded video data between terminal devices (310), (320), (330) and (340), including, for example, wired (wired) and / or wireless communication networks. The communication network (350) can exchange data in circuit switching channels, packet switching channels and / or other types of channels. Representative networks include telecommunication networks, local area networks, wide area networks and / or the Internet. For the purpose of this discussion, unless explicitly stated herein, the architecture and topology of the network (350) may be irrelevant to the operation of the present disclosure.

[0062] As examples of applications for the disclosed subject matter, Figure 4 The placement of the video encoder and the video decoder in a video streaming environment is shown. The disclosed subject matter can be equally applicable to other video applications, including, for example, video conferencing, digital TV, broadcasting, gaming, virtual reality, storing compressed video on digital media including CD (Compact Disc, CD), DVD (Digital Versatile Disc, DVD), memory sticks, etc.

[0063] The video streaming system may include a video capture subsystem (413) that may include a video source (401), such as a digital camera, for creating an uncompressed video picture or image stream (402). In an example, the video picture stream (402) includes samples recorded by the digital camera of the video source 401. The video picture stream (402) is depicted as a thick line to emphasize the high amount of data when compared to the encoded video data (404) (or encoded video bitstream), which may be processed by an electronic device (420) coupled to the video source (401) including a video encoder (403). The video encoder (403) may include hardware, software, or a combination thereof to implement or embody aspects of the disclosed subject matter as described in more detail below. The encoded video data (404) (or encoded video bitstream (404)) is depicted as a thin line to emphasize the lower data volume when compared to the uncompressed video picture stream (402), which can be stored on the streaming server (405) for future use or directly to a downstream video device (not shown). One or more streaming client subsystems such as Figure 4 The client subsystems (406) and (408) in the streaming server (405) can access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). The client subsystem (406) may include, for example, a video decoder (410) in an electronic device (430). The video decoder (410) decodes the incoming copy (407) of the encoded video data and creates an output video picture stream (411) that is uncompressed and can be presented on a display (412) (e.g., a display screen) or another rendering device (not depicted). The video decoder 410 can be configured to perform some or all of the various functions described in the present disclosure. In some streaming systems, the encoded video data (404), (407) and (409) (e.g., a video bitstream) can be encoded according to certain video encoding / compression standards. Examples of these standards include ITU-T Recommendation H.265. In the example, the video coding standard under development is informally referred to as Versatile Video Coding (VVC). The disclosed subject matter can be used in the context of VVC as well as other video coding standards.

[0064] Note that the electronic devices (420) and (430) may include other components (not shown). For example, the electronic device (420) may include a video decoder (not shown), and the electronic device (430) may also include a video encoder (not shown).

[0065] Figure 5A block diagram of a video decoder (510) according to any of the following embodiments of the present disclosure is shown. The video decoder (510) may be included in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., receiving circuitry). The video decoder (510) may be used to replace Figure 4 A video decoder (410) is shown in an example.

[0066] The receiver (531) can receive one or more encoded video sequences to be decoded by the video decoder (510). In the same embodiment or another embodiment, one encoded video sequence can be decoded at a time, wherein the decoding of each encoded video sequence is independent of other encoded video sequences. Each video sequence can be associated with multiple video frames or images. The encoded video sequence can be received from a channel (501), which can be a hardware / software link to a storage device storing the encoded video data or to a streaming source transmitting the encoded video data. The receiver (531) can receive the encoded video data and other data, such as encoded audio data and / or auxiliary data streams, which can be forwarded to their respective processing circuit systems (not depicted). The receiver (531) can separate the encoded video sequence from the other data. To prevent network jitter, a buffer memory (515) can be set between the receiver (531) and the entropy decoder / parser (520) (hereinafter referred to as "parser (520)"). In some applications, the buffer memory (515) can be implemented as part of the video decoder (510). In other applications, the buffer memory (515) may be external to the video decoder (510) and separate from the video decoder (510) (not depicted). In still other applications, there may be a buffer memory (not depicted) external to the video decoder (510) for purposes such as preventing network jitter, and another additional buffer memory (515) may be internal to the video decoder (510) to, for example, handle playback timing. When the receiver (531) receives data from a store / forward device with sufficient bandwidth and controllability or from an isochronous network, the buffer memory (515) may not be required, or the buffer memory (515) may be smaller. For use on an optimal packet network such as the Internet, a buffer memory (515) of sufficient size may be required, and its size may be relatively large. Such a buffer memory may be implemented with an adaptive size and may be implemented at least in part in an operating system or similar element (not depicted) external to the video decoder (510).

[0067] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the encoded video sequence. The categories of these symbols include information for managing the operation of the video decoder (510) and potential information for controlling a presentation device such as a display (512) (e.g., a display screen), which may or may not be an integral part of the electronic device (530), but may be coupled to the electronic device (530), such as Figure 5 As shown. The control information for the rendering device may be in the form of a supplementary enhancement information (SEI (Supplemental Enhancement Information, SEI) message) or a video usability information (VUI (Video Usability Information, VUI)) parameter set fragment (not depicted). The parser (520) may parse / entropy decode the encoded video sequence received by the parser (520). The entropy encoding of the encoded video sequence may be performed according to a video coding technique or standard, and may follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (520) may extract a subgroup parameter set for at least one of the subgroups of pixels in the video decoder from the encoded video sequence based on at least one parameter corresponding to the subgroup. The subgroup may include a group of pictures (Group of Pictures, GOP), a picture, a tile, a slice, a macroblock, a coding unit (Coding Unit, CU), a block, a transform unit (Transform Unit, TU), a prediction unit (Prediction Unit, PU), etc. The parser (520) may also extract information such as transform coefficients (eg, Fourier transform coefficients), quantizer parameter values, motion vectors, etc. from the encoded video sequence.

[0068] The parser (520) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515) to create symbols (521).

[0069] Depending on the type of coded video picture or part of coded video picture (e.g., inter-picture and intra-picture, inter-block and intra-block) and other factors, the reconstruction of symbol (521) may involve multiple different processing or functional units. The units involved and the way of involvement can be controlled by subgroup control information parsed by parser (520) from the coded video sequence. For simplicity, such subgroup control information flow between parser (520) and the following multiple processing or functional units is not depicted.

[0070] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into a plurality of functional units as described below. In a practical implementation operating under commercial constraints, many of these functional units interact closely with each other and can be at least partially integrated with each other. However, for the purpose of clearly describing the various functions of the disclosed subject matter, the conceptual subdivision into the following functional units is adopted in the following disclosure.

[0071] The first unit may include a sealer / inverse transform unit (551). The sealer / inverse transform unit (551) may receive quantized transform coefficients as symbols (521) and control information from the parser (520), including information indicating which type of inverse transform to use, block size, quantization factors / parameters, quantization scaling matrices, etc. The sealer / inverse transform unit (551) may output a block including sample values, which may be input into an aggregator (555).

[0072] In some cases, the output samples of the scaler / inverse transform (551) may belong to an intra-coded block; that is, a block that does not use predictive information from a previously reconstructed picture, but may use predictive information from a previously reconstructed portion of the current picture. Such predictive information may be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) may use surrounding block information that has already been reconstructed and stored in a current picture buffer (558) to generate a block of the same size and shape as the block being reconstructed. For example, the current picture buffer (558) buffers a partially reconstructed current picture and / or a fully reconstructed current picture. In some implementations, the aggregator (555) may add the prediction information that has been generated by the intra-prediction unit (552) to the output sample information as provided by the scaler / inverse transform unit (551) on a per-sample basis.

[0073] In other cases, the output samples of the scaler / inverse transform unit (551) may belong to an inter-frame coded block and potentially a motion compensated block. In this case, the motion compensated prediction unit (553) may access the reference picture memory (557) to obtain samples for inter-frame picture prediction. After motion compensation of the obtained samples according to the symbol (521) belonging to the block, these samples may be added to the output of the scaler / inverse transform unit (551) by the aggregator (555) (the output of unit 551 may be referred to as residual samples or residual signals) to generate output sample information. The address in the reference picture memory (557) at which the motion compensated prediction unit (553) obtains the predicted samples may be controlled by a motion vector, the MV being available to the motion compensated prediction unit (553) in the form of a symbol (521), the symbol (521) may have, for example, an X component, a Y component (shift) and a reference picture component (time). Motion compensation may also include interpolation of sample values ​​such as those obtained from a reference picture memory (557) when using sub-sample accurate motion vectors, and may also be associated with motion vector prediction mechanisms, etc.

[0074] The output samples of the aggregator (555) may be subjected to various loop filtering techniques in a loop filter unit (556). The video compression techniques may include in-loop filter techniques controlled by parameters included in the coded video sequence (also referred to as the coded video bitstream) and available to the loop filter unit (556) as symbols (521) from the parser (520), but the video compression techniques may also be responsive to meta-information obtained during decoding of a previous (in decoding order) portion of the coded picture or coded video sequence, as well as to previously reconstructed and loop filtered sample values. Several types of loop filters may be included as part of the loop filter unit 556 in various orders, as will be described in further detail below.

[0075] The output of the loop filter unit (556) may be a sample stream that may be output to a rendering device (512) and stored in a reference picture memory (557) for use in future inter-picture prediction.

[0076] Once fully reconstructed, certain coded pictures can be used as reference pictures for future inter-picture prediction. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture has been identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557) and a new current picture buffer can be reallocated before starting to reconstruct a subsequent coded picture.

[0077] The video decoder (510) may perform decoding operations according to a predetermined video compression technique adopted in a standard such as ITU-T H.265 Recommendation. The encoded video sequence may conform to the syntax specified by the video compression technology or standard used in the sense that the encoded video sequence follows both the syntax of the video compression technology or standard and the profile recorded in the video compression technology or standard. Specifically, the profile may select certain tools from all the tools available in the video compression technology or standard as tools that can only be used under the profile. In order to conform to the standard, the complexity of the encoded video sequence may be within a range defined by the hierarchy of the video compression technology or standard. In some cases, the hierarchy limits the maximum picture size, the maximum frame rate, the maximum reconstruction sampling rate (measured in, for example, mega samples per second), the maximum reference picture size, etc. In some cases, the limits set by the hierarchy may be further defined by the Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management signaled in the encoded video sequence.

[0078] In some example embodiments, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence. The video decoder (510) may use the additional data to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0079] Figure 6 A block diagram of a video encoder (603) according to an example implementation of the present disclosure is shown. The video encoder (603) may be included in an electronic device (620). The electronic device (620) may also include a transmitter (640) (e.g., a transmission circuit system). The video encoder (603) may be used to replace Figure 4 The video encoder (403) in the example.

[0080] The video encoder (603) can be obtained from a video source (601) (not Figure 6 In another example, the video source (601) may be implemented as a part of the electronic device (620).

[0081] The video source (601) may provide a source video sequence in the form of a digital video sample stream to be encoded by a video encoder (603), the digital video sample stream may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 YCrCb, RGB, XYZ, ...), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, the video source (601) may be a storage device capable of storing previously prepared videos. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of separate pictures or images that are given motion when viewed sequentially. The picture itself may be organized as a spatial pixel array, wherein each pixel may include one or more samples, depending on the sampling structure, color space, etc. used. The relationship between pixels and samples may be easily understood by a person skilled in the art. The following description focuses on samples.

[0082] According to some example embodiments, the video encoder (603) may encode and compress pictures of a source video sequence into an encoded video sequence (643) in real time or under any other time constraints required by the application. Implementing an appropriate encoding speed constitutes a function of the controller (650). In some embodiments, the controller (650) may be functionally coupled to and control other functional units as described below. For simplicity, the coupling is not depicted. The parameters set by the controller (650) may include rate control related parameters (picture skipping, quantizer, lambda value of rate distortion optimization technique, ...), picture size, picture group (Group of Picture, GOP) layout, maximum motion vector search range, etc. The controller (650) may be configured to have other suitable functions belonging to the video encoder (603) optimized for a specific system design.

[0083] In some example embodiments, the video encoder (603) may be configured to operate in an encoding loop. As a simplified description, in an example, the encoding loop may include a source encoder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on an input picture to be encoded and a reference picture) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder would also create sample data, even if the embedded decoder 633 processes the encoded video stream of the source encoder 630 without entropy coding (because in the video compression techniques contemplated by the disclosed subject matter, any compression between the symbols and the encoded video bitstream in entropy coding can be lossless). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since the decoding of the symbol stream produces a bit-accurate result that is independent of the decoder location (local or remote), the contents of the reference picture memory (634) are also bit-accurate between the local encoder and the remote encoder. In other words, the reference picture samples "seen" by the prediction part of the encoder are exactly the same sample values ​​that the decoder will "see" when using the prediction during decoding. The basic principle of reference picture synchronization (and the drift that occurs when synchronization cannot be maintained, for example due to channel errors) is used to improve encoding quality.

[0084] The operation of the "local" decoder (633) can be combined with the "remote" decoder such as has been described above. Figure 5 The operation of the video decoder (510) described in detail is the same. However, reference is also briefly made to Figure 5 , since the symbols are available and the encoding of the symbols into a coded video sequence by the entropy encoder (645) and the decoding of the symbols by the parser (520) can be lossless, the entropy decoding portion of the video decoder (510) including the buffer memory (515) and the parser (520) may not be fully implemented in the local decoder (633) in the encoder.

[0085] At this point, it can be observed that any decoder technology other than the parsing / entropy decoding that may only exist in the decoder may also necessarily need to exist in the corresponding encoder in substantially the same functional form. For this reason, the disclosed subject matter may sometimes focus on the decoder operation related to the decoding part of the encoder. Therefore, the description of the encoder technology can be simplified because the encoder technology is opposite to the decoder technology described comprehensively. A more detailed description of the encoder is provided only in some areas or aspects below.

[0086] In some example implementations, during operation, the source encoder (630) may perform motion compensated predictive coding that predictively encodes an input picture with reference to one or more previously encoded pictures from a video sequence designated as "reference pictures". In this manner, the encoding engine (632) encodes the differences (or residuals) in color channels between pixel blocks of the input picture and pixel blocks of a reference picture that may be selected as a prediction reference for the input picture. The term "residual" and its adjective form "residual" may be used interchangeably.

[0087] The local video decoder (633) can decode the encoded video data of the picture that can be designated as the reference picture based on the symbols created by the source encoder (630). The operation of the encoding engine (632) can advantageously be a lossy process. When the encoded video data can be decoded at the video decoder ( Figure 6 When the video encoder (603) is decoded at a local video decoder (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (633) replicates the decoding process that may be performed by the video decoder on the reference picture and may cause the reconstructed reference picture to be stored in the reference picture cache (634). In this way, the video encoder (603) may store a copy of the reconstructed reference picture locally that has common content (absent transmission errors) with the reconstructed reference picture to be obtained by the far-end (remote) video decoder.

[0088] The predictor (635) may perform a prediction search for the encoding engine (632). That is, for a new picture to be encoded, the predictor (635) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or specific metadata such as reference picture motion vectors, block shapes, etc. that may be used as appropriate prediction references for the new picture. The predictor (635) may operate on a pixel block by pixel block basis to find an appropriate prediction reference. In some cases, as determined by the search results obtained by the predictor (635), the input picture may have prediction references taken from multiple reference pictures stored in the reference picture memory (634).

[0089] The controller (650) can manage encoding operations of the source encoder (630), including, for example, setting parameters and subgroup parameters for encoding video data.

[0090] The outputs of all the above mentioned functional units may be subjected to entropy coding in an entropy encoder (645). The entropy encoder (645) converts the symbols generated by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0091] The transmitter (640) can buffer the encoded video sequence created by the entropy encoder (645) in preparation for transmission via the communication channel (560), which can be a hardware / software link to a storage device where the encoded video data will be stored. The transmitter (640) can combine the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).

[0092] The controller (650) may manage the operation of the video encoder (603). During encoding, the controller (650) may assign a certain encoding picture type to each encoded picture, which may affect the encoding techniques that may be applied to the corresponding picture. For example, a picture may generally be assigned one of the following picture types:

[0093] An intra picture (I picture) may be a picture that can be encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those of ordinary skill in the art are aware of those variations of I pictures and their corresponding applications and features.

[0094] A predictive picture (P picture) may be a picture that can be encoded and decoded using inter prediction or intra prediction to predict sample values ​​of each block using at most one motion vector and a reference index.

[0095] Bi-directional predictive pictures (B pictures), which can be pictures that can be encoded and decoded using inter-prediction or intra-prediction that uses up to two motion vectors and reference indices to predict sample values ​​for each block. Similarly, multi-predictive pictures can use more than two reference pictures and associated metadata for reconstruction of a single block.

[0096] The source picture may typically be spatially subdivided into a plurality of coding blocks of samples (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples, respectively), and encoded block by block. These blocks may be predictively encoded with reference to other (already encoded) blocks, which are determined by the coding allocation applied to the corresponding picture of the block. For example, a block of an I picture may be non-predictively encoded, or a block of an I picture may be predictively encoded (spatial prediction or intra-frame prediction) with reference to an already encoded block of the same picture. A pixel block of a P picture may be predictively encoded via spatial prediction or via temporal prediction with reference to a previously encoded reference picture. A block of a B picture may be predictively encoded via spatial prediction or via temporal prediction with reference to one or two previously encoded reference pictures. A source picture or an intermediately processed picture may be subdivided into other types of blocks for other purposes. The division of coding blocks and other types of blocks may or may not follow the same approach, as described in further detail below.

[0097] The video encoder (603) may perform encoding operations according to a predetermined video encoding technique or standard, such as ITU-T H.265 Recommendation. In the operation of the video encoder (603), the video encoder (603) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the encoded video data may conform to the syntax specified by the video encoding technique or standard used.

[0098] In some example embodiments, the transmitter (640) may transmit additional data along with the encoded video. The source encoder (630) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0099] Video can be captured as multiple source pictures (video pictures) in a temporal sequence. Intra-picture prediction (often referred to simply as intra-picture prediction) exploits spatial correlations in a given picture, while inter-picture prediction exploits temporal or other correlations between pictures. For example, a particular picture in encoding / decoding, which is called the current picture, can be divided into blocks. The blocks in the current picture can be encoded by a vector called a motion vector if they are similar to reference blocks in a reference picture that has been previously encoded and still buffered in the video. The motion vector points to a reference block in a reference picture, and in the case of using multiple reference pictures, the motion vector can have a third dimension that identifies the reference picture.

[0100] In some example embodiments, bidirectional prediction techniques may be used for inter-picture prediction. According to such bidirectional prediction techniques, two reference pictures are used, such as a first reference picture and a second reference picture that are both subsequent to the current picture in the video in decoding order (but may be in the past or future in display order, respectively). A block in the current picture may be encoded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. The block may be jointly predicted by a combination of the first reference block and the second reference block.

[0101] In addition, merge mode techniques can be used in inter-picture prediction to improve coding efficiency.

[0102] According to some example embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks. For example, a picture in a video picture sequence is divided into coding tree units (Coding Tree Unit, CTU) for compression, and the CTUs in the picture may have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU may include three parallel coding tree blocks (Coding Tree Block, CTB): a luminance CTB and two chrominance CTBs. Each CTU may be recursively split into one or more coding units (Coding Unit, CU) in a quadtree. For example, a 64×64 pixel CTU may be split into a 64×64 pixel CU, or four 32×32 pixel CUs. Each of one or more of the 32×32 blocks may be further split into four 16×16 pixel CUs. In some example embodiments, each CU may be analyzed during encoding to determine a prediction type for the CU among various prediction types, such as an inter prediction type or an intra prediction type. According to temporal and / or spatial predictability, a CU can be split into one or more prediction units (PUs). Typically, each PU includes a luminance prediction block (PB) and two chrominance PBs. In an embodiment, the prediction operation in decoding (encoding / decoding) is performed in units of prediction blocks. Splitting a CU into PUs (or PBs of different color channels) can be performed in various spatial modes. For example, a luminance or chrominance PB may include a matrix of values ​​(e.g., luminance values) for samples such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 samples, etc.

[0103] Figure 7A diagram of a video encoder (703) according to another example implementation of the present disclosure is shown. The video encoder (703) is configured to receive a processed block (e.g., a prediction block) of sample values ​​within a current video picture in a sequence of video pictures, and encode the processed block into an encoded picture that is part of an encoded video sequence. The example video encoder (703) may be used instead of Figure 4 The video encoder (403) in the example.

[0104] For example, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as a prediction block of 8×8 samples. The video encoder (703) then uses, for example, Rate-Distortion Optimization (RDO) to determine whether to use intra mode, inter mode, or bidirectional prediction mode to best encode the processing block. In the case where it is determined that the processing block is to be encoded in intra mode, the video encoder (703) can use intra prediction techniques to encode the processing block into an encoded picture; and in the case where it is determined that the processing block is to be encoded in inter mode or bidirectional prediction mode, the video encoder (703) can use inter prediction or bidirectional prediction techniques to encode the processing block into an encoded picture, respectively. In some example embodiments, merge mode can be used as a sub-mode of inter-picture prediction, in which a motion vector is derived from one or more motion vector predictors without the aid of an encoded motion vector component external to the predictor. In some other example embodiments, there may be a motion vector component applicable to the subject block. Therefore, the video encoder (703) may include Figure 7 Components not explicitly shown, such as a mode decision module for determining the extinction mode of a processing block.

[0105] exist Figure 7 In this example, the video encoder (703) includes Figure 7 An inter-frame encoder (730), an intra-frame encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) coupled together are shown in the example arrangement.

[0106] The inter-frame encoder (730) is configured to: receive samples of a current block (e.g., a processing block); compare the block with one or more reference blocks in a reference picture (e.g., blocks in a previous picture and a subsequent picture in a display sequence); generate inter-frame prediction information (e.g., description of redundant information, motion vectors, merge mode information according to an inter-frame coding technique); and calculate an inter-frame prediction result (e.g., a predicted block) based on the inter-frame prediction information using any suitable technique. In some examples, the reference picture is embedded in Figure 6The decoding unit 633 (shown as Figure 7 The residual decoder 728, as described in further detail below, decodes the decoded reference picture based on the encoded video information.

[0107] The intra encoder (722) is configured to: receive samples of a current block (e.g., a processing block); compare the block with an already encoded block in the same picture; and generate quantization coefficients after transformation; and in some cases also generate intra prediction information (e.g., intra prediction direction information according to one or more intra coding techniques). The intra encoder (722) can calculate an intra prediction result (e.g., a predicted block) based on the intra prediction information and a reference block in the same picture.

[0108] The general controller (721) can be configured to determine general control data and control other components of the video encoder (703) based on the general control data. In an example, the general controller (721) determines a prediction mode for a block and provides a control signal to a switch (726) based on the prediction mode. For example, when the prediction mode is intra mode, the general controller (721) controls the switch (726) to select an intra mode result for use by the residual calculator (723), and controls the entropy encoder (725) to select intra prediction information and include the intra prediction information in the bitstream; and when the prediction mode for the block is inter mode, the general controller (721) controls the switch (726) to select an inter prediction result for use by the residual calculator (723), and controls the entropy encoder (725) to select inter prediction information and include the inter prediction information in the bitstream.

[0109] The residual calculator (723) may be configured to calculate the difference (residual data) between the received block and the prediction result for the block selected from the intra encoder (722) or the inter encoder (730). The residual encoder (724) may be configured to encode the residual data to generate a transform coefficient. For example, the residual encoder (724) may be configured to convert the residual data from the spatial domain to the frequency domain to generate a transform coefficient. Then, the transform coefficient is subjected to a quantization process to obtain a quantized transform coefficient. In various example embodiments, the video encoder (703) further includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data may be appropriately used by the intra encoder (722) and the inter encoder (730). For example, the inter encoder (730) may generate a decoded block based on the decoded residual data and the inter prediction information, and the intra encoder (722) may generate a decoded block based on the decoded residual data and the intra prediction information. The decoded blocks are appropriately processed to generate decoded pictures, and these decoded pictures may be buffered in a memory circuit (not shown) and used as reference pictures.

[0110] The entropy encoder (725) may be configured to format the bitstream to include the encoded blocks and perform entropy encoding. The entropy encoder (725) may be configured to include various information in the bitstream. For example, the entropy encoder (725) may be configured to include general control data, selected prediction information (e.g., intra-frame prediction information or inter-frame prediction information), residual information, and other suitable information in the bitstream. When the block is encoded in an inter-frame mode or a merge sub-mode of a bi-prediction mode, there may be no residual information.

[0111] Figure 8 A diagram of an example video decoder (810) according to another embodiment of the present disclosure is shown. The video decoder (810) is configured to receive an encoded picture as part of an encoded video sequence and decode the encoded picture to generate a reconstructed picture. In an example, the video decoder (810) can be used to replace Figure 4 A video decoder (410) is shown in an example.

[0112] exist Figure 8 In this example, the video decoder (810) includes Figure 8 An entropy decoder (871), an inter-frame decoder (880), a residual decoder (873), a reconstruction module (874), and an intra-frame decoder (872) are shown coupled together in the example arrangement.

[0113] The entropy decoder (871) can be configured to reconstruct certain symbols from the encoded picture, which represent syntax elements constituting the encoded picture. Such symbols may include, for example, a mode for encoding a block (e.g., intra-frame mode, inter-frame mode, bidirectional prediction mode, merge sub-mode, or another sub-mode), prediction information (e.g., intra-frame prediction information or inter-frame prediction information) that can identify certain samples or metadata used by the intra-frame decoder (872) or the inter-frame decoder (880) for prediction, respectively, residual information in the form of, for example, quantized transform coefficients, etc. In the example, when the prediction mode is inter-frame mode or bidirectional prediction mode, the inter-frame prediction information is provided to the inter-frame decoder (880); and when the prediction type is an intra-frame prediction type, the intra-frame prediction information is provided to the intra-frame decoder (872). The residual information can be subjected to inverse quantization and provided to the residual decoder (873).

[0114] The inter-frame decoder (880) may be configured to receive the inter-frame prediction information and generate an inter-frame prediction result based on the inter-frame prediction information.

[0115] The intra decoder (872) may be configured to receive intra prediction information and generate prediction results based on the intra prediction information.

[0116] The residual decoder (873) can be configured to perform inverse quantization to extract dequantized transform coefficients, and process the dequantized transform coefficients to convert the residual from the frequency domain to the spatial domain. The residual decoder (873) can also utilize certain control information (to include quantizer parameters (Quantizer Parameter, QP)), which can be provided by the entropy decoder (871) (since this may only be a small amount of control information, the data path is not depicted).

[0117] The reconstruction module (874) can be configured to combine the residual output by the residual decoder (873) with the prediction result (in this case, it can be output by the inter-frame prediction module or the intra-frame prediction module) in the spatial domain to form a reconstructed block, which forms a part of the reconstructed picture as part of the reconstructed video. Note that other suitable operations such as deblocking operations can also be performed to improve visual quality.

[0118] Note that the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using any suitable technology. In some example embodiments, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using one or more processors executing software instructions.

[0119] (603) and (703) and video decoders (410), (510) and (810).

[0120] Turning to block segmentation for encoding and decoding, the general segmentation can start from the basic block and can follow a predefined set of rules, a specific pattern, a segmentation tree, or any segmentation structure or scheme. The segmentation can be hierarchical and recursive. After following any of the example segmentation processes described below or other processes or combinations thereof to divide or segment the basic block, a final set of partitions or coding blocks can be obtained. Each of these partitions can be at one of the various segmentation levels in the segmentation hierarchy and can have various shapes. Each partition in the partition can be referred to as a coding block (Coding Block, CB). For various example segmentation implementations further described below, each generated CB can have any allowed size and segmentation level. Such partitions are called coding blocks because they can form the following units, for which some basic encoding / decoding decisions can be made and encoding / decoding parameters can be optimized, determined, and signaled in the coded video bitstream. The highest or deepest level in the final partition represents the depth of the coding block segmentation tree structure. The coding block can be a luminance coding block or a chrominance coding block. The CB tree structure for each color can be referred to as a coding block tree (Coding Block Tree, CBT).

[0121] The coding blocks of all color channels may be collectively referred to as coding units (CUs). The hierarchical structure of all color channels may be collectively referred to as coding tree units (CTUs). The partitioning modes or structures for various color channels in a CTU may be the same or different.

[0122] In some implementations, the segmentation tree schemes or structures for the luma channel and the chroma channels may not need to be the same. In other words, the luma channel and the chroma channel may have separate coding tree structures or modes. In addition, whether the luma channel and the chroma channel use the same or different coding segmentation tree structures and the actual coding segmentation tree structures to be used may depend on whether the slice being encoded is a P slice, a B slice, or an I slice. For example, for an I slice, the chroma channel and the luma channel may have separate coding segmentation tree structures or coding segmentation tree structure modes, while for a P slice or a B slice, the luma channel and the chroma channel may share the same coding segmentation tree scheme. When a separate coding segmentation tree structure or mode is applied, the luma channel may be segmented into CBs by one coding segmentation tree structure, and the chroma channel may be segmented into chroma CBs by another coding segmentation tree structure.

[0123] In some example implementations, a predetermined partitioning pattern may be applied to a basic block. Fig. 9As shown, the example 4-way partitioning tree can start from a first predefined level (e.g., 64×64 block level or other size as basic block size), and the basic block can be hierarchically partitioned down to a predefined lowest level (e.g., 4×4 level). For example, the basic block can be subjected to four predefined partitioning options or modes indicated by 902, 904, 906 and 908, where the partition designated as R is allowed for recursive partitioning because it can be repeated at lower levels as shown in FIG. Fig. 9 In some implementations, the same partitioning options indicated can be used for Fig. 9 Additional restrictions apply to the partitioning scheme. Fig. 9 In the implementation of , rectangular partitioning (e.g., 1:2 / 2:1 rectangular partitioning) may be allowed, but rectangular partitioning may not be allowed to be recursive, while square partitioning may be allowed to be recursive. If necessary, follow Fig. 9 The recursive case of the split generates a final set of coding blocks. The coding tree depth can be further defined to indicate the split depth from the root node or root block. For example, the coding tree depth for the root node or root block, such as a 64×64 block, can be set to 0, and the root block follows Fig. 9 After being further split once, the coding tree depth increases by 1. For the above scheme, the maximum or deepest level of the minimum split from 64×64 basic blocks to 4×4 will be 4 (starting from level 0). Such a splitting scheme can be applied to one or more color channels. It can be followed Fig. 9 The scheme of splitting each color channel independently (for example, the splitting mode or option in the predefined mode can be independently determined for each of the color channels at each hierarchical level). Alternatively, two or more color channels in the color channel can share Fig. 9 The same hierarchical pattern tree (e.g., the same segmentation pattern or option in the predefined pattern can be selected for two or more color channels at each hierarchical level).

[0124] Fig.10 Another example predefined partitioning pattern that allows recursive partitioning to form a partitioning tree is shown. Fig.10 As shown, an example 10-way partitioning structure or pattern may be predefined. A root block may start at a predefined level (eg, from a basic block at a 128x128 level or a 64x64 level). Fig.10 Example partitioning structures include various 2:1 / 1:2 and 4:1 / 1:4 rectangular partitionings. Fig.10 The partition type with 3 sub-partitions indicated as 1002, 1004, 1006 and 1008 in the second row of can be referred to as a "T-type" partition. The "T-type" partitions 1002, 1004, 1006 and 1008 can be referred to as a left T-type, a top T-type, a right T-type and a bottom T-type. In some example implementations, Fig.10 The rectangular partition is not allowed to be further subdivided. The coding tree depth can be further defined to indicate the split depth from the root node or root block. For example, the coding tree depth for the root node or root block, such as a 128×128 block, can be set to 0, and the root block follows Fig.10 After being further split once, the coding tree depth increases by 1. In some implementations, only full square partitions in 1010 may be allowed to follow Fig.10 The pattern recursively splits to the next level of the partition tree. In other words, for the square partition with T-patterns 1002, 1004, 1006 and 1008, recursive partitioning may not be allowed. If necessary, follow Fig.10 The recursive case of segmentation generates the final set of coding blocks. Such a scheme can be applied to one or more color channels. In some implementations, more flexibility can be added to the use of segmentation below the 8×8 level. For example, 2×2 chroma inter-frame prediction can be used in some cases.

[0125] In some other example implementations for coding block segmentation, a quadtree structure can be used to split a basic block or an intermediate block into quadtree partitions. Such a quadtree split can be applied hierarchically and recursively to any square-shaped partition. Whether a basic block or an intermediate block or a partition is further quadtree split can be adapted to various local characteristics of the basic block or intermediate block / partition. The quadtree splitting at the image boundary can be further adjusted. For example, an implicit quadtree splitting can be performed at a picture boundary so that the block will remain quadtree split until the size fits the picture boundary.

[0126] In some other example implementations, hierarchical binary segmentation from basic blocks can be used. For such a scheme, a basic block or an intermediate level block can be segmented into two partitions. The binary segmentation can be horizontal or vertical. For example, horizontal binary segmentation can segment a basic block or an intermediate block into equal right and left partitions. Similarly, vertical binary segmentation can split a basic block or an intermediate block into equal upper and lower partitions. Such binary segmentation can be hierarchical and recursive. The following decision can be made at each of the basic blocks or intermediate blocks: whether the binary segmentation scheme should continue, and if the scheme does continue further, whether horizontal binary segmentation or vertical binary segmentation should be used. In some implementations, further segmentation can end at a predefined minimum segmentation size (in one or two dimensions). Alternatively, further segmentation can end once a predefined segmentation level or depth from the basic block is reached. In some implementations, the aspect ratio of the partition can be limited. For example, the aspect ratio of the partition can be no less than 1:4 (or greater than 4:1). Thus, a vertical stripe partition having a vertical to horizontal aspect ratio of 4:1 can only be further vertically binary-partitioned into upper and lower partitions each having a vertical to horizontal aspect ratio of 2:1.

[0127] In yet other examples, the ternary partitioning scheme may be used to partition a basic block or any intermediate block, such as Fig.13 The ternary model can be implemented vertically, as Fig.13 1302, or implemented horizontally, as Fig.13 1304. Although Fig.13 The example split ratios in are shown as 1:2:1 vertically or horizontally, but other ratios may also be predefined. In some implementations, two or more different ratios may be predefined. Such a ternary segmentation scheme may be used to supplement a quadtree segmentation structure or a binary segmentation structure, because such a ternary tree segmentation is able to capture objects located at the center of a block in one continuous partition, while quadtrees and binary trees always split along the center of a block and thus split objects into separate partitions. In some implementations, the width and height of the partitions of the example ternary tree are always powers of 2 to avoid additional transformations.

[0128] The above partitioning schemes can be combined in any way at different partitioning levels. As an example, the above quadtree partitioning scheme and binary partitioning scheme can be combined to partition the basic block into a quadtree-binary-tree (QTBT) structure. In such a scheme, a basic block or an intermediate block / partition can be quadtree-split or binary-split according to a set of predefined conditions (if specified). Fig.14 A specific example is shown in FIG. Fig.14In the example of , the basic block is first split into four partitions by a quadtree, as shown in 1402, 1404, 1406 and 1408. Thereafter, each of the generated partitions is split into four additional partitions (e.g., 1408) by a quadtree at the next level, or is binary split into two additional partitions (e.g., horizontally or vertically, such as 1402 or 1406, both of which are symmetrical), or is not split (e.g., 1404). For square-shaped partitions, binary splits or quadtree splits can be recursively allowed, as shown in the entire example partitioning pattern of 1410 and the corresponding tree structure / representation in 1420, where solid lines represent quadtree splits and dashed lines represent binary splits. A flag can be used for each binary split node (non-leaf binary partition) to indicate whether the binary split is horizontal or vertical. For example, as shown in 1420, consistent with the partitioning structure of 1410, the flag "0" can represent a horizontal binary split and the flag "1" can represent a vertical binary split. For quadtree split partitions, there is no need to indicate the split type, because quadtree splits always split a block or partition horizontally and vertically to produce 4 sub-blocks / partitions of equal size. In some implementations, a flag "1" may indicate a horizontal binary split and a flag "0" may indicate a vertical binary split.

[0129] In some example implementations of QTBT, the quadtree and binary splitting rule set may be represented by the following predefined parameters and corresponding functions associated therewith:

[0130] –CTU size: the root node size of the quadtree (the size of the basic block)

[0131] –MinQTSize: Minimum allowed quadtree leaf node size

[0132] –MaxBTSize: Maximum allowed binary tree root node size

[0133] –MaxBTDepth: Maximum allowed binary tree depth

[0134] –MinBTSize: The minimum allowed binary tree leaf node size

[0135] In some example implementations of the QTBT partitioning structure, the CTU size can be set to 128×128 luma samples with two corresponding 64×64 blocks of chroma samples (when example chroma subsampling is considered and used), MinQTSize can be set to 16×16, MaxBTSize can be set to 64×64, MinBTSize (for both width and height) can be set to 4×4, and MaxBTDepth can be set to 4. Quadtree partitioning can be first applied to the CTU to generate quadtree leaf nodes. The quadtree leaf node can have a size from its minimum allowed size of 16×16 (i.e., MinQTSize) to 128×128 (i.e., the CTU size). If the node is 128×128, the node will not be split first through the binary tree because the size exceeds the MaxBTSize (i.e., 64×64). Otherwise, nodes that do not exceed MaxBTSize can be split through the binary tree. Fig.14 In the example, the basic block is 128×128. According to a predefined set of rules, the basic block can only be quadtree split. The basic block has a split depth of 0. Each of the four partitions generated is 64×64, does not exceed MaxBTSize, and can be further quadtree or binary split at level 1. Continue with this process. When the binary tree depth reaches MaxBTDepth (i.e., 4), no further splitting can be considered. When the binary tree node has a size equal to MinBTSize

[0136] (ie, 4) width, no further horizontal splitting may be considered. Similarly, when the binary tree node has a height equal to MinBTSize, no further vertical splitting is considered.

[0137] In some example implementations, the above-mentioned QTBT scheme can be configured to support the flexibility of having the same QTBT structure or separate QTBT structures for luma and chroma. For example, for P slices and B slices, the luma CTB and chroma CTB in one CTU can share the same QTBT structure. However, for I slices, the luma CTB can be partitioned into CBs by the QTBT structure, and the chroma CTB can be partitioned into chroma CBs by another QTBT structure. This means that a CU can be used to refer to different color channels in an I slice, for example, an I slice can consist of coding blocks of a luma component or coding blocks of two chroma components, and a CU in a P or B slice can consist of coding blocks of all three color components.

[0138] In some other implementations, the QTBT solution can be supplemented with the above ternary solution. Such an implementation can be called a multi-type tree (MTT) structure. For example, in addition to the binary split of the node, Fig.13 In some implementations, only square nodes can be subjected to a ternary split. An additional flag may be used to indicate whether the ternary split is horizontal or vertical.

[0139] The design of two-level trees or multi-level trees, such as the QTBT implementation and the QTBT implementation supplemented by ternary splitting, can be mainly for reducing complexity. In theory, the complexity of traversing the tree is T D , where T represents the number of split types and D is the depth of the tree. A trade-off can be made by using multiple types (T) while reducing the depth (D).

[0140] In some implementations, the CB may be further segmented. For example, for the purpose of intra-frame or inter-frame prediction during the encoding and decoding process, the CB may also be segmented into a plurality of prediction blocks (Prediction Block, PB). In other words, the CB may also be divided into different sub-partitions, in which separate prediction decisions / configurations may be made. In parallel, for the purpose of depicting the level of transformation or inverse transformation of the video data, the CB may also be segmented into a plurality of transform blocks (Transform Block, TB). The schemes for segmenting the CB into PBs and TBs may be the same or different. For example, each segmentation scheme may be performed using its own process based on, for example, various features of the video data. In some example implementations, the PB segmentation scheme and the TB segmentation scheme may be independent. In some other example implementations, the PB segmentation scheme and the TB segmentation scheme may be related to the boundary. In some implementations, for example, the TB may be segmented after the PB segmentation, and in particular, after each PB is determined after segmenting the coding block, the PB may then be further segmented into one or more TBs. For example, in some implementations, the PB may be split into one, two, four, or other number of TBs.

[0141] In some implementations, in order to split the basic block into coding blocks and further split into prediction blocks and / or transform blocks, the luma channel and the chroma channel may be processed differently. For example, in some implementations, splitting the coding block into prediction blocks and / or transform blocks may be allowed for the luma channel, while such splitting of the coding block into prediction blocks and / or transform blocks may not be allowed for the chroma channel. In such an implementation, the transformation and / or prediction of the luma block can therefore be performed only at the coding block level. For another example, the minimum transform block size for the luma channel and the chroma channel may be different, for example, the coding block for the luma channel may be allowed to be split into smaller transform blocks and / or prediction blocks than the chroma channel. For another example, the maximum depth of splitting the coding block into transform blocks and / or prediction blocks may be different between the luma channel and the chroma channel, for example, the coding block for the luma channel may be allowed to be split into deeper transform blocks and / or prediction blocks than the chroma channel. For a specific example, the luma coding block may be partitioned into transform blocks of multiple sizes that may be represented by recursive partitioning down to up to 2 levels, and transform block shapes such as square, 2:1 / 1:2, and 4:1 / 1:4 and transform block sizes from 4×4 to 64×64 may be allowed. However, for chroma blocks, only the largest possible transform block specified for the luma block may be allowed.

[0142] In some example implementations for partitioning a coding block into PBs, the depth, shape, and / or other characteristics of the PB partitioning may depend on whether the PB is intra-coded or inter-coded.

[0143] Splitting a coding block (or prediction block) into transform blocks can be implemented in various example schemes, including but not limited to quadtree splitting and predefined pattern splitting in a recursive or non-recursive manner, and with additional consideration of transform blocks at the boundaries of coding blocks or prediction blocks. In general, the resulting transform blocks may be at different split levels, may not be of the same size, and may not need to be square in shape (e.g., they may be rectangular with some allowed size and aspect ratio). Fig.15 , Fig.16 and Fig.17 Additional examples are described in further detail.

[0144] However, in some other implementations, the CB obtained via any partitioning scheme can be used as a basic or minimum coding block for prediction and / or transformation. In other words, no further splitting is performed for the purpose of performing inter-frame prediction / intra-frame prediction and / or for the purpose of transformation. For example, the CB obtained according to the above-mentioned QTBT scheme can be directly used as a unit for performing prediction. Specifically, such a QTBT structure removes the concept of multiple partitioning types, that is, such a QTBT structure removes the separation of CU, PU and TU, and supports more flexibility for the CU / CB partition shape as described above. In such a QTBT block structure, the CU / CB can have a square or rectangular shape. The leaf nodes of such a QTBT are used as units for prediction and transformation processing without any further partitioning. This means that in such an example QTBT coding block structure, CU, PU and TU have the same block size.

[0145] The various CB segmentation schemes described above and further segmenting the CB into PB and / or TB (including not performing PB / TB segmentation) may be combined in any manner. The following specific implementations are provided as non-limiting examples.

[0146] A specific example implementation of coding block and transform block segmentation is described below. In such an example implementation, the recursive quadtree splitting described above or a predefined splitting pattern (e.g. Fig. 9 and Fig.10 The basic block is split into coding blocks according to those splitting modes in . At each level, whether the further quadtree splitting of a specific partition should continue can be determined by local video data features. The generated CB can be at various quadtree splitting levels and have various sizes. A decision can be made at the CB level (or CU level, for all three color channels) about whether to use inter-frame picture (time) prediction or intra-frame picture (spatial) prediction to encode the picture area. Each CB can be further split into one, two, four or other number of PBs according to a predefined PB splitting type. Inside a PB, the same prediction process can be applied, and relevant information can be sent to the decoder based on the PB. After obtaining the residual block by applying the prediction process based on the PB splitting type, the CB can be divided into TBs according to another quadtree structure similar to the coding tree for the CB. In this specific implementation, the CB or TB can be, but not necessarily limited to, a square shape. In addition, in this specific example, the PB for inter-frame prediction can be a square or rectangular shape, while the PB for intra-frame prediction can be only a square. The coding block can be split into, for example, four square-shaped TBs. Each TB can be further recursively split (using quadtree splitting) into smaller TBs, which are called Residual Quad-Tree (RQT).

[0147] Another example implementation for partitioning a basic block into CBs, PBs, and / or TBs is further described below. Fig. 9 or Fig.10 Instead of using multiple segmentation unit types such as those shown in the figure, a quadtree with nested multi-type trees using binary and ternary split partition structures (e.g., QTBT or QTBT with ternary split as described above) can be used. The separation of CB, PB and TB (i.e., CB is split into PB and / or TB, and PB is split into TB) can be abandoned, except in the following case: a CB with a size that is too large for the maximum transform length is required, where such a CB may need to be further split. The example segmentation scheme can be designed to support more flexibility for the CB partition shape, so that both prediction and transformation can be performed at the CB level without further division. In such a coding tree structure, the CB can have a square or rectangular shape. Specifically, the coding tree block (CTB) can be first split by a quadtree structure. Then, the quadtree leaf nodes can be further split by nesting the multi-type tree structure. Fig.11 An example of a nested multi-type tree structure using binary or ternary splitting is shown in FIG. Fig.11 The example multi-type tree structure includes four split types, referred to as vertical binary split (SPLIT_BT_VER) (1102), horizontal binary split (SPLIT_BT_HOR) (1104), vertical ternary split (SPLIT_TT_VER) (1106), and horizontal ternary split (SPLIT_TT_HOR) (1108). The CB then corresponds to the leaves of the multi-type tree. In this example implementation, unless the CB is too large for the maximum transform length, the partition is used for both prediction and transform processing without any further partitioning. This means that in most cases, CB, PB, and TB have the same block size in a quadtree with a nested multi-type tree coding block structure. The exception occurs when the maximum supported transform length is less than the width or height of the color component of the CB. In some implementations, in addition to the binary or ternary split, Fig.11 The nested pattern can also include quadtree splits.

[0148] Fig.12 A specific example of a quadtree with a nested multi-type tree coding block structure (including quadtree, binary and ternary split options) for block partitioning of a basic block is shown in FIG. Fig.12 The basic block 1200 is shown to be split into four square partitions 1202, 1204, 1206 and 1208 by a quadtree. Further use is made for each of the quadtree split partitions Fig.11 The multi-type tree structure and quadtree are used to make further splitting decisions. Fig.12 In the example of , partition 1204 is not further split. Partitions 1202 and 1208 are each split using another quadtree. For partition 1202, the upper left, upper right, lower left, and lower right partitions of the second level quadtree split are split using the third level quadtree, Fig.11 Horizontal binary split 1104, non-split and Fig.11 The horizontal ternary split 1108 of the partition 1208 is adopted by another quadtree split, and the upper left, upper right, lower left and lower right partitions of the second level quadtree split are respectively adopted by the third level Fig.11 The vertical ternary split 1106 split, non-split, non-split and Fig.11 The horizontal binary split 1104. The two sub-partitions in the sub-partition of the third level upper left partition 1208 are respectively based on Fig.11 The horizontal binary split 1104 and the horizontal ternary split 1108 are further split. The partition 1206 adopts the following Fig.11 The second level splitting pattern of the vertical binary split 1102 forms two partitions, and the two partitions are divided into two partitions according to the third level. Fig.11 The horizontal ternary split 1108 and vertical binary split 1102 are further split. Fig.11 A fourth level split is further applied to one of the partitions by horizontal binary split 1104 .

[0149] For the specific example above, the maximum luma transform size may be 64x64, and the maximum supported chroma transform size may be different from luma, such as 32x32. Fig.12 The above example CB is usually not further split into smaller PBs and / or TBs, but when the width or height of the luma coding block or the chroma coding block is larger than the maximum transform width or height, the luma coding block or the chroma coding block can be automatically split in the horizontal and / or vertical directions to meet the transform size limit in that direction.

[0150] In a specific example for partitioning a basic block into the above-mentioned CBs, and as described above, the coding tree scheme can support the ability for luma and chroma to have separate block tree structures. For example, for P slices and B slices, the luma CTB and chroma CTB in one CTU can share the same coding tree structure. For example, for I slices, luma and chroma can have separate coding block tree structures. When a separate block tree structure is applied, the luma CTB can be partitioned into luma CBs by one coding tree structure, and the chroma CTB can be partitioned into chroma CBs by another coding tree structure. This means that a CU in an I slice can include coding blocks for a luma component or coding blocks for two chroma components, while a CU in a P slice or a B slice always includes coding blocks for all three color components unless the video is monochrome.

[0151] When a coding block is further partitioned into multiple transform blocks, the transform blocks in the coding block can be ordered in the bitstream in various orders or scanning modes. The following is further described in detail example implementations for partitioning a coding block or prediction block into transform blocks and the encoding order of the transform blocks. In some example implementations, as described above, transform partitioning can support transform blocks of multiple shapes such as 1:1 (square), 1:2 / 2:1, and 1:4 / 4:1, where the transform block size ranges from, for example, 4×4 to 64×64. In some implementations, if the coding block is less than or equal to 64×64, the transform block partitioning can be applied only to the luminance component, so that for the chrominance block, the transform block size is the same as the coding block size. Otherwise, if the coding block width or height is greater than 64, both the luminance coding block and the chrominance coding block can be implicitly split into multiple min(W,64)×min(H,64) transform blocks and min(W,32)×min(H,32) transform blocks, respectively.

[0152] In some example implementations of transform block partitioning, for both intra-coded blocks and inter-coded blocks, the coding block may be further partitioned into multiple transform blocks with a partitioning depth up to a predefined number of levels (e.g., 2 levels). The transform block partitioning depth and size may be related. For some example implementations, the mapping from the transform size of the current depth to the transform size of the next depth is shown below in Table 1.

[0153] Table 1: Transform segment size settings

[0154]

[0155] Based on the example mapping of Table 1, for a 1:1 square block, the next level transform split can create four 1:1 square sub-transform blocks. The transform partitioning can stop at 4×4, for example. Therefore, the transform size of 4×4 at the current depth corresponds to the same size of 4×4 at the next depth. In the example of Table 1, for a 1:2 / 2:1 non-square block, the next level transform split can create two 1:1 square sub-transform blocks, and for a 1:4 / 4:1 non-square block, the next level transform split can create two 1:2 / 2:1 sub-transform blocks.

[0156] In some example implementations, for the luma component of an intra-coded block, additional restrictions may be applied with respect to transform block partitioning. For example, for each level of transform partitioning, all sub-transform blocks may be restricted to be of equal size. For example, for a 32×16 coding block, a level 1 transform split creates two 16×16 sub-transform blocks and a level 2 transform split creates eight 8×8 sub-transform blocks. In other words, the second level split must be applied to all first level sub-blocks to keep the transform units of equal size. Fig.15An example of transform block partitioning for intra-coded square blocks following Table 1 and the coding order shown by arrows is shown in 1502. Specifically, a square coding block is shown at 1502. A first level splitting into 4 equal-sized transform blocks according to Table 1 is shown at 1504, where the coding order is indicated by arrows. A second level splitting of all first level equal-sized blocks into 16 equal-sized transform blocks according to Table 1 is shown at 1506, where the coding order is indicated by arrows.

[0157] In some example implementations, the above restrictions on intra coding may not apply to the luma component of an inter-coded block. For example, after the first level of transform splitting, any of the sub-transform blocks may be further independently split into a higher level. Thus, the resulting transform blocks may or may not have the same size. Fig.16 An example of splitting an inter-frame coding block into transform blocks and the coding order of the transform blocks is shown in FIG. Fig.16 In the example of FIG. 1602, according to Table 1, the inter-coded block 1602 is split into two levels of transform blocks. At the first level, the inter-coded block is split into four transform blocks of equal size. Then, only one of the four transform blocks (rather than all transform blocks) is further split into four sub-transform blocks, resulting in a total of 7 transform blocks with two different sizes, as shown in 1604. The example coding order of these 7 transform blocks is as follows: Fig.16 As shown by the arrow in 1604.

[0158] In some example implementations, for chroma components, some additional restrictions on transform blocks may be applied. For example, for chroma components, the transform block size may be as large as the coding block size, but not smaller than a predefined size, such as 8×8.

[0159] In some other example implementations, for coding blocks having a width (W) or a height (H) greater than 64, both luma coding blocks and chroma coding blocks may be implicitly split into multiple min(W, 64)×min(H, 64) transform units and min(W, 32)×min(H, 32) transform units, respectively. Here, in the present disclosure, "min(a, b)" may produce a smaller value between a and b.

[0160] Fig.17 Another alternative example scheme for partitioning a coding block or prediction block into transform blocks is also shown. Fig.17 As shown, instead of using recursive transform partitioning, a predefined set of partition types can be applied to the coding block according to the transform type of the coding block. Fig.17In the specific example shown, one of six example partitioning types may be applied to split the coding block into various numbers of transform blocks. Such a generated transform block partitioning scheme may be applied to either a coding block or a prediction block.

[0161] In more detail, Fig.17 The partitioning scheme of provides up to 6 example partitioning types for any given transform type (transform type refers to, for example, the type of the main transform, such as ADST and others). In this scheme, a transform partitioning type can be assigned to each coding block or prediction block based on, for example, rate-distortion cost. In an example, the transform partitioning type assigned to a coding block or prediction block can be determined based on the transform type of the coding block or prediction block. A specific transform partitioning type can correspond to a transform block split size and mode (or partitioning type), such as Fig.17 The six transform partition types shown are shown. The correspondence between various transform types and various transform partition types can be predefined. An example is shown below, where the capitalized mark indicates the transform partition type that can be assigned to the coding block or prediction block based on the rate-distortion cost:

[0162] PARTITION_NONE: Allocate a transform size equal to the block size

[0163] PARTITION_SPLIT: Allocate a transform size of 1 / 2 the block size in width and 1\2 the block size in height.

[0164] PARTITION_HORZ: Allocates a transform size with the same width as the block size and a height of 1\2 of the block size.

[0165] PARTITION_VERT: Allocates a transform size with a width of 1\2 the block size and the same height as the block size.

[0166] PARTITION_HORZ4: Allocates a transform size with the same width as the block size and a height of 1\4 of the block size.

[0167] PARTITION_VERT4: Allocates a transform size with a width of 1\4 of the block size and the same height as the block size.

[0168] In the above example, if Fig.17 The transform partition type shown contains uniform transform sizes for all the partitioned transform blocks. This is only an example and not a limitation. In some other implementations, a mixed transform block size may be used for the partitioned transform blocks under a specific partition type (or mode).

[0169] The PB (or CB, also referred to as a PB when not further partitioned into prediction blocks) obtained according to any of the above partitioning schemes can then become a separate block for encoding via intra-frame prediction or inter-frame prediction. For inter-frame prediction for the current PB, a residual between the current block and the prediction block can be generated, encoded, and included in the encoded bitstream.

[0170] Inter prediction can be implemented, for example, in a single reference mode or a composite reference mode. In some implementations, a skip flag may be first included in the bitstream for the current block (or at a higher level) to indicate whether the current block is inter-coded and the current block is not skipped. If the current block is inter-coded, another flag may be further included in the bitstream as a signal for indicating whether a single reference mode or a composite reference mode is used to predict the current block. For a single reference mode, one reference block may be used to generate a prediction block for the current block. For a composite reference mode, two or more reference blocks may be used to generate a prediction block, for example, by weighted averaging. A composite reference mode may be referred to as more than one reference mode, two reference modes, or multiple reference modes. A reference block or multiple reference blocks may be identified using a reference frame index or multiple reference frame indexes and additionally using a corresponding motion vector or multiple motion vectors indicating a displacement between the reference block and the current block in position, for example, in horizontal pixels and vertical pixels. For example, an inter-frame prediction block for a current block may be generated from a single reference block identified as a prediction block in a single reference mode by a motion vector in a reference frame, while for a composite reference mode, the prediction block may be generated by a weighted average of two reference blocks in two reference frames indicated by two reference frame indices and two corresponding motion vectors. The motion vector may be encoded and included in a bitstream in various ways.

[0171] In some implementations, the encoding or decoding system may maintain a decoded picture buffer (Decoded PictureBuffer, DPB). Some images / pictures may be kept in the DPB to wait for display (in the decoding system), and some images / pictures in the DPB may be used as reference frames to enable inter-frame prediction (in the decoding system or the encoding system). In some implementations, the reference frames in the DPB may be marked as short-term references or long-term references for the current image being encoded or decoded. For example, a short-term reference frame may include a frame used to inter-frame predict a block in the current frame or a block in a predefined number (e.g., 2) of subsequent video frames closest to the current frame in decoding order. A long-term reference frame may include a frame in the DPB that may be used to predict image blocks in frames that are more than a predefined number of frames away from the current frame in decoding order. Information about such tags for short-term reference frames and long-term reference frames may be referred to as a reference picture set (Reference Picture Set, RPS) and may be added to the header of each frame in the encoded bitstream. Each frame in the coded video stream may be identified by a Picture Order Counter (POC), which is numbered in an absolute manner according to playback order or relative to a group of pictures, eg, starting with an I-frame.

[0172] In some example implementations, one or more reference picture lists containing identifications of short-term reference frames and long-term reference frames for inter-frame prediction may be formed based on information in the RPS. For example, a single picture reference list may be formed for unidirectional inter-frame prediction, denoted as L0 reference (or reference list 0), while two picture reference lists may be formed for bidirectional inter-frame prediction, denoted as L0 (or reference list 0) and L1 (or reference list 1) for each of the two prediction directions. The reference frames included in the L0 list and the L1 list may be ordered in various predetermined ways. The lengths of the L0 list and the L1 list may be signaled in the video bitstream. Unidirectional inter-frame prediction may be performed in a single reference mode, or in a composite reference mode when multiple references used to generate a prediction block by weighted averaging in a composite prediction mode are on the same side of the block to be predicted. Bidirectional inter-frame prediction may be performed only in composite mode because bidirectional inter-frame prediction involves at least two reference blocks.

[0173] Block segmentation

[0174] Fig.18 An example partition tree for block partitioning is shown. VP9 uses a 4-way partition tree starting at the 64×64 level down to the 4×4 level, with additional restrictions for blocks 8×8 and below. This is Fig.18. The partition designated as R can be recursive in that the same partition tree is repeated at a lower scale until the lowest 4×4 level is reached. As shown in the figure, AV1 expands the partition tree to a 10-way structure as shown, but also increases the maximum size (called a superblock in VP9 / AV1 terminology) to start at 128x128. This includes 4:1 / 1:4 rectangular partitions that did not exist in VP9. Rectangular partitions cannot be further subdivided. In addition, AV1 adds more flexibility for using partitions below the 8×8 level. For example, 2×2 chroma inter-frame prediction is now possible in some cases.

[0175] In HEVC, the coding tree unit (CTU) is split into coding units (CU) using a quadtree structure represented as a coding tree to adapt to various local features. CU can also be considered as a block, including a prediction block or a coding block. A decision is made at the CU level as to whether to use inter-picture (temporal) prediction or intra-picture (spatial) prediction to encode the picture area. Each CU can be further split into one, two or four prediction units (PUs) according to the PU split type. Within a PU, the same prediction process is applied, and the relevant information is sent to the decoder on a PU basis. After obtaining the residual block by applying the prediction process based on the PU split type, the CU can be split into transform units (TUs) according to another quadtree structure similar to the coding tree for the CU. The HEVC structure has multiple partitioning concepts including CU, PU and TU. In HEVC, a CU or TU can be only square in shape, while a PU can be a square or rectangular shape for an inter-frame prediction block. In HEVC, a coding block can be further split into four square sub-blocks, and a transform is performed on each sub-block (ie, TU). Each TU can be further recursively split (using quadtree splitting) into smaller TUs, which is called residual quadtree (RQT). At the picture boundary, HEVC uses implicit quadtree splitting so that the block will remain quadtree split until the size fits the picture boundary.

[0176] There may be a block partitioning structure using a quadtree (Quadtree, QT) plus a binary tree (Binary Tree, BT). In HEVC, a CTU can be split into CUs using a quadtree structure represented as a coding tree to accommodate various local features. A decision is made at the CU level as to whether to use inter-picture (temporal) prediction or intra-picture (spatial) prediction to encode a picture area. In some embodiments, each CU can be further split into one, two, or four PUs according to the PU splitting type. Within a PU, the same prediction process can be applied, and relevant information is sent to the decoder on a PU basis. After obtaining the residual block by applying a prediction process based on the PU splitting type, the CU can be partitioned into transform units (TUs) according to another quadtree structure similar to the coding tree for the CU. The HEVC structure may have multiple partitioning concepts including CU, PU, ​​and TU.

[0177] Fig.19 An example partition and tree for a quadtree plus binary tree (QTBT) structure is shown. The QTBT structure may not have a concept similar to multiple partition types, and the QTBT structure may remove the separation of CU, PU, ​​and TU concepts. The QTBT structure may support increased flexibility for CU partition shapes. In some embodiments of the QTBT block structure, the CU may have a square or rectangular shape.

[0178] like Fig.19 As shown, the coding tree unit (CTU) is first partitioned according to a quadtree structure. The quadtree leaf nodes are further partitioned according to a binary tree structure. In the binary tree splitting, there are two types of splitting, symmetric horizontal splitting and symmetric vertical splitting. The binary tree leaf nodes are called coding units (CUs), and the partitions can be used for prediction and transform processing without any other partitioning. CU, PU, ​​and TU can have the same block size in the QTBT coding block structure. In JEM, a CU may include coding blocks (CBs) of different color components (for example, in the case of P slices and B slices in 4:2:0 chroma format, a CU includes a luminance CB and two chroma CBs). In other embodiments, a CU may include a CB of a single component (for example, in the case of an I slice, a CU includes a luminance CB or two chroma CBs).

[0179] For the QTBT segmentation scheme, the following parameters can be defined:

[0180] CTU size: The size of the root node of the quadtree, the same concept as in HEVC

[0181] MinQTSize: The minimum allowed size of a quadtree leaf node

[0182] MaxBTSize: Maximum allowed binary tree root node size

[0183] MaxBTDepth: Maximum allowed binary tree depth

[0184] MinBTSize: The minimum allowed binary tree leaf node size

[0185] In one embodiment of the QTBT partitioning structure, the CTU size can be set to 128×128 luma samples with two corresponding 64×64 chroma sample blocks. In one embodiment, MinQTSize can be set to 16×16, MaxBTSize can be set to 64×64, MinBTSize (for both width and height) can be set to 4×4, and MaxBTDepth can be set to 4. Quadtree partitioning can first be applied to the CTU to generate quadtree leaf nodes. The quadtree leaf node can have a size of 16×16 (i.e., MinQTSize) to 128×128 (i.e., CTU size). If the leaf quadtree node is 128×128, since the size exceeds MaxBTSize (i.e., 64×64), the leaf quadtree node cannot be further split by a binary tree. Otherwise, the leaf quadtree node can be further split by a binary tree. The quadtree leaf node can also be the root node of a binary tree and it has a binary tree depth of 0. When the binary tree depth reaches MaxBTDepth (i.e., 4), no further splitting can be considered. When the binary tree node has a width equal to MinBTSize (i.e., 4), no further horizontal splitting can be considered. Similarly, when the binary tree node has a height equal to MinBTSize, no further vertical splitting can be considered. In the absence of any further splitting, the leaf nodes of the binary tree can be further processed by prediction and transform processing. In JEM, the maximum CTU size can be 256×256 luma samples.

[0186] Fig.19 Shown by using QTBT( Fig.19 ) and shows the corresponding tree representation ( Fig.19 ). Solid lines indicate quadtree splits, and dashed lines indicate binary tree splits. In each split (i.e., non-leaf) node of the binary tree, a flag is signaled to indicate which split type (i.e., horizontal or vertical) is used, where 0 indicates horizontal split and 1 indicates vertical split. For quadtree splits, there is no need to indicate the split type, because quadtree splits split the block horizontally and vertically to produce 4 sub-blocks of equal size.

[0187] In addition, the QTBT scheme supports the flexibility of having separate QTBT structures for luma and chroma. For example, for P slices and B slices, the luma CTB and chroma CTB in one CTU share the same QTBT structure. However, for I slices, the luma CTB is partitioned into CUs through the QTBT structure, while the chroma CTB is partitioned into chroma CUs through another QTBT structure. In this example, the CU in the I slice includes a coding block of the luma component or a coding block of two chroma components, and the CU in the P slice or B slice includes coding blocks of all three color components. In HEVC, inter prediction for small blocks may be restricted to reduce memory access for motion compensation, so that bidirectional prediction is not supported for 4×8 blocks and 8×4 blocks, and inter prediction is not supported for 4×4 blocks. In QTBT as implemented in JEM-7.0, these restrictions can be removed.

[0188] The block partition may have a ternary tree (TT) structure. Fig. 20 An example ternary tree segmentation is shown. In VVC, a multi-type tree (MTT) structure can add horizontal and vertical center-side ternary trees on top of the QTBT, such as Fig. 20 (a and b respectively). Ternary tree segmentation can complement quadtree and binary tree segmentation. Ternary tree segmentation can be able to capture objects located at the center of the block, while quadtree and binary trees are partitioned along the center of the block. In the case of ternary tree segmentation, the width and height of the proposed ternary tree segmentation can be a power of 2, so that no additional transformation is required. The design of the two-level tree can be to reduce the complexity, and the complexity of traversing the tree is T D , where T represents the number of split types and D is the depth of the tree.

[0189] Template Matching (TM)

[0190] Fig.21 An example of template matching (TM) on a search area around an initial motion vector (MV) is shown. Template matching can be a decoder-side MV derivation method for refining the motion information of a current coding unit (CU) or block by finding the closest match between a template in the current picture (i.e., the top and / or left neighboring blocks of the current CU) and a block in a reference (i.e., the same size as the template). The motion information may include spatial reference motion information (or spatial motion vectors) and / or temporal reference motion information (or temporal motion vectors). Fig.21 As shown, a better MV around the initial motion of the current CU is searched within the [-8, +8] pixel search range.

[0191] In AMVP mode, the MVP candidate with the minimum TM error between the current block template and the reference block template is selected at the encoder and decoder side. TM is performed on this MVP candidate for MV refinement. In merge mode, a similar search method can be applied to the merge candidate indicated by the merge index.

[0192] Decoder-Side Motion Vector Refinement (DMVR)

[0193] Fig. 22 An example of decoder-side motion vector refinement is shown. In order to increase the accuracy of the MV in merge mode, decoder-side motion vector refinement based on bilateral matching (BM) can be applied in VVC. In the bidirectional prediction operation, the refined MV is searched around the initial MV in the reference picture list L0 and the reference list L1. The BM method calculates the distortion between two candidate blocks in the reference list L0 and the list L1. Fig. 22 As shown, the SAD (Sum of Absolute Differences, SAD) between two blocks is calculated based on each MV candidate around the initial MV (MV 1'). The MV candidate with the smallest SAD becomes the refined MV and is used to generate a bidirectional prediction signal.

[0194] The refined MV derived by the DMVR process is used to generate inter-frame prediction samples for the current block and is also used for temporal motion vector prediction for future picture coding. The original MV is used for deblocking within the current frame and spatial motion vector prediction processing for future CU coding. In DMVR, the search point is around the initial MV, and the MV offset can comply with the MV difference mirror rule. In other words, any point represented by the candidate MV pair (MV0, MV1) checked by DMVR can comply with the following two equations:

[0195] MV0′=MV0+MV_offset (1)

[0196] MV1′=MV1-MV_offset (2)

[0197] Wherein, MV_offset represents the refinement offset between the initial MV (MV0, MV1) and the refined MV (MV0', MV1') in one of the reference pictures. The refinement search range is two integer luma samples starting from the initial MV. The search includes an integer sample offset search stage and a fractional sample refinement stage.

[0198] In one embodiment, a 25-point full search is applied for the integer sample offset search. The SAD of the initial MV pair is first calculated. If the SAD of the initial MV pair is less than a threshold, the integer sample stage of DMVR is terminated. Otherwise, the SAD of the remaining 24 points is calculated and checked in raster scan order. The point with the smallest SAD is selected as the output of the integer sample offset search stage. In order to reduce the loss of uncertainty in DMVR refinement, the original MV may be biased during the DMVR process. The SAD between the reference blocks referenced by the initial motion vector can be reduced by 1 / 4 of the SAD value.

[0199] The integer sample search may be followed by fractional sample refinement. To reduce computational complexity, fractional sample refinement is derived by using the parameter error surface equation instead of performing an additional search using SAD comparison. In the sub-pixel offset estimation based on the parameter error surface, a 2-D parabolic surface equation of the following form is fit using the cost at the center position and the costs at the four adjacent integer positions from the center:

[0200] E(x, y) = A(x min ) 2 +B(yy min ) 2 +C (3)

[0201] Among them, (x min ,y min ) corresponds to the coordinates of the fractional position with the minimum cost, and C corresponds to the minimum cost value. By solving the above equation using the cost values ​​of the five search points, (x min ,y min ) is calculated as:

[0202] x min =(E(-1,0)-E(1,0)) / (2(E(-1,0)+E(1,0)-2E(0,0))) (4)

[0203] y min =(E(0,-1)-E(0,1)) / (2((E(0,-1)+E(0,1)-2E(0,0))) (5)

[0204] Since all cost values ​​are positive and the minimum is E(0,0), x min and min The value of is limited to between -8 and 8. This corresponds to a half-pixel shift of 1 / 16 pixel MV precision in VVC. min ,y min ) is added to the integer distance refinement MV to obtain the final refinement delta MV with sub-pixel accuracy.

[0205] Reference frame coding in AV1

[0206] In AV1, for each coding block in an inter frame, if the mode of the current block is not skip mode but inter coding mode, a flag can be signaled to indicate whether single reference mode or composite reference mode is used for the current block. The prediction block is generated by one motion vector in single reference mode, while the prediction block is generated by weighted average of two prediction blocks derived from two motion vectors in composite reference mode. For the single reference case, a reference frame index with a value from 1 to 7 is signaled into the bitstream to indicate which reference frame is used for the current block. Reference frame indices 1 to 4 can be assigned for reference frames that precede the current frame in display order, while indices 5 to 7 are reference frames that follow the current frame in display order.

[0207] For the composite reference case, another flag may be signaled to indicate whether it is a unidirectional composite reference mode or a bidirectional composite reference mode. For the unidirectional composite reference mode, both reference frames are either before the current frame or after the current frame in display order. For the bidirectional composite reference mode, one reference frame is before the current frame and the other reference frame is after the current frame in display order. In addition, the allowed unidirectional reference frame combinations are limited to four possible pairs, namely (1, 2), (1, 3), (1, 4) and (5, 7), but all 12 combinations in the bidirectional case are supported. The basic principle behind this is that if the number of reference frames on both sides of the current frame is balanced in display order, then the bidirectional reference prediction may provide better prediction. When most reference frames are on one side of the current frame, the inference involving the nearest reference frame may be more relevant to the current frame.

[0208] Motion vector reference scheme in AV1 and CWG-B049

[0209] The bits used to signal motion vectors account for a significant portion of the total bitrate. Modern video codecs can use predictive coding for motion vectors and encode the difference using entropy coding. The prediction accuracy has a significant impact on coding efficiency. AV1 uses the following dynamic motion vector referencing scheme, which obtains candidate motion vectors from spatial and temporal neighbors and sorts them for efficient entropy coding. In one embodiment, using spatial motion vector referencing, the coding block will search its spatial neighbors in units of 8×8 luma samples to find a neighbor with the same reference frame index as the current block. For composite inter prediction mode, it can include the same reference frame pair. Fig.23 An example spatial motion vector search pattern is shown. Fig.23As shown, the search area includes three 8×8 block rows above the current block and three 8×8 block columns to the left of the current block, where the search order is indicated by the index. In CWG-B049, the number of 8×8 block rows above can be reduced from 3 to 1, while the number of left columns remains at 3.

[0210] In another embodiment, the temporal motion vector reference may first include a motion trajectory between the current frame and a previously encoded frame, the motion trajectory being constructed by using motion vectors from previously encoded frames by means of linear interpolation or extrapolation. Next, a motion field between the current frame and a given reference frame may be formed by extending the motion trajectory from the current frame toward the reference frame.

[0211] With the current design of the signaling method for reference frames, the order of reference frames in reference frame list 0 and reference frame list 1 is fixed for all blocks within a frame, regardless of the reference frames in neighboring blocks. Therefore, the spatial and / or temporal correlation of the reference frame selection is not exploited.

[0212] For composite reference mode, if the POC of both reference frames of a motion vector pair is greater than or less than the POC of the current frame, the directions of the two reference frames are the same. Otherwise, if the POC of one reference frame is greater than the POC of the current frame and the POC of the other reference frame is less than the POC of the current frame, the directions of the two reference frames are different.

[0213] The proposed embodiments may be used alone or in combination in any order. In addition, each of the method (or embodiment), encoder, and decoder may be implemented by a processing circuit system (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer-readable medium. In the following, the term block may be interpreted as a prediction block, a coding block, or a coding unit, i.e., a CU. The direction of the reference frame may be determined by whether the reference frame is before the current frame in display order or after the current frame in display order.

[0214] Fig.24A flowchart of a method according to an example implementation of the present disclosure is shown. For a single reference mode or a composite reference mode, the template matching (TM) method can be used to reorder the reference frames or reference frame pairs for each block. In block 2402, the template of the current block is compared with the template of the reference block. The comparison can be for the difference between the template of the current block (i.e., the top and / or left neighboring blocks) and the template of the reference block referenced by the motion information of the spatial reference motion information (or spatial motion vector) and / or the temporal reference motion information (or temporal motion vector). Based on the comparison, the difference between the templates is calculated in block 2404. The calculation can be based on the sum of absolute differences (SAD), the sum of squared differences (SSD), the mean squared error (MSE), or the sum of absolute transform differences (SATD). In one embodiment, for each spatial reference motion information and / or temporal reference motion information, a difference between a template of the current block and a template of a reference block is calculated and the difference is marked as a score value for the associated reference frame or reference frame pair, as in block 2406. A score is determined for each reference frame based on the calculated difference. All available reference frames are then sorted based on the score value in block 2408. In block 2410, the sorting can be used to reorder the frames.

[0215] Although described as a reference frame, it may also include a reference frame pair. In one embodiment, if multiple motion vectors point to a reference frame or a reference frame pair, the minimum difference value referenced by these motion vectors may be marked as a score value for a reference frame or a reference frame pair. In another embodiment, if no motion vector points to a reference frame or a reference frame pair, the score value of the reference frame or reference frame pair is marked as the maximum allowed value. In another embodiment, if the score values ​​of multiple reference frames or multiple reference frame pairs are the same, the sorting order of these reference frames or reference frame pairs is the same as the scanning order of the spatial reference motion information and / or the temporal reference motion information. In another embodiment, if the score values ​​of multiple reference frames or multiple reference frame pairs are the same, the sorting order of these reference frames or reference frame pairs depends on the frequency of occurrence of these reference frames or reference frame pairs used in the spatial reference motion information and / or the temporal reference motion information.

[0216] In one embodiment, all allowed unidirectional and bidirectional composite reference frame pairs are sorted together by using the TM method, and the index of the reference frame pair for the current block is signaled in the bitstream in the sorted order. In one embodiment, all allowed single reference frames are sorted together by using the TM method, and the index of the reference frame for the current block in the sorted order is signaled in the bitstream.

[0217] In one embodiment, a method for reordering reference frames for each block by template matching (TM) includes: comparing a template of a current block with a template of a reference block for motion information, wherein the motion information includes spatial reference motion information or temporal reference motion information; calculating a difference between the template of the current block and the template of the reference block; determining a score value of an associated reference frame based on the calculated difference; and reordering the reference frames based on the determined score. The reference frame also includes a reference frame pair. The TM includes a decoder-side motion vector derivation for refining the motion information of the current block. The reordering also includes ordering the available reference frames based on the score value of each reference frame in the reference frame. When the score values ​​of multiple reference frames are equal, the ordering corresponds to the scanning order of the spatial reference motion information or the temporal reference motion information. When the score values ​​of multiple reference frames are equal, the ordering is based on the frequency of occurrence of these reference frames used in the spatial reference motion information or the temporal reference motion information. The calculation includes at least one of the sum of absolute differences (SAD), the sum of squared differences (SSD), the mean square error (MSE), or the sum of absolute transform differences (SATD). The template includes the top or left neighboring block. The spatial reference motion information includes one or more spatial motion vectors. The temporal reference motion information includes one or more temporal motion vectors. When multiple motion vectors point to one of the reference frames, the one of the motion vectors with the smallest difference is used to determine the score value. When no motion vector points to one of the reference frames, the score value is determined to be the maximum allowed value. The allowed unidirectional and bidirectional composite reference frames are sorted together using TM, where the index of the reference frame for the current block is signaled in the bitstream. The allowed single reference frames are sorted together using TM, where the index of the reference frame for the current block is signaled in the bitstream.

[0218] The embodiments of the present disclosure may be used alone or in combination in any order. In addition, each of the method (or embodiment), encoder and decoder may be implemented by a processing circuit system (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer-readable medium. The term block may include a prediction block, a coding block, or a coding unit, i.e., a CU. The embodiments of the present disclosure may be applied to a luminance block or a chrominance block.

[0219] The techniques described above may be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. Fig.25 A computer system (2500) suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0220] Computer software may be encoded using any suitable machine code or computer language, which may be subjected to mechanisms such as assembly, compilation, and linking to create code comprising instructions that may be executed directly by one or more computer central processing units (CPU), graphics processing units (GPU), etc., or by interpretation, microcode execution, etc.

[0221] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, etc.

[0222] Fig.25 The components for the computer system (2500) shown in the example are exemplary in nature and are not intended to suggest any limitation on the scope of use or functionality of computer software implementing embodiments of the present disclosure. Nor should the configuration of components be interpreted as having any dependency or requirement relating to any one or combination of components shown in the exemplary embodiment of the computer system (2500).

[0223] The computer system (2500) may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input by one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, tapping), visual input (e.g., gestures), olfactory input (not depicted). Human-machine interface devices may also be used to capture certain media that are not necessarily directly related to human conscious input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0224] Input human interface devices may include one or more of the following (only one of each is depicted): keyboard (2501), mouse (2502), touchpad (2503), touch screen (2510), data gloves (not shown), joystick (2505), microphone (2506), scanner (2507) and camera (2508).

[0225] The computer system (2500) may also include certain human interface output devices. Such human interface output devices may stimulate one or more senses of a human user through, for example, tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include: tactile output devices (e.g., tactile feedback via a touch screen (2510), a data glove (not shown), or a joystick (2505), but there may also be tactile feedback devices that do not function as input devices); audio output devices (e.g., speakers (2509), headphones (not depicted)); visual output devices (e.g., screens (2510), including CRT (Cathode Ray Tube, CRT) screens, LCD (Liquid Crystal Display, LCD) screens, plasma screens, OLED (Organic Light Emitting Diode, OLED) screens, each with or without touch screen input capabilities, each with or without tactile feedback capabilities - some of which may be able to output two-dimensional visual output or more than three-dimensional output by means such as stereoscopic image output; virtual reality glasses (not depicted); holographic displays and cigarette cans (not depicted)); and printers (not depicted).

[0226] The computer system (2500) may also include storage devices and their associated media accessible to humans, such as optical media including CD / DVD ROM (Read Only Memory, ROM) / RW (2520) with CD / DVD and other media (2521), thumb drives (2522), removable hard drives or solid-state drives (2523), legacy magnetic media such as tapes and floppy disks (not depicted), dedicated ROM / ASIC (Application Specific Integrated Circuit, ASIC) / PLD (Programmable Logic Device, PLD) based devices such as security dongles (not depicted), etc.

[0227] Those skilled in the art should also understand that the term "computer-readable media" used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0228] The computer system (2500) may also include an interface (2554) to one or more communication networks (2555). The network may be, for example, wireless, wired, optical. The network may also be local, wide, metropolitan, vehicle-mounted and industrial, real-time, delay-tolerant, etc. Examples of networks include: local area networks such as Ethernet; wireless LAN (Local Area Network, LAN); cellular networks including GSM (Global System for Mobile Communications, GSM), 3G (the Third Generation, 3G), 4G (the Fourth Generation, 4G), 5G (the Fifth Generation, 5G), LTE (Long Term Evolution, LTE), etc.; TV wired or wireless wide area digital networks including cable TV, satellite TV and terrestrial broadcast TV; vehicle and industrial networks including CAN bus, etc. Some networks typically require an external network interface adapter attached to some general data port or peripheral bus (2549) (such as, for example, a USB port of the computer system (2500)); other networks are typically integrated into the core of the computer system (2500) by attaching to a system bus as described below (e.g., to an Ethernet interface in a PC computer system or to a cellular network interface in a smart phone computer system). Using any of these networks, the computer system (2500) can communicate with other entities. Such communications can be one-way receive only (e.g., broadcast TV), one-way send only (e.g., a CAN bus to some CAN bus devices), or two-way, for example, to other computer systems using a local area digital network or a wide area digital network. Certain protocols and protocol stacks can be used on each of these networks and network interfaces as described above.

[0229] The above-mentioned human-machine interface devices, human-accessible storage devices, and network interfaces may be attached to the core ( 2540 ) of the computer system ( 2500 ).

[0230] The core (2540) may include one or more central processing units (CPUs) (2541), graphics processing units (GPUs) (2542), dedicated programmable processing units in the form of field programmable gate areas (FPGAs) (2543), hardware accelerators for certain tasks (2544), graphics adapters (2550), etc. These devices, together with read-only memory (ROM), may be used to store and process data.

[0231] (2545), random access memory (2546), internal mass storage devices (2547) such as internal non-user accessible hard disk drives, SSDs (Solid State Drives, SSDs), etc. can be connected via a system bus (2548). In some computer systems, the system bus (2548) can be accessed in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, etc. Peripheral devices can be attached to the core's system bus (2548) directly or via a peripheral bus (2549). In an example, a screen (2510) can be connected to a graphics adapter (2550). The architecture of the peripheral bus includes PCI (Peripheral Component Interconnect / Interface, PCI), USB (Universal SerialBus, USB), etc.

[0232] The CPU (2541), GPU (2542), FPGA (2543) and accelerator (2544) can execute certain instructions, which can be combined to form the above-mentioned computer code. The computer code can be stored in ROM (2545) or RAM (Random Access Memory, RAM) (2546). Transient data can also be stored in RAM (2546), while permanent data can be stored in, for example, an internal large-capacity storage device (2547). Fast storage and retrieval of any storage device in the storage device can be achieved by using a cache memory, which can be closely associated with one or more CPUs (2541), GPUs (2542), large-capacity storage devices (2547), ROMs (2545), RAMs (2546), etc.

[0233] The computer readable medium may have computer code thereon for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or the medium and computer code may be of a type well known and available to those skilled in the art of computer software.

[0234] As a non-limiting example, a computer system (2500) having an architecture and in particular a core (2540) can provide functionality due to a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software contained in one or more tangible computer-readable media. Such computer-readable media can be media associated with a user-accessible mass storage device as introduced above, as well as certain storage devices of the core (2540) that are non-transitory, such as a core-internal mass storage device (2547) or ROM (2545). Software that implements various embodiments of the present disclosure can be stored in such a device and executed by the core (2540). Depending on specific needs, the computer-readable medium may include one or more memory devices or chips. The software can enable the core (2540) and in particular the processor therein (including a CPU, GPU, FPGA, etc.) to perform specific processing or specific parts of specific processing described herein, including limiting storage in RAM

[0235] (2546) and modifying such data structures according to processing defined by software. Additionally or alternatively, the computer system may provide functionality as a result of hard-wired logic or otherwise embodied in circuits (e.g., accelerators (2544)), which may replace or operate in conjunction with software to perform specific processing or specific portions of specific processing described herein. Where appropriate, references to software may include logic, and references to logic may also include software. Where appropriate, references to computer-readable media may include circuits (e.g., integrated circuits (ICs)) storing software for execution, circuits implementing logic for execution, or both. The present disclosure includes any suitable combination of hardware and software.

[0236] Although the present disclosure has described several exemplary embodiments, there are changes, permutations, and various alternative equivalents that fall within the scope of the present disclosure. It will therefore be appreciated that those skilled in the art will be able to conceive of many systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure.

[0237] Appendix A: Acronyms

[0238] ALF: Adaptive Loop Filter

[0239] AMVP: Advanced Motion Vector Prediction

[0240] APS: Adaptive Parameter Set

[0241] ASIC: Application-Specific Integrated Circuit

[0242] AV1: Alliance for Open Media Video 1

[0243] AV2: Alliance for Open Media Video 2

[0244] BCW: Bidirectional Prediction with CU-Level Weights

[0245] BM: Bilateral Matching

[0246] BMS: Benchmark Set

[0247] CANBus: Controller Area Network Bus

[0248] CC-ALF: Cross-Component Adaptive Loop Filter

[0249] CCSO: Cross Component Sample Offset

[0250] CD: Compact Disc

[0251] CDEF: Constrained Directional Enhancement Filter

[0252] CDF: Cumulative density function

[0253] CfL: Predicting Chroma from Luminance

[0254] CIIP: Combined Intra-Inter Prediction

[0255] CPU: Central Processing Unit

[0256] CRT: cathode ray tube

[0257] CTB: Coding Tree Block

[0258] CTU: Coding Tree Unit

[0259] CTU: Coding Tree Unit

[0260] CU: Coding Unit

[0261] DMVR: Decoder-side Motion Vector Refinement

[0262] DPB: Decoded Picture Buffer

[0263] DPS: Decoding Parameter Set

[0264] DVD: Digital Video Disc

[0265] FPGA: Field Programmable Gate Array

[0266] GBI: Generalized Bidirectional Prediction

[0267] GOP: Group of Pictures

[0268] GPU: Graphics Processing Unit

[0269] GSM: Global System for Mobile Communications

[0270] HDR: High Dynamic Range

[0271] HEVC: High Efficiency Video Coding

[0272] HRD: Hypothesized Reference Decoder

[0273] IBC (or IntraBC): Intra Block Copy

[0274] IC: Integrated Circuit

[0275] ISP: Intra-frame sub-partitioning

[0276] JEM: Joint Exploration Model

[0277] JVET: Joint Video Exploration Team

[0278] LAN: Local Area Network

[0279] LCD: Liquid Crystal Display

[0280] LR: Loop Restoration Filter

[0281] LSO: Local Sample Offset

[0282] LTE: Long Term Evolution

[0283] MMVD: Merge mode with motion vector difference

[0284] MPM: Most Probable Mode

[0285] MV: Motion Vector

[0286] MV: Motion Vector

[0287] MVD: Motion Vector Difference

[0288] MVD: Motion Vector Difference

[0289] MVP: Motion Vector Predictor

[0290] OLED: Organic Light Emitting Diode

[0291] PB: prediction block

[0292] PCI: Peripheral Component Interconnect

[0293] PDPC: Position Dependent Prediction Combination

[0294] PLD: Programmable Logic Device

[0295] POC: Image Sequential Counting

[0296] PPS: Picture Parameter Set

[0297] PU: prediction unit

[0298] PU: prediction unit

[0299] RAM: Random Access Memory

[0300] ROM: Read Only Memory

[0301] RPS: Reference Picture Set

[0302] SAD: Sum of Absolute Differences

[0303] SAO: Sample Adaptive Offset

[0304] SB: Super Block

[0305] SCC: Screen Content Coding

[0306] SDP: Semi-Decoupled Partitioning

[0307] SDR: Standard Dynamic Range

[0308] SDT: Semi-Decoupled Tree

[0309] SEI: Supplemental Enhancement Information

[0310] SNR: Signal to Noise Ratio

[0311] SPS: Sequence Parameter Set

[0312] SSD: Solid State Drive

[0313] SST: Semi-Separating Tree

[0314] TM: Template Matching

[0315] TU: Transform Unit

[0316] TU: Transform Unit

[0317] USB: Universal Serial Bus

[0318] VPS: Video Parameter Set

[0319] VUI: Video Availability Information

[0320] VVC: Versatile Video Coding

[0321] WAIP: Wide Angle Intra Prediction

Claims

1. A method for reordering reference frames for each block by template matching (TM), characterized in that The method comprises: Comparing a template of a current block with a template of a reference block for motion information, wherein the motion information comprises spatial reference motion information or temporal reference motion information, wherein the spatial reference motion information comprises one or more spatial motion vectors, and the temporal reference motion information comprises one or more temporal motion vectors; Calculating a difference between the template of the current block and the template of the reference block; determining a score value for the associated reference frame based on the calculated difference; and reordering the reference frames based on the determined scores; When a plurality of motion vectors point to one of the reference frames, a minimum difference value referenced by the plurality of motion vectors is marked as a fractional value of the one reference frame; the plurality of motion vectors include spatial motion vectors and / or temporal motion vectors.

2. The method according to claim 1, characterized in that The reference frames also include reference frame pairs.

3. The method according to claim 1, characterized in that The TM includes a decoder-side motion vector derivation for refining motion information of the current block.

4. The method according to any one of claims 1 to 3, characterized in that The reordering also includes: The available reference frames are sorted based on the score value of each of the reference frames.

5. The method according to claim 4, characterized in that When the score values ​​of the multiple reference frames are equal, the sorting corresponds to the scanning order of the spatial reference motion information or the temporal reference motion information.

6. The method according to claim 4, characterized in that When the score values ​​of a plurality of reference frames are equal, the ranking is based on the occurrence frequencies of these reference frames used in the spatial reference motion information or the temporal reference motion information.

7. The method according to any one of claims 1 to 3, characterized in that The calculation includes at least one of a sum of absolute differences (SAD), a sum of squared differences (SSD), a mean square error (MSE), or a sum of absolute transformed differences (SATD).

8. The method according to any one of claims 1 to 3, characterized in that The template includes a top or left adjacent block.

9. The method according to any one of claims 1 to 3, characterized in that When no motion vector points to one of the reference frames, the fractional value is determined to be a maximum allowed value.

10. The method according to any one of claims 1 to 3, characterized in that The allowed unidirectional composite reference frames and bidirectional composite reference frames are ordered together by using the TM, wherein the index of the reference frame for the current block is signaled in the bitstream.

11. The method according to any one of claims 1 to 3, characterized in that The allowed single reference frames are ordered together by using the TM, wherein the index of the reference frame for the current block is signaled in the bitstream.

12. A computer device, characterized in that: The computer device comprises: one or more computer-readable non-transitory storage media configured to store computer program code; and One or more computer processors configured to access the computer program code and execute the method according to any one of claims 1 to 11 as instructed by the computer program code.

13. An apparatus for decoding a video bit stream, characterized in that: The device comprises: a memory storing instructions; and A processor in communication with the memory, wherein when the processor executes the instructions, the processor is configured to cause the apparatus to perform the method according to any one of claims 1 to 11.

14. A non-transitory computer-readable storage medium storing instructions, characterized in that: When the instructions are executed by a processor, the instructions are configured to cause the processor to perform the method according to any one of claims 1 to 11.

Images