Method and apparatus for decoding a video stream, storage medium

Abstract

The present disclosure relates to using reference signaling to derive motion vector differences for encoding and / or decoding a video stream. There can be a syntax flag or value that indicates a particular inter coding mode. Based on the inter coding mode, a first motion vector of a current reference frame is used to derive a second motion vector that points to a different reference frame. The encoded video stream is decoded based on the derived second motion vector.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Application No. 18 / 049,914, filed October 26, 2022, entitled “DERIVED MOTION VECTOR WITH SINGLE REFERENCE SIGNALING”; U.S. Provisional Application No. 63 / 280,978, filed November 18, 2021, entitled “DERIVED MOTION VECTOR WITH SINGLE REFERENCE SIGNALING”; and U.S. Provisional Application No. 63 / 289,008, filed December 13, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure describes a set of advanced video encoding / decoding techniques. More specifically, the disclosed techniques relate to deriving motion vector differences using a single reference signaling for encoding and / or decoding, and more specifically to methods, apparatus, and storage media for decoding video streams. Background Technology

[0004] The background description provided herein is for the purpose of presenting the general content of this application. The extent of the work of the currently named inventors described in the background section and various aspects of this specification does not indicate that it was prior art at the time of filing of this application, nor is it expressly or implied that it was acknowledged as prior art of this application.

[0005] Video encoding and decoding can be performed using inter-frame image prediction with motion compensation. Uncompressed digital video can comprise a series of images, each with a spatial size of, for example, 1920×1080 luminance samples and associated full-sampled or subsampled chrominance samples. This series of images can have a fixed or variable image rate (alternatively referred to as the frame rate), for example, 60 images per second or 60 frames per second. Uncompressed video has specific bit rate requirements for streaming or data processing. For example, video with a pixel resolution of 1920×1080, a frame rate of 60 frames per second, and 4:2:0 chrominance subsampling with 8 bits per pixel per color channel requires bandwidth close to 1.5 Gbit / s. One hour of such video would require more than 600 GB of storage space.

[0006] One objective of video encoding and decoding is to reduce redundancy in the input video signal through compression. Compression can help reduce bandwidth or storage requirements by two or more orders of magnitude in some cases. Lossless compression, lossy compression, and combinations thereof can be employed. Lossless compression refers to a technique that reconstructs an exact copy of the original signal from the compressed original signal through a decoding process. Lossy compression refers to an encoding / decoding process in which the original video information is not fully preserved during encoding and is not fully recovered during decoding. When using lossy compression, the reconstructed signal may differ from the original signal, but despite the loss of some information, the distortion between the original and reconstructed signals is small enough that the reconstructed signal can be used for the intended application. In the case of video, lossy compression is widely used in many applications. The tolerable amount of distortion depends on the application; for example, users of some consumer video streaming applications may tolerate higher distortion than users of movie or television broadcasting applications. Compression ratios achievable through specific encoding algorithms can be selected or adjusted to reflect various distortion tolerances: higher tolerable distortion generally allows encoding algorithms to produce higher loss and higher compression ratios.

[0007] Video encoders and decoders can utilize techniques from a wide range of categories and steps, including, for example, motion compensation, Fourier transform, quantization, and entropy coding.

[0008] Video codec techniques can include techniques called intra-frame coding. In intra-frame coding, sample values ​​are represented without reference to samples or other data from a previously reconstructed reference image. In some video codecs, an image is spatially subdivided into sample blocks. When all sample blocks are encoded in intra-frame mode, the image can be an intra-frame image. Intra-frame images and their derivatives (such as standalone decoder refresh images) can be used to reset the decoder state and thus can be used as the first image in an encoded video stream and video session, or as a still image. Samples of blocks after intra-frame prediction can then be transformed into the frequency domain, and the transformed coefficients thus generated can be quantized before entropy coding. Intra-frame prediction represents a technique that minimizes the sample values ​​in the pre-transform domain. In some cases, the smaller the transformed DC value and the smaller the AC coefficients, the fewer bits are needed to represent the entropy-coded block for a given quantization step size.

[0009] Traditional intra-frame coding techniques, such as those known from MPEG-2 generation coding technologies, do not use intra-frame prediction. However, some newer video compression techniques attempt to encode / decode blocks based on, for example, surrounding sample data and / or metadata obtained during spatially adjacent encoding and / or decoding, preceding the data blocks encoded or decoded intra-frame in the decoding order. Such techniques are hereinafter referred to as "intra-frame prediction" techniques. It should be noted that, at least in some cases, intra-frame prediction uses only reference data from the current image being reconstructed, and not reference data from other reference images.

[0010] Intra-prediction can take many different forms. When more than one such technique can be used in a given video coding technique, the technique in use is called an intra-prediction mode. One or more intra-prediction modes can be provided in a particular codec. In some cases, a mode can have sub-modes and / or can be associated with various parameters, and the mode / sub-mode information and intra-coding parameters for the video block can be encoded separately or included in the mode codeword. Which codeword is used for a given mode, sub-mode, and / or parameter combination can affect the coding efficiency gain of intra-prediction, and therefore the entropy coding technique used to convert the codeword into a bitstream can also have an impact.

[0011] H.264 introduced a certain intra-frame prediction mode, which was improved in H.265 and further refined in new coding techniques such as the Joint Exploration Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). Typically, for intra-frame prediction, prediction blocks are formed using the values ​​of neighboring samples that have become available. For example, available values ​​from a specific set of neighboring samples can be copied into the prediction block along some directions and / or lines. The reference to the direction used can be encoded in the bitstream or can be predicted on itself.

[0012] refer to Figure 1A The lower right corner depicts a subset of nine prediction directions specified among the 33 possible prediction directions in H.265 (corresponding to 33 angular modes out of the 35 intra-frame modes specified in H.265). The point (101) where the arrow converges represents the sample being predicted. The arrow indicates that the direction of the sample is predicted using neighboring samples at 101. For example, arrow (102) indicates that sample (101) is predicted from one or more samples at an angle of 45 degrees to the upper right of the horizontal direction. Similarly, arrow (103) indicates that sample (101) is predicted from one or more samples at an angle of 22.5 degrees to the lower left of the horizontal direction.

[0013] Still referencing Figure 1A A 4×4 square block (104) containing samples is depicted in the upper left (represented by a bold dashed line). The square block (104) contains 16 samples, each labeled with "S" and its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (from the top) and the first sample in the X dimension (from the left). Similarly, sample S44 is the fourth sample in block (104) in both the Y and X dimensions. Since the block size is 4×4 samples, S44 is located in the lower right corner. Figure 1A Reference samples are also shown, which follow a similar numbering scheme. Reference samples are identified by R and their Y position (e.g., row index) and X position (column index) relative to block (104). In both H.264 and H.265, predicted samples adjacent to the block being reconstructed are used.

[0014] Intra-frame image prediction in block 104 can begin by copying reference sample values ​​from neighboring samples in a prediction direction indicated by a signal. For example, suppose the encoded video stream includes signaling indicating the prediction direction of arrow (102) for block 104, meaning the samples are predicted from one or more prediction samples at a 45-degree angle to the upper right of the horizontal direction. In this case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Then, sample S44 is predicted based on reference sample R08.

[0015] In some cases, the values ​​of multiple reference samples can be combined, for example, by interpolation, in order to calculate the reference sample, especially when the direction is not divisible by 45 degrees.

[0016] With the development of video coding techniques, the number of possible directions has increased. In H.264 (2003), nine different directions were available for intra-frame prediction. This number increased to 33 in H.265 (2013), and at the time of this disclosure, JEM / VVC / BMS supports up to 65 directions. Experiments have been conducted to identify the most suitable intra-frame prediction directions, and certain techniques in entropy coding have been used to encode those most suitable directions with a small number of bits, while for some directions, a certain bit cost is accepted. Furthermore, sometimes the direction itself can be predicted based on the adjacent directions used in the intra-frame prediction of already decoded adjacent blocks.

[0017] Figure 1B This is a schematic diagram (180) showing 65 intra-frame prediction directions according to JEM, thus illustrating the increase in the number of prediction directions in various coding techniques developed over time.

[0018] The mapping of intra-prediction direction bits representing direction in the encoded video bitstream can vary depending on the video coding technique. For example, the mapping can range from a simple direct mapping of the prediction direction to the intra-prediction mode, to a codeword, to a complex adaptive scheme involving the most probable mode, and similar techniques. However, in all cases, for intra-prediction, there may be some directions that are statistically less likely to occur in the video content compared to others. Since the goal of video compression is to reduce redundancy, in a well-designed video coding technique, those less likely directions will be represented by a greater number of bits than the likely directions.

[0019] Inter-frame image prediction, or inter-prediction, can be based on motion compensation. In motion compensation, sample data from a previously reconstructed image or a portion thereof (the reference image) is used to predict a newly reconstructed image or a portion of an image (e.g., a patch) after being spatially offset along a direction indicated by a motion vector (hereafter referred to as MV). In some cases, the reference image can be the same as the image currently being reconstructed. MV can have two dimensions, X and Y, or three dimensions, with the third dimension indicating the reference image being used (the latter being similar to the time dimension).

[0020] In some video compression techniques, the MV applicable to a region of the sample data can be predicted based on other MVs, such as those spatially adjacent to the region being reconstructed and decoded before the current MV. This can significantly reduce the amount of data required to encode the MV by eliminating redundancy in related MVs, thereby increasing the compression ratio. MV prediction can work effectively, for example, because when encoding an input video signal (called natural video) from a camera, there is a statistical probability that a larger region than the region applicable to a single MV moves in a similar direction in the video sequence. Therefore, in some cases, the larger region can be predicted using similar motion vectors derived from the MVs of neighboring regions. This makes the MV of a given region similar or identical to the MV predicted based on the surrounding MVs, and thus, after entropy coding, the MV of that given region can be represented with fewer bits than when directly encoding the MV (rather than predicting the MV based on neighboring MVs). In some cases, MV prediction can be an example of lossless compression of the signal (i.e., the MV) derived from the original signal (i.e., the sample stream). In other cases, MV prediction itself can be lossy, for example, due to rounding errors when calculating predictions based on multiple surrounding MVs.

[0021] H.265 / HEVC (ITU-T H.265 Recommendation, “High Efficiency Video Coding”, December 2016) describes various MV prediction mechanisms. Among the various MV prediction mechanisms provided by H.265, this application describes the technique referred to below as “spatial combining”.

[0022] For details, please refer to Figure 2 The current block (201) includes samples that have been discovered by the encoder during the motion search process and can be predicted based on previous blocks of the same size that have generated spatial offsets. The MV can be derived from metadata associated with one or more reference images, rather than being directly encoded. For example, using the MV associated with any of the five surrounding samples A0, A1 and B0, B1, B2 (corresponding to 202 to 206 respectively), the MV is derived from the metadata of the nearest reference image (in decoding order). In H.265, MV prediction can use predictions from the same reference image being used by adjacent blocks.

[0023] AOMedia Video 1 (AV1) is an open video coding format designed specifically for video transmission over the Internet. It was developed as the successor to VP9, ​​building upon the VP9 codebase and incorporating additional technologies. The AV1 bitstream specification includes reference video codecs such as H.265, High Efficiency Video Coding (HEVC), or Universal Video Coding (VVC). Summary of the Invention

[0024] Embodiments of this disclosure provide a method and apparatus for deriving motion vector differences using a single reference signaling for decoding a video stream. Syntax flags / values ​​indicating a specific inter-frame coding mode may exist. Based on the inter-frame coding mode, a second motion vector pointing to a different reference frame is derived using a first motion vector of the current reference frame. The encoded video stream is decoded based on the derived second motion vector.

[0025] In one embodiment, a method for decoding a video stream includes: receiving the video stream, the video stream including a syntax flag indicating whether a first motion vector of a first reference frame is used to derive a second motion vector of a second reference frame / different reference frame; receiving the first reference frame of a current frame in the video stream; determining, based on the value of the syntax flag, whether the first motion vector of the first reference frame is used to derive the second motion vector of the second reference frame; in response to the value of the syntax flag indicating that the first motion vector of the current reference frame is used to derive the second motion vector of the second reference frame, deriving the second motion vector of the second reference frame, wherein the derivation is based on the value of the syntax flag and the first motion vector of the first reference frame; and decoding the video stream based on the derived second motion vector. The first motion vector points to the first reference frame, the second motion vector points to a second reference frame different from the first reference frame, and motion vector difference (MVD) provides a prediction based on the first motion vector and the second motion vector. The derivation includes inter-frame coding modes, wherein the inter-frame coding modes include either a DERIVED_NEWMV mode or a DERIVED_NEARMV mode. The DERIVED_NEWMV mode involves directly deriving the second motion vector from the first reference frame. The DERIVED_NEARMV mode involves deriving the second motion vector from adjacent frames of the first reference frame. The method includes, in response to the syntax flag, indicating which inter-frame coding mode is applied, wherein the second motion vector is derived from the first motion vector when it indicates that the first motion vector of the individual reference frame can derive the second motion vector. The method includes using signaled motion vector difference (MVD) based on the indicated inter-frame coding mode. The method includes using a signaled reference frame index, the reference frame index indicating whether the current block uses the first or second reference frame for the inter-frame prediction mode. The reference frame index is sent using a signal when there are at least two reference frames available for the current frame in the reference frame buffer, and one of the at least two reference frames precedes the current frame in display order, while the other of the at least two reference frames follows the current frame in display order. The orientation of the reference frame is based on whether the reference frame precedes or follows the current frame in display order. The orientation of a single reference frame differs from the orientation of the second reference frame. The orientations of the two reference frames are the same when the Image Order Count (POC) of both reference frames in a motion vector pair is greater than or less than the POC of the current frame.The pair of reference frames are considered to have different orientations when the Image Order Count (POC) of one reference frame in a pair is greater than the POC of the current frame, and the POC of the other reference frame in the pair is less than the POC of the current frame. The direction of the derived second motion vector pointing to these different reference frames differs from the direction of the reference frame transmitted using the signal or the direction of the single reference frame.

[0026] In another embodiment, an apparatus for decoding a video stream includes: a memory for storing instructions; and a processor communicating with the memory, wherein, when the processor executes the instructions, the processor is configured to cause the apparatus to: process the video stream, the video stream including syntax flags indicating an inter-frame coding mode; identify a first reference frame from the video stream, wherein a first motion vector points to the first reference frame; determine, based on the value of the syntax flags, whether the first motion vector of the first reference frame is used to derive a second motion vector of a second reference frame; in response to determining that the first motion vector of the first reference frame is used to derive the second motion vector of the second reference frame, derive the second motion vector based on the first motion vector; and decode the video stream based on the derived second motion vector. The inter-frame coding modes include either the DERIVED_NEWMV mode or the DERIVED_NEARMV mode. The DERIVED_NEWMV mode derives the second motion vector from the first reference frame, and the DERIVED_NEARMV mode derives the second motion vector from adjacent frames of the first reference frame. The first motion vector points to the first reference frame, the second motion vector points to the second reference frame, and the motion vector difference (MVD) provides a prediction based on the first and second motion vectors.

[0027] In another embodiment, a non-transitory computer-readable storage medium stores instructions thereon, wherein, when executed by a processor, the instructions are configured to cause the processor to: process a video stream, the video stream including a syntax indicator, the syntax indicator including an inter-frame coding mode; receive a first reference frame of a current frame from the video stream, wherein a first motion vector points to the first reference frame; determine, based on the value of the syntax indicator, whether the first motion vector of the first reference frame is used to derive a second motion vector of a second reference frame; in response to determining that the first motion vector of the first reference frame is used to derive the second motion vector of the second reference frame, derive the second motion vector based on the syntax indicator and based on the first motion vector, wherein the second motion vector points to the second reference frame; and decode the video stream based on the derived second motion vector. The inter-frame coding modes include either the DERIVED_NEWMV mode or the DERIVED_NEARMV mode, wherein the DERIVED_NEWMV mode includes deriving the second motion vector from the first reference frame, and the DERIVED_NEARMV mode includes deriving the second motion vector from adjacent frames of the first reference frame.

[0028] In some other embodiments, a device for processing video information is disclosed. The device may include circuitry configured to perform any of the embodiments described above.

[0029] Embodiments of this disclosure also provide a non-transitory computer-readable medium having instructions stored thereon that, when executed by a computer for video decoding and / or encoding, cause the computer to perform video decoding and / or encoding. Attached Figure Description

[0030] Other features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which:

[0031] Figure 1A A schematic diagram of an exemplary subset of intra-frame prediction direction patterns is shown.

[0032] Figure 1B An illustration of an exemplary intra-frame prediction direction is shown.

[0033] Figure 2 A schematic diagram of the current block and its surrounding spatial merging candidate blocks is shown in an example for motion vector prediction.

[0034] Figure 3 A simplified block diagram of a communication system (300) according to an exemplary embodiment is shown.

[0035] Figure 4 A simplified block diagram of a communication system (400) according to an exemplary embodiment is shown.

[0036] Figure 5 A simplified block diagram of a video decoder according to an exemplary embodiment is shown.

[0037] Figure 6 A simplified block diagram of a video encoder according to an exemplary embodiment is shown.

[0038] Figure 7 A block diagram of a video encoder according to another exemplary embodiment is shown.

[0039] Figure 8 A block diagram of a video decoder according to another exemplary embodiment is shown.

[0040] Figure 9 A scheme for dividing coded blocks according to an exemplary embodiment of the present disclosure is shown.

[0041] Figure 10 Another scheme for coded block partitioning according to exemplary embodiments of the present disclosure is shown.

[0042] Figure 11 Another scheme for coded block partitioning according to exemplary embodiments of the present disclosure is shown.

[0043] Figure 12 An example of dividing a basic block into coded blocks according to an exemplary partitioning scheme is shown.

[0044] Figure 13 An exemplary ternary partitioning scheme is shown.

[0045] Figure 14 An exemplary quadtree binary tree (QTBT) coding block partitioning scheme is shown.

[0046] Figure 15 A scheme for dividing a coded block into multiple transform blocks and the encoding order of these transform blocks, according to an exemplary embodiment of the present disclosure, is shown.

[0047] Figure 16 Another scheme for dividing a coded block into multiple transform blocks and the encoding order of these transform blocks, according to exemplary embodiments of the present disclosure, is shown.

[0048] Figure 17 Another scheme for dividing a coded block into multiple transform blocks according to an exemplary embodiment of the present disclosure is shown.

[0049] Figure 18 An exemplary partitioning tree for block partitioning is shown.

[0050] Figure 19 An exemplary ternary tree partition is shown.

[0051] Figure 20a An example of a merged pattern (MMVD) search point with motion vector difference is shown.

[0052] Figure 20b Another example of a merged pattern (MMVD) search point with motion vector difference is shown.

[0053] Figure 21 A flowchart of a method according to an exemplary embodiment of the present disclosure is shown.

[0054] Figure 22 A schematic diagram of a computer system according to an exemplary embodiment of the present disclosure is shown. Detailed Implementation

[0055] Throughout the specification and claims, terms may have subtle meanings implied or implied in the context that go beyond their expressly stated meanings. The phrases “in one embodiment” or “in some embodiments” as used herein do not necessarily refer to the same embodiment, and the phrases “in another embodiment” or “in other embodiments” as used herein do not necessarily refer to different embodiments. Similarly, the phrases “in one implementation” or “in some implementations” as used herein do not necessarily refer to the same implementation, and the phrases “in another implementation” or “in other embodiments” as used herein do not necessarily refer to different implementations. For example, the claimed subject matter includes combinations of all or some exemplary embodiments / implementations.

[0056] Generally, terms can be understood, at least in part, from their usage in the context. For example, terms such as “and,” “or,” or “and / or” as used herein can include a variety of meanings, which can depend at least in part on the context in which they are used. Typically, “or,” when used to relate a list such as A, B, or C, is intended to mean A, B, and C (used here in an inclusive sense) and A, B, or C (used here in an exclusive sense). Furthermore, the terms “one or more” or “at least one,” as used herein, can be used, at least in part, to describe any feature, structure, or characteristic in a singular sense, or to describe a combination of features, structures, or characteristics in a plural sense. Similarly, terms such as “a (a / an)” or “described” can also be understood to convey either a singular or a plural usage, at least in part on the context. Moreover, the words “based on” or “determined by” can be understood to not necessarily convey a set of exclusive factors, but may allow for the presence of other factors that are not necessarily explicitly described, which also depends at least in part on the context.

[0057] Figure 3 A simplified block diagram of a communication system (300) according to one embodiment of the present disclosure is shown. The communication system (300) includes a plurality of terminal devices that can communicate with each other via, for example, a network (350). For example, the communication system (300) includes terminal devices (310) and (320) interconnected via a network (350). Figure 3 In this embodiment, terminal device (310) and terminal device (320) perform unidirectional data transmission. For example, terminal device (310) may encode video data (e.g., a video image stream acquired by terminal device (310)) for transmission over a network (350) to a second terminal device (320). The encoded video data is transmitted in the form of one or more encoded video streams. Terminal device (320) may receive the encoded video data from the network (350), decode the encoded video data to recover the video data, and display video images based on the recovered video data. Unidirectional data transmission can be implemented in applications such as media services.

[0058] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) performing bidirectional transmission of encoded video data, which may occur, for example, during a video conference. For bidirectional data transmission, in one example, each of the terminal devices (330) and (340) may encode video data (e.g., a video image stream captured by the terminal device) for transmission over a network (350) to the other terminal device (330) and (340). Each of the terminal devices (330) and (340) may also receive encoded video data transmitted by the other terminal device (330) and (340), and may decode the encoded video data to recover the video data, and may display the video image on an accessible display based on the recovered video data.

[0059] exist Figure 3In the embodiments disclosed herein, terminal devices (310), (320), (330), and (340) may be implemented as servers, personal computers, and smartphones, but the applicability of the basic principles of this disclosure is not limited thereto. The embodiments disclosed herein may be implemented on desktop computers, laptop computers, tablet computers, media players, wearable computers, dedicated video conferencing equipment, and / or similar devices. Network (350) refers to any number or type of network that transmits encoded video data between terminal devices (310), (320), (330), and (340), including, for example, wired (connected) and / or wireless communication networks. Communication networks (350) may exchange data in circuit-switched channels, packet-switched channels, and / or other types of channels. Representative networks may include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, unless explained below, the architecture and topology of the network (350) may be irrelevant to the operation of this disclosure.

[0060] As an example of the application to the disclosed topic, Figure 4 The placement of a video encoder and a video decoder in a streaming environment is illustrated. The subject matter disclosed in this application is equally applicable to other video applications, including, for example, video conferencing, digital television (TV) broadcasting, gaming, virtual reality, and storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0061] The video streaming system may include a video acquisition subsystem (413) that may include a video source (401) such as a digital camera, which creates a stream (402) of uncompressed video images or pictures. In one example, the video image stream (402) includes samples recorded by a digital camera. The video image stream (402) is depicted as a thick line to emphasize the high data volume of the video image stream compared to encoded video data (404) (or encoded video bitstream), and the video image stream (402) may be processed by an electronic device (420) including a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination of hardware and software to implement or enforce aspects of the disclosed subject matter as described in more detail below. Compared to the uncompressed video image stream (402), the encoded video data (404) (or the encoded video bitstream (404)) depicted as thin lines to emphasize its lower data volume can be stored on a streaming server (405) for future use or directly stored to downstream video devices (not shown). One or more streaming client subsystems, such as Figure 4Client subsystems (406) and (408) can access a streaming server (405) to retrieve copies (407) and (409) of encoded video data (404). Client subsystem (406) may include, for example, a video decoder (410) in an electronic device (430). The video decoder (410) decodes the incoming copy (407) of the encoded video data and produces an uncompressed output video image stream (411) that can be displayed on a display (412) (e.g., a screen) or other display device (not depicted). The video decoder 410 may be configured to perform some or all of the various functions described in this disclosure. In some streaming systems, the encoded video data (404), video data (407), and video data (409) (e.g., video streams) may be encoded according to certain video encoding / compression standards. Embodiments of these standards include ITU-T Recommendation H.265. In one example, the video coding standard under development is informally referred to as next-generation video coding (VVC), and the topics disclosed can be used in the context of VVC and other video coding standards.

[0062] It should be noted that the electronic devices (420) and (430) may include other components (not shown). For example, the electronic device (420) may include a video decoder (not shown), and the electronic device (430) may also include a video encoder (not shown).

[0063] Figure 5 A block diagram of a video decoder (510) according to any embodiment of the present disclosure is shown. The video decoder (510) may be disposed in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., receiving circuitry). The video decoder (510) may be used in place of Figure 4 The video decoder (410) in the embodiment.

[0064] The receiver (531) may receive one or more encoded video sequences to be decoded by the video decoder (510). In the same embodiment or another embodiment, one encoded video sequence may be decoded at a time, wherein the decoding of each encoded video sequence is independent of other encoded video sequences. Each video sequence may be associated with multiple video frames or pictures. Encoded video sequences may be received from a channel (501), which may be a hardware / software link leading to a storage device storing the encoded video data or a streaming source transmitting the encoded video data. The receiver (531) may receive encoded video data as well as other data, such as encoded audio data and / or auxiliary data streams that may be forwarded to their respective user entities (not indicated). The receiver (531) may separate the encoded video sequences from other data. To prevent network jitter, a buffer (515) may be coupled between the receiver (531) and the entropy decoder / parser (520) (hereinafter referred to as "parser (520)"). In some applications, the buffer (515) is part of the video decoder (510). In other applications, the buffer memory (515) may be located externally to and separate from the video decoder (510) (not shown). In other applications, the buffer memory (not shown) may be located externally to the video decoder (510) to, for example, prevent network jitter, and another additional buffer memory (515) may be configured internally to, for example, handle broadcast timing. When the receiver (531) receives data from a store / forward device with sufficient bandwidth and controllability or from an isochronous synchronization network, the buffer memory (515) may not be required, or it may be made smaller. For use on packet networks such as the Internet, a buffer memory (515) of sufficient size may also be required, which may be relatively large. Such a buffer memory may have an adaptive size and may be implemented at least partially in an operating system or a similar component (not shown) external to the video decoder (510).

[0065] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the encoded video sequence. These symbols may or may not be part of the electronic device (530), but may be coupled to the electronic device (530), such as... Figure 5As shown in the diagram. The control information for the display device may be a parameter set fragment (not indicated) of Supplemental Enhancement Information (SEI message) or Video Usability Information (VUI). The parser (520) may parse / entropy decode the received encoded video sequence. The entropy coding of the encoded video sequence may be performed according to video coding techniques or standards and may follow various principles, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (520) may extract a subgroup parameter set of at least one subgroup of pixels in the subgroup from the encoded video sequence for use in the video decoder based on at least one parameter corresponding to a subgroup. The subgroup may include a Group of Pictures (GOP), image, tile, slice, macroblock, coding unit (CU), block, transform unit (TU), prediction unit (PU), etc. The parser (520) can also extract information from the encoded video sequence, such as transform coefficients (e.g., Fourier transform coefficients), quantizer parameter values, motion vectors, etc.

[0066] The parser (520) can perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515) to create symbols (521).

[0067] Depending on the type of encoded video image or a portion thereof (e.g., inter-frame and intra-frame images, inter-frame blocks and intra-frame blocks) and other factors, the reconstruction of the symbol (521) may involve multiple different processing or functional units. Which units are involved and how they are involved can be controlled by the subgroup control information parsed from the encoded video sequence by the parser (520). For the sake of brevity, the flow of such subgroup control information between the parser (520) and the various processing or functional units described below is not described.

[0068] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into several functional units as described below. In practical embodiments operating under commercial constraints, many of these functional units interact closely with each other and can be integrated with one another. However, for the purpose of clearly describing the various functions of the disclosed subject matter, the present disclosure is hereinafter adopted as a conceptual subdivision into multiple functional units.

[0069] The first unit may include a scaler / inverse transform unit (551). The scaler / inverse transform unit (551) may receive quantization transform coefficients as symbols (521) from the parser (520) and control information, including information indicating which inverse transform to use, block size, quantization factor / parameter, quantization scaling matrix, etc. The scaler / inverse transform unit (551) may output a block including sample values, which may be input into the aggregator (555).

[0070] In some cases, the output samples of the scaler / inverse transform unit (551) may belong to intra-coded blocks; that is, blocks that do not use predictive information from previously reconstructed images but can use predictive information from previously reconstructed portions of the current image. Such predictive information may be provided by the intra-image prediction unit (552). In some cases, the intra-image prediction unit (552) may use information from surrounding blocks that have been reconstructed and stored in the current image buffer (558) to generate blocks of the same size and shape as the blocks being reconstructed. For example, the current image buffer (558) buffers partially reconstructed current images and / or fully reconstructed current images. In some implementations, the aggregator (555) may add the predictive information generated by the intra-prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) based on each sample.

[0071] In other cases, the output samples of the scaler / inverse transform unit (551) may belong to inter-frame coding and latent motion compensation blocks. In this case, the motion compensation prediction unit (553) may access the reference image memory (557) to extract samples for inter-frame image prediction. After motion compensation is performed on the extracted samples according to the symbols (521) belonging to the blocks, these samples may be added by the aggregator (555) to the output of the scaler / inverse transform unit (551) (the output of unit 551 may be referred to as residual samples or residual signals), thereby generating output sample information. The motion compensation prediction unit (553) may obtain the prediction samples from the address in the reference image memory (557) under the control of a motion vector, and the motion vector is available to the motion compensation prediction unit (553) in the form of the symbol (521), which may have, for example, X, Y components (offset) and a reference image component (time). Motion compensation may also include interpolation of sample values ​​extracted from a reference image memory (557) when using subsampled precise motion vectors, and may also be associated with motion vector prediction mechanisms, etc.

[0072] The output samples of the aggregator (555) can be employed by various loop filtering techniques in the loop filter unit (556). Video compression techniques may include in-loop filtering techniques controlled by parameters included in the encoded video sequence (also referred to as the encoded video stream), and these parameters can be used as symbols (521) from the parser (520) in the loop filter unit (556). However, video compression techniques may also respond to metadata obtained during decoding of a previous (in decoding order) portion of the encoded image or encoded video sequence, and to previously reconstructed and loop-filtered sample values. Various types of loop filters may be included as part of the loop filter unit 556 in various orders, which will be described in further detail below.

[0073] The output of the loop filter unit (556) can be a sample stream, which can be output to a display device (512) and stored in a reference image memory (557) for subsequent inter-frame image prediction.

[0074] Once fully reconstructed, some of the encoded images can be used as reference images for future inter-frame image prediction. For example, once the encoded image corresponding to the current image has been fully reconstructed and the encoded image (by, for example, the parser (520)) is identified as the reference image, the current image buffer (558) can become part of the reference image memory (557), and a new current image buffer can be reallocated before the reconstruction of subsequent encoded images begins.

[0075] The video decoder (510) can perform decoding operations according to a predetermined video compression technique adopted in, for example, the ITU-T H.265 Recommendation. The encoded video sequence may conform to the syntax specified by the video compression technique or standard in the sense that the encoded video sequence follows the syntax of the video compression technique or standard and the configuration file recorded in the video compression technique or standard. Specifically, the configuration file may select certain tools from all available tools in the video compression technique or standard as the only tools available under that configuration file. For standard compliance, the complexity of the encoded video sequence is also required to be within the range defined by the hierarchy of the video compression technique or standard. In some cases, the hierarchy limits the maximum image size, maximum frame rate, maximum reconstruction sampling rate (measured in, for example, megasamples per second), maximum reference image size, etc. In some cases, the limitations set by the hierarchy can be further defined by the Hypothetical Reference Decoder (HRD) specification and the metadata managed by the HRD buffer, which is represented as a signal in the encoded video sequence.

[0076] In some exemplary embodiments, the receiver (531) may receive supplemental (redundant) data along with the encoded video. This supplemental data may be a portion of the encoded video sequence. The supplemental data may be used by the video decoder (510) to properly decode the data and / or more accurately reconstruct the original video data. The supplemental data may take the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0077] Figure 6 This is a block diagram of a video encoder (603) according to an exemplary embodiment of the present disclosure. The video encoder (603) is disposed in an electronic device (620). The electronic device (620) may further include a transmitter (640) (e.g., transmission circuitry). The video encoder (603) can be used in place of... Figure 4 The example video encoder (403).

[0078] The video encoder (603) can obtain data from the video source (601) (not...). Figure 6 In one embodiment, a portion of the electronic device (620) receives video samples, the video source being capable of capturing video images to be encoded by a video encoder (603). In another embodiment, the video source (601) is a portion of the electronic device (620).

[0079] A video source (601) can provide a sequence of source videos in the form of a digital video sample stream to be encoded by a video encoder (603). This digital video sample stream can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit…), any color space (e.g., BT.601 YCrCB, RGB…), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, the video source (601) can be a storage device storing previously prepared video. In a video conferencing system, the video source (601) can be a camera that captures local image information as a video sequence. Video data can be provided as multiple individual images or pictures, which are given motion when viewed sequentially. The images themselves can be constructed as spatial pixel arrays, where each pixel can include one or more samples, depending on the sampling structure, color space, etc., used. Those skilled in the art can readily understand the relationship between pixels and samples. The following focuses on describing samples.

[0080] According to some exemplary embodiments, a video encoder (603) can encode and compress images of a source video sequence into an encoded video sequence (643) in real time or under any other time constraints required by the application. Implementing an appropriate encoding rate is a function of the controller (650). In some embodiments, the controller (650) controls and is functionally coupled to other functional units described below. For simplicity, coupling is not shown in the figures. Parameters set by the controller (650) may include rate control-related parameters (image skipping, quantizer, λ value of rate-distortion optimization techniques, etc.), image size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) may be used with other suitable functions related to a video encoder (603) optimized for a particular system design.

[0081] In some exemplary embodiments, the video encoder (603) may be configured to operate within an encoding loop. As a simplified description, in one example, the encoding loop may include a source encoder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on the input image to be encoded and a reference image) and a (local) decoder (633) embedded within the video encoder (603). Even if the embedded decoder 633 processes the video stream encoded by the source encoder 630 without entropy encoding, the decoder (633) reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder creates sample data (because in the video compression techniques considered in this application, any compression between the symbols and the encoded video stream is lossless). The reconstructed sample stream (sample data) is input to a reference image memory (634). Since the decoding of the symbol stream produces bit-accurate results regardless of the decoder's location (local or remote), the contents of the reference image memory (634) are also bit-accurately corresponding between the local encoder and the remote encoder. In other words, the reference image samples "seen" by the encoder's prediction portion are exactly the same sample values ​​that the decoder will "see" when using the prediction during decoding. This fundamental principle of reference image synchronization (and the drift that occurs when synchronization cannot be maintained, for example, due to channel errors) is used to improve coding quality.

[0082] The operation of the “local” decoder (633) can be combined with, for example, the above-mentioned Figure 5 The video decoder (510) is described in detail as the same as the "remote" decoder. However, a further brief reference is provided. Figure 5Since symbols are available and the entropy encoder (645) and parser (520) are able to encode / decode symbols into an encoded video sequence without loss, the entropy decoding portion of the video decoder (510), including the buffer (515) and parser (520), may not be fully implemented in the local decoder (633) or the encoder.

[0083] It can be observed that any decoder technique other than parsing / entropy decoding present in the decoder must also exist in the corresponding encoder in essentially the same functional form. For this reason, the disclosed topics sometimes focus on decoder operations associated with the decoding part of the encoder. The description of encoder techniques can be simplified because encoder techniques are inverses of the fully described decoder techniques. Only certain areas or aspects of the encoder are described in more detail below.

[0084] During operation, in some exemplary implementations, the source encoder (630) may perform motion-compensated predictive coding. This motion-compensated predictive coding predictively encodes the input image, referencing one or more previously encoded images designated as "reference images" from the video sequence. In this manner, the encoding engine (632) encodes the differences in color channels between pixel blocks of the input image and pixel blocks of the reference image, which may be selected as a predictive reference for the input image. The terms "residue" and its adjective form "residual" are used interchangeably.

[0085] The local video decoder (633) can decode encoded video data of an image that can be designated as a reference image, based on symbols created by the source encoder (630). The operation of the encoding engine (632) can be a lossy process. When the encoded video data can be decoded by the video decoder (633), Figure 6 When decoded (not shown), the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder (633) replicates the decoding process, which can be performed on the reference image by the video decoder, and allows the reconstructed reference image to be stored in a reference image cache (634). In this way, the video encoder (603) can locally store a copy of the reconstructed reference image that shares common content (no transmission errors) with the reconstructed reference image that will be obtained by the remote video decoder.

[0086] The predictor (635) can perform a prediction search against the encoding engine (632). That is, for a new image to be encoded, the predictor (635) can search in the reference image memory (634) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., that can serve as appropriate prediction references for the new image. The predictor (635) can operate pixel-by-pixel based on the sample blocks to find suitable prediction references. In some cases, based on the search results obtained by the predictor (635), it can be determined that the input image can have prediction references obtained from multiple reference images stored in the reference image memory (634).

[0087] The controller (650) can manage the encoding operations of the source encoder (630), including, for example, setting parameters and subgroup parameters for encoding video data.

[0088] The outputs of all the above functional units can be entropy encoded in the entropy encoder (645). The entropy encoder (645) performs lossless compression on the symbols generated by various functional units according to techniques such as Huffman coding, variable length coding, and arithmetic coding, thereby converting the symbols generated by various functional units into an encoded video sequence.

[0089] The transmitter (640) can buffer the encoded video sequence created by the entropy encoder (645) in preparation for transmission via a communication channel (660), which may be a hardware / software link to a storage device that will store the encoded video data. The transmitter (640) can combine the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).

[0090] The controller (650) manages the operation of the video encoder (603). During encoding, the controller (650) can assign a specific encoded image type to each encoded image, but this may affect the encoding techniques applicable to the corresponding images. For example, images can typically be assigned to any of the following image types:

[0091] Intraframe images (I-images) are images that can be encoded and decoded without using any other images in the sequence as prediction sources. Some video codecs allow different types of intraframe images, including, for example, Independent Decoder Refresh (IDR) images. Those skilled in the art are familiar with variations of I-images and their corresponding applications and characteristics.

[0092] A predictive image (P-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most one motion vector and reference index to predict sample values ​​for each block.

[0093] A bidirectional predictive image (B-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses up to two motion vectors and a reference index to predict sample values ​​for each block. Similarly, multiple predictive images can use more than two reference images and associated metadata to reconstruct a single block.

[0094] Source images are typically spatially subdivided into multiple sample blocks (e.g., 4×4, 8×8, 4×8, or 16×16 sample blocks) and encoded block by block. These blocks can be predictively coded with reference to other (already coded) blocks, determined by the coding assignment of the corresponding image applied to the block. For example, a block of an I-image can be non-predictively coded, or it can be predictively coded (spatial or intra-frame prediction) with reference to already coded blocks of the same image. A pixel block of a P-image can be predictively coded with reference to a previously coded reference image via spatial or temporal prediction. A block of a B-image can be predictively coded with reference to one or two previously coded reference images via spatial or temporal prediction. For other purposes, source or intermediate images can be subdivided into other types of blocks. As described in further detail below, the partitioning of coded blocks and other types of blocks may or may not follow the same pattern.

[0095] The video encoder (603) can perform encoding operations according to a predetermined video coding technique or standard, such as ITU-T H.265 Recommendation. In operation, the video encoder (603) can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard used.

[0096] In some exemplary embodiments, the transmitter (640) may transmit additional data while transmitting encoded video. The source encoder (630) may include such data as part of the encoded video sequence. The additional data may include other forms of redundant data such as temporal / spatial / SNR enhancement layers, redundant images and slices, SEI messages, VUI parameter set fragments, etc.

[0097] The acquired video can be presented as multiple source images (video images) in a time series. Intra-frame image prediction (often simplified to intra-frame prediction) utilizes spatial correlations within a given image, while inter-frame image prediction utilizes (temporal or other) correlations between images. In an embodiment, a specific image being encoded / decoded is divided into blocks, referred to as the current image. When a block in the current image resembles a reference block in a previously encoded and still buffered reference image in the video, the block in the current image can be encoded using a vector called a motion vector. This motion vector points to the reference block in the reference image, and when using multiple reference images, the motion vector may have a third dimension that identifies the reference image.

[0098] In some exemplary embodiments, bidirectional prediction techniques can be used in inter-frame image prediction. According to bidirectional prediction techniques, two reference images are used, such as a first reference image and a second reference image, both preceding the current image in the video in decoding order (but possibly past and future in display order). A block in the current image can be encoded using a first motion vector pointing to a first reference block in the first reference image and a second motion vector pointing to a second reference block in the second reference image. The block can be predicted using a combination of the first and second reference blocks.

[0099] In addition, merging mode techniques can be used in inter-frame image prediction to improve coding efficiency.

[0100] According to some exemplary embodiments disclosed in this application, predictions such as inter-frame image prediction and intra-frame image prediction are performed on a block-by-block basis. For example, images in a video image sequence are divided into coding tree units (CTUs) for compression. CTUs in the images may have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Generally, a CTU comprises three parallel coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Furthermore, each CTU may be partitioned into one or more coding units (CUs) using a quadtree. For example, a 64×64 pixel CTU may be partitioned into one 64×64 pixel CU, or four 32×32 pixel CUs. Each of one or more 32×32 blocks may be further partitioned into four 16×16 pixel CUs. In some exemplary embodiments, each CU may be analyzed during encoding to determine the prediction type used for the CU among various prediction types, such as inter-frame prediction or intra-frame prediction. Furthermore, depending on temporal and / or spatial predictability, the CU can be divided into one or more prediction units (PUs). Typically, each PU includes a luma prediction block (PB) and two chroma PBs. In embodiments, prediction operations in encoding (encoding / decoding) are performed on a per-prediction-block basis. The division of the CU into PUs (or PBs for different color channels) can be performed in various spatial modes. For example, a luma or chroma PB may comprise a matrix of pixel values ​​(e.g., luma values) for a sample, such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0101] Figure 7 This is a diagram of a video encoder (703) according to another exemplary embodiment of the present disclosure. The video encoder (703) is used to receive a processing block (e.g., a prediction block) of sample values ​​within a current video image in a video image sequence, and to encode the processing block into an encoded image that is part of an encoded video sequence. In this embodiment, the video encoder (703) is used instead of Figure 4 The example video encoder (403).

[0102] For example, a video encoder (703) receives a matrix of sample values ​​for a processing block, such as an 8×8 sample prediction block. The video encoder (703) uses, for example, rate-distortion optimization (RDO) to determine whether to encode the processing block using intra-frame mode, inter-frame mode, or bidirectional prediction mode. When it is determined that the processing block is to be encoded in intra-frame mode, the video encoder (703) can use intra-frame prediction techniques to encode the processing block into the encoded image; and when it is determined that the processing block is to be encoded in inter-frame mode or bidirectional prediction mode, the video encoder (703) can use inter-frame prediction or bidirectional prediction techniques to encode the processing block into the encoded image, respectively. In some exemplary embodiments, the merging mode may be an inter-frame image prediction sub-mode, wherein motion vectors are derived from one or more motion vector prediction values ​​without the aid of encoded motion vector components outside the predictor. In some other exemplary embodiments, motion vector components applicable to the subject block may exist. In some other exemplary embodiments, the video encoder (703) includes... Figure 7 Other components not shown include, for example, a pattern decision module for determining the prediction pattern of a processing block.

[0103] exist Figure 7 In the example, the video encoder (703) includes, for example, Figure 7 The exemplary arrangement shown in the figure includes an inter-frame encoder (730), an intra-frame encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) coupled together.

[0104] The inter-frame encoder (730) is used to receive samples of the current block (e.g., the processing block), compare the block with one or more reference blocks in a reference image (e.g., blocks in previous and subsequent images in display order), generate inter-frame prediction information (e.g., descriptions of redundancy information, motion vectors, and merging mode information based on the inter-frame coding technique), and calculate inter-frame prediction results (e.g., predicted blocks) based on the inter-frame prediction information using any suitable technique. In some examples, the reference image is a decoded reference image based on encoded video information decoded using the decoding unit 633, which is embedded in... Figure 6 In the exemplary encoder 620 (which is shown as...) Figure 7 The residual decoder 728, as described in further detail below.

[0105] The intra encoder (722) is used to receive samples of the current block (e.g., the processing block), compare the block with encoded blocks in the same image, generate quantization coefficients after transformation, and in some cases, generate intra prediction information (e.g., based on intra prediction direction information of one or more intra coding techniques). The intra encoder (722) also calculates intra prediction results (e.g., predicted blocks) based on the intra prediction information and reference blocks in the same image.

[0106] A general controller (721) is used to determine general control data and, based on this general control data, control other components of the video encoder (703). In one example, the general controller (721) determines the prediction mode of a block and, based on this prediction mode, provides control signals to a switch (726). For example, when the mode is intra-frame mode, the general controller (721) controls the switch (726) to select an intra-frame mode result for use by the residual calculator (723) and controls the entropy encoder (725) to select intra-frame prediction information and include the intra-frame prediction information in the bitstream; and when the prediction mode of a block is inter-frame mode, the general controller (721) controls the switch (726) to select an inter-frame prediction result for use by the residual calculator (723) and controls the entropy encoder (725) to select inter-frame prediction information and include the inter-frame prediction information in the bitstream.

[0107] A residual calculator (723) is used to calculate the difference (residual data) between the received block and the prediction result selected from the intra encoder (722) or the inter encoder (730). A residual encoder (724) is used to encode the residual data to generate transform coefficients. For example, the residual encoder (724) is used to transform the residual data from the spatial domain to the frequency domain and generate transform coefficients. The transform coefficients are then quantized to obtain quantized transform coefficients. In various exemplary embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is used to perform an inverse transform and generate decoded residual data. The decoded residual data can be used appropriately by the intra encoder (722) and the inter encoder (730). For example, the inter encoder (730) can generate a decoded block based on the decoded residual data and inter-frame prediction information, and the intra encoder (722) can generate a decoded block based on the decoded residual data and intra-frame prediction information. The decoded blocks are processed appropriately to generate a decoded image, which can be buffered in a memory circuit (not shown) and used as a reference image.

[0108] An entropy encoder (725) is used to format the bitstream to include encoded blocks and perform entropy encoding. The entropy encoder (725) is used to include various types of information in the bitstream. For example, the entropy encoder (725) is used to include general control data, selected prediction information (e.g., intra-frame prediction information or inter-frame prediction information), residual information, and other suitable information in the bitstream. Residual information may be absent when blocks are encoded in a merged sub-mode of inter-frame mode or bidirectional prediction mode.

[0109] Figure 8 This is a diagram of an exemplary video decoder (810) according to another embodiment of the present disclosure. The video decoder (810) is used to receive an encoded image as part of an encoded video sequence and decode the encoded image to generate a reconstructed image. In one example, the video decoder (810) is used instead of Figure 4 The video decoder (410) in the example.

[0110] exist Figure 8 In the example, the video decoder (810) includes, for example, Figure 8 The exemplary arrangement shows an entropy decoder (871), an inter-frame decoder (880), a residual decoder (873), a reconstruction module (874), and an intra-frame decoder (872) coupled together.

[0111] An entropy decoder (871) can be used to reconstruct certain symbols from an encoded image, representing the syntax elements constituting the encoded image. Such symbols may include, for example, the mode used to encode the block (e.g., intra-frame mode, inter-frame mode, bidirectional prediction mode, merged sub-mode, or another sub-mode), prediction information (e.g., intra-frame prediction information or inter-frame prediction information) that can be identified for prediction by either the intra-frame decoder (872) or the inter-frame decoder (880), residual information in the form of, for example, quantized transform coefficients, and so on. In one example, when the prediction mode is inter-frame or bidirectional prediction mode, inter-frame prediction information is provided to the inter-frame decoder (880); and when the prediction type is intra-frame prediction type, intra-frame prediction information is provided to the intra-frame decoder (872). Residual information may be provided to the residual decoder (873) via inverse quantization.

[0112] The inter-frame decoder (880) is used to receive inter-frame prediction information and generate inter-frame prediction results based on the inter-frame prediction information.

[0113] The intra-frame decoder (872) is used to receive intra-frame prediction information and generate prediction results based on the intra-frame prediction information.

[0114] The residual decoder (873) performs inverse quantization to extract the dequantized transform coefficients and processes the dequantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder (873) may also use certain control information (to include quantizer parameters (QP)), which may be provided by the entropy decoder (871) (the data path is not indicated because this is only a low amount of control information).

[0115] The reconstruction module (874) is used to combine the residual output by the residual decoder (873) with the prediction result (which may be output by the inter-frame prediction module or the intra-frame prediction module, as appropriate) in the spatial domain to form a reconstructed block, which may be a part of a reconstructed image, which in turn may be a part of a reconstructed video. It should be noted that other suitable operations, such as deblocking, may be performed to improve visual quality.

[0116] It should be noted that any suitable technology can be used to implement the video encoder (403), video encoder (603), and video encoder (703), as well as the video decoder (410), video decoder (510), and video decoder (810). In some exemplary embodiments, one or more integrated circuits can be used to implement the video encoder (403), video encoder (603), and video encoder (703), as well as the video decoder (410), video decoder (510), and video decoder (810). In another embodiment, one or more processors executing software instructions can be used to implement the video encoder (403), video encoder (603), and video encoder (703), as well as the video decoder (410), video decoder (510), and video decoder (810).

[0117] Turning to block partitioning for encoding and decoding, partitioning can generally begin with basic blocks and can follow a predefined set of rules, a specific pattern, a partition tree, or any partitioning structure or scheme. Partitioning can be hierarchical and recursive. After splitting or partitioning the basic blocks according to any of the exemplary partitioning processes described below, or other processes or combinations thereof, a final set of partitions or coding blocks can be obtained. Each of these partitions can be at one of the different partitioning levels in the partitioning hierarchy and can have various shapes. Each partition can be referred to as a coding block (CB). For the various exemplary partitioning implementations further described below, each generated CB can be of any allowed size and partitioning level. Such partitions are called coding blocks because they can form units for which some basic encoding / decoding can be performed, and encoding / decoding parameters can be optimized, determined, and written into the encoded video stream. The highest or deepest level in the final partition represents the depth of the coding block partitioning structure of the tree. Coding blocks can be luma coding blocks or chroma coding blocks. The CB tree structure for each color can be referred to as a coding block tree (CBT).

[0118] The coding blocks for all color channels can be collectively referred to as a coding unit (CU). The hierarchical structure of all color channels can be collectively referred to as a coding tree unit (CTU). The partitioning pattern or structure of different color channels in a CTU may be the same or different.

[0119] In some implementations, the partitioning tree schemes or structures used for the luma and chroma channels may not need to be the same. In other words, the luma and chroma channels can have separate coding tree structures or patterns. Furthermore, whether the luma and chroma channels use the same or different coding tree structures, and the actual coding tree structure used, can depend on whether the slice being encoded is a P-slice, a B-slice, or an I-slice. For example, for an I-slice, the chroma and luma channels can have separate coding tree structures or coding tree structure patterns, while for P-slices or B-slices, the luma and chroma channels can share the same coding tree scheme. When separate coding tree structures or patterns are applied, the luma channel can be partitioned into CBs using one coding tree structure, and the chroma channel can be partitioned into chroma CBs using another coding tree structure.

[0120] In some exemplary implementations, a predetermined partitioning pattern can be applied to basic blocks. For example... Figure 9As shown, an exemplary 4-way partitioning tree can begin at a first predefined level (e.g., a 64×64 block level or other size, as the basic block size), and the basic block can be hierarchically partitioned down to a predefined lowest level (e.g., a 4×4 level). For example, the basic block can have four predefined partitioning options or patterns indicated by 902, 904, 906, and 908, where the partition designated as R is allowed for recursive partitioning, such that... Figure 9 The same partitioning options indicated in the code can be repeated at lower scales up to the lowest level (e.g., a 4x4 level). In some implementations, this can be... Figure 9 The partitioning scheme applies additional restrictions. Figure 9 In the implementation, rectangular partitioning (e.g., 1:2 / 2:1) is allowed, but these rectangular partitions cannot be recursive, while square partitioning is allowed to be recursive. If necessary, according to... Figure 9 The recursive partitioning generates a final set of coded blocks. The coded tree depth can be further defined to indicate the partitioning depth from the root node or root block. For example, the coded tree depth of the root node or root block (e.g., 64×64 blocks) can be set to 0, and the partitioning depth from the root block can be determined according to... Figure 9 After being further divided once, the coding tree depth increases by 1. For the above scheme, the maximum or deepest level from the 64×64 basic block to the 4×4 minimum partition will be 4 (starting from level 0). This partitioning scheme can be applied to one or more color channels. It can be followed as... Figure 9 The scheme independently partitions each color channel (e.g., the partitioning mode or option in a predefined pattern can be determined independently for each color channel at each layer level). Optionally, two or more color channels can share... Figure 9 The same hierarchical pattern tree (for example, the same partitioning pattern or option can be selected in a predefined pattern for two or more color channels at each hierarchical level).

[0121] Figure 10 This illustrates another exemplary predefined partitioning pattern that allows recursive partitioning to form a partition tree. For example... Figure 10 As shown, an exemplary 10-way partitioning structure or pattern can be predefined. The root block can start from a predefined level (e.g., from a 128×128 level or a 64×64 level basic block). Figure 10 The example partitioning structures include various rectangular partitions of 2:1 / 1:2 and 4:1 / 1:4. Figure 10 The three sub-partitions 1002, 1004, 1006, and 1008 indicated in the second line can be referred to as "T-shaped" partitions. These "T-shaped" partitions 1002, 1004, 1006, and 1008 can be called left T-shaped, upper T-shaped, right T-shaped, and lower T-shaped, respectively. In some example implementations, further subdivision is not allowed. Figure 10 Any of the rectangular partitions. The encoding tree depth can be further defined to indicate the partition depth from the root node or root block. For example, the encoding tree depth of the root node or root block (e.g., a 128×128 block) can be set to 0, and the root block can be partitioned according to... Figure 10 After being further segmented once, the coding tree depth increases by 1. In some implementations, only full square partitions within 1010 are allowed according to... Figure 10 The pattern is recursively partitioned to the next level of the partition tree. In other words, for the square partitions within T-patterns 1002, 1004, 1006, and 1008, recursive partitioning is not allowed. If necessary, it can be done according to... Figure 10 The recursive partitioning process generates a final set of coded blocks. This scheme can be applied to one or more color channels. In some implementations, more flexibility can be added when using partitions at levels smaller than 8×8. For example, 2×2 chroma inter-frame prediction can be used in certain situations.

[0122] In some other exemplary implementations of coded block partitioning, a quadtree structure can be used to partition a base block or intermediate block into quadtree partitions. This quadtree partitioning can be applied hierarchically and recursively to any square partition. Whether a base block, intermediate block, or partition is further quadtree-partitioned can be adapted to various local features of the base block or intermediate block / partition. Quadtree partitions can be further adapted at image boundaries. For example, implicit quadtree partitioning can be performed at image boundaries such that the block will remain quadtree-partitioned until its size fits the image boundary.

[0123] In some other example implementations, hierarchical binary partitioning from the base block can be used. For such a scheme, the base block or intermediate block can be divided into two partitions. The binary partition can be horizontal or vertical. For example, a horizontal binary partition can divide the base block or intermediate block into equal right and left partitions. Similarly, a vertical binary partition can divide the base block or intermediate block into equal upper and lower partitions. This binary partitioning can be hierarchical and recursive. At each base block or intermediate block, it can be decided whether the binary partitioning scheme should continue, and if so, whether a horizontal or vertical binary partition should be used. In some implementations, further partitioning may stop at a predefined minimum partition size (in one or two dimensions). Alternatively, further partitioning can stop once a predefined partitioning level or depth from the base block is reached. In some implementations, the aspect ratio of the partitions may be limited. For example, the aspect ratio of the partitions may be no less than 1:4 (or greater than 4:1). Therefore, a vertical strip partition with an aspect ratio of 4:1 can only be further divided into two vertically bifurcated partitions, each with an aspect ratio of 2:1.

[0124] In some other examples, such as Figure 13 As shown, the ternary partitioning scheme can be used to partition basic blocks or any intermediate blocks. A ternary pattern can be shown as follows: Figure 13 As shown in 1302, it is implemented vertically, or as... Figure 13 As shown in 1304, it is implemented horizontally. Although Figure 13 An exemplary partitioning ratio (vertically or horizontally) is shown as 1:2:1, but other ratios can be predefined. In some implementations, two or more different ratios can be predefined. This ternary partitioning scheme can be used to complement quadtree or binary partitioning structures because this ternary partitioning can capture objects located at the center of a block within a contiguous partition, while quadtrees and binary trees always partition along the block center, thus dividing the object into separate partitions. In some implementations, the width and height of the example ternary partition are always powers of 2 to avoid additional transformations.

[0125] The partitioning schemes described above can be combined in any way at different partitioning levels. As an example, the quadtree and binary partitioning schemes described above can be combined to partition the basic blocks into a quadtree-binary-tree (QTBT) structure. In this scheme, the basic blocks or intermediate blocks / partitions can be quadtree partitions or binary partitions, depending on a set of predefined conditions, if specified. Figure 14 A specific example is shown in the figure.

[0126] Figure 14 Exemplary partitions and trees for a Quadtree Plus Binary Tree (QTBT) structure are shown. A QTBT structure may not have a concept similar to multiple partition types, and it can eliminate the separation of CU, PU, ​​and TU concepts. The QTBT structure can support increased flexibility regarding the shape of the CU partition. In some embodiments of the QTBT block structure, the CU can have a square or rectangular shape.

[0127] exist Figure 14In the examples, a basic block is first divided into four partitions by a quadtree, as shown in 1402, 1404, 1406, and 1408. Subsequently, each generated partition at the next level is either divided into four further partitions by the quadtree (e.g., 1408), or into two further partitions by a binary tree (e.g., horizontally or vertically, e.g., 1402 or 1406, both symmetrical), or not divided at all (e.g., 1404). For square partitions, recursively allowing binary or quadtree partitions is possible, as shown in the overall partitioning pattern example in 1410 and the corresponding tree structure / representation in 1420, where solid lines represent quadtree partitions and dashed lines represent binary tree partitions. Flags can be used for each binary partition node (non-leaf binary partition) to indicate whether the binary partition is horizontal or vertical. For example, as shown in 1420, consistent with the partitioning structure of 1410, a flag "0" can indicate a horizontal binary partition, and a flag "1" can indicate a vertical binary partition. For quadtree partitioning, it is not necessary to indicate the partition type, because quadtree partitioning always divides blocks or partitions horizontally and vertically to produce four sub-blocks / partitions of equal size. In some implementations, the flag "1" can indicate a horizontal binary partition, and the flag "0" can indicate a vertical binary partition.

[0128] like Figure 14 As shown, a quadtree structure is first used to partition the coding tree units (CTUs). The leaf nodes of the quadtree are further partitioned using a binary tree structure. In the binary tree partitioning, there can be two types of partitioning: symmetrical horizontal partitioning and symmetrical vertical partitioning. The leaf nodes of the binary tree are called coding units (CUs), and this segment can be used for prediction and transform processing without any further partitioning. In the QTBT coding block structure, CUs, PUs, and TUs can have the same block size. In JEM, a CU can include coding blocks (CBs) of different color components (e.g., in the case of P and B slices in a 4:2:0 chroma format, a CU includes one luma CB and two chroma CBs). In other embodiments, a CU can include a CB of a single component (e.g., in the case of I slices, a CU includes one luma CB or two chroma CBs).

[0129] In some exemplary implementations of QTBT, quadtrees and binary partitioning rule sets can be represented by the following predefined parameters and their associated corresponding functions:

[0130] CTU size: The size of the root node of a quadtree (the size of a basic block).

[0131] MinQTSize: Minimum allowed size of a quadleaf node

[0132] MaxBTSize: Maximum allowed size of binary tree root node

[0133] MaxBTDepth: Maximum allowed binary tree depth

[0134] MinBTSize: Minimum allowed size of binary leaf nodes

[0135] In some example implementations of the QTBT partitioning structure, the CTU size can be set to 128×128 luminance samples and two corresponding 64×64 chrominance sample blocks (when considering and using example chrominance subsampling). MinQTSize It can be set to 16×16. MaxBTSize It can be set to 64×64. MinBTSize (For width and height) it can be set to 4×4, MaxBTDepth It can be set to 4. Quadtree partitioning can first be applied to CTU to generate quadtree leaf nodes. The size of quadtree leaf nodes can be set from its minimum allowed size of 16×16 (i.e., ... MinQTSize The size of a binary tree is 128×128 (i.e., CTU size). If a node is 128×128, it will not be partitioned by the binary tree first because its size exceeds the limit. MaxBTSize (i.e., 64×64). Otherwise, not exceeding MaxBTSize The nodes can be partitioned using a binary tree. Figure 14 In the example, the basic block is 128×128. According to the predefined rule set, the basic block can only be partitioned into quadtrees. The partitioning depth of the basic block is 0. Each of the four resulting partitions is 64×64, not exceeding... MaxBTSize Further quadtree or binary tree partitioning can be performed at level 1. This process continues. When the binary tree depth reaches... MaxBTDepth (i.e., 4) , further partitioning can be disregarded. When the width of a binary tree node is equal to... MinBTSize (i.e., 4) When the height of a binary tree node is equal to 1, further horizontal partitioning can be disregarded. Similarly, when the height of a binary tree node is equal to 1... MinBTSize At that point, further vertical segmentation is no longer considered.

[0136] In some exemplary embodiments, the QTBT scheme described above can be configured to support the flexibility of having the same QTBT structure for luma and chroma, or having separate QTBT structures. For example, for P-slices and B-slices, the luma and chroma CTBs in a CTU can share the same QTBT structure. However, for I-slices, the luma CTB can be partitioned into CBs using a QTBT structure, and the chroma CTB can be partitioned into chroma CBs using a different QTBT structure. This means that CUs can be used to refer to different color channels in an I-slice; for example, an I-slice can consist of a coded block for the luma component or coded blocks for the two chroma components, and a CU in a P-slice or B-slice can consist of coded blocks for all three color components.

[0137] Figure 14 An example of block partitioning using QTBT is shown ( Figure 14 (on the left), and shows the corresponding tree representation ( Figure 14 (To the right of the text). Solid lines represent quadtree partitions, and dashed lines represent binary tree partitions. In each partition (i.e., non-leaf) node of a binary tree, a flag is written to indicate which partition type (i.e., horizontal or vertical) is used, where 0 indicates a horizontal partition and 1 indicates a vertical partition. For quadtree partitions, it is not necessary to indicate the partition type because quadtree partitions divide blocks horizontally and vertically to produce 4 sub-blocks of equal size.

[0138] In some other implementations, the QTBT scheme can be supplemented with the ternary branch scheme described above. This implementation can be referred to as a multi-type-tree (MTT) structure. For example, in addition to binary branching of nodes, other options can be selected... Figure 13 One of the ternary partitioning patterns. In some implementations, only square nodes can be ternary partitioned. Additional flags can be used to indicate whether the ternary partition is horizontal or vertical.

[0139] The design of two- or multi-level trees, such as QTBT implementations and QTBT implementations supplemented by ternary partitioning, is likely primarily for complexity reduction. Theoretically, the complexity of traversing a tree is O(n log n). ,in Indicates the number of partition types. It is the depth of the tree. A trade-off can be made by using multiple types (T) while reducing the depth (D).

[0140] Furthermore, the QTBT scheme supports the flexibility of having independent QTBT structures for luma and chroma. For example, for P-slices and B-slices, the luma CTB and chroma CTB in a CTU share the same QTBT structure. However, for I-slices, the luma CTB is partitioned into multiple CUs using a QTBT structure, and the chroma CTB is partitioned into multiple chroma CUs using a different QTBT structure. In this example, one CU in an I-slice includes a coding block for the luma component or a coding block for both chroma components, and one CU in a P-slice or B-slice includes a coding block for all three color components. In HEVC, inter-frame prediction for small blocks can be restricted to reduce memory access for motion compensation, resulting in no bidirectional prediction for 4×8 and 8×4 blocks, and no inter-frame prediction for 4×4 blocks. These restrictions can be removed in the QTBT implementation in JEM-7.0.

[0141] In some implementations, the CB can be further subdivided. For example, for the purpose of intra-frame or inter-frame prediction during the encoding and decoding process, the CB can be further subdivided into multiple prediction blocks. In other words, the CB can be further subdivided into different sub-partitions in which individual prediction decisions / configurations can be made. In parallel, to depict the level at which the transform or inverse transform of the video data is performed, the CB can be further subdivided into multiple transform blocks (TBs). The schemes for subdividing the CB into PBs and TBs can be the same or different. For example, each partitioning scheme can be performed using its own process, such as based on various features of the video data. In some example implementations, the PB and TB partitioning schemes may be independent. In other example implementations, the PB and TB partitioning schemes and boundaries may be interrelated. For example, in some implementations, TBs can be partitioned after PB partitioning; specifically, each PB is determined after coded block partitioning and can then be further subdivided into one or more TBs. For example, in some implementations, a PB can be divided into one, two, four, or other numbers of TBs.

[0142] In some implementations, the luma and chroma channels can be processed differently to divide the basic block into coded blocks and further into prediction blocks and / or transform blocks. For example, in some implementations, for the luma channel, dividing the coded block into prediction blocks and / or transform blocks may be permitted, while for the chroma channel, dividing the coded block into prediction blocks and / or transform blocks may not be permitted. Thus, in such an implementation, the transform and / or prediction of the luma block can be performed only at the coded block level. For another example, the minimum transform block size for the luma and chroma channels can be different; for example, the coded block for the luma channel may be allowed to be divided into transform and / or prediction blocks smaller than those for the chroma channel. For yet another example, the maximum depth at which the coded block is divided into transform and / or prediction blocks can be different between the luma and chroma channels; for example, the coded block for the luma channel may be allowed to be divided into transform and / or prediction blocks deeper than those for the chroma channel. For a specific example, the luma coding block can be divided into multiple transform blocks of various sizes, which can be represented recursively, with a maximum of two levels of recursion. Transform block shapes such as squares, 2:1 / 1:2, and 4:1 / 1:4, and transform block sizes ranging from 4×4 to 64×64 are allowed. However, for the chroma block, only the largest possible transform block specified for the luma block is permitted.

[0143] In some example implementations that divide a coded block into a PB, the depth, shape, and / or other characteristics of the PB division can depend on whether the PB is intra-frame coded or inter-frame coded.

[0144] Dividing a coded block (or prediction block) into transform blocks can be implemented in various example schemes, including but not limited to recursive or non-recursive quadtree partitioning and predefined pattern partitioning, with additional consideration given to transform blocks at the boundaries of the coded or prediction blocks. Typically, the resulting transform blocks can be at different partitioning levels, can have different sizes, and do not need to be square (e.g., they can be rectangles with some allowed size and aspect ratio). The following combines... Figure 15 , Figure 16 and Figure 17 Other examples are described in more detail.

[0145] However, in some other implementations, the CB obtained via any of the above partitioning schemes can be used as the basic or minimum coded block for prediction and / or transform. In other words, no further partitioning is performed for inter-frame / intra-frame prediction and / or transform purposes. For example, the CB obtained according to the QTBT scheme described above can be directly used as the unit for performing prediction. Specifically, this QTBT structure eliminates the concept of multiple partitioning types; that is, it eliminates the separation of CU, PU, ​​and TU, and provides greater flexibility for the CU / CB partitioning shape as described above. In this QTBT block structure, the CU / CB can have a square or rectangular shape. The leaf nodes of this QTBT are used as units for prediction and transform processing without any further partitioning. This means that in this exemplary QTBT coded block structure, the CU, PU, ​​and TU have the same block size.

[0146] The various CB partitioning schemes described above, as well as further partitioning CB into PB and / or TB (excluding PB / TB partitioning), can be combined in any way. The following specific implementations are provided as non-limiting examples.

[0147] The following describes a specific example implementation of the partitioning of coded blocks and transform blocks. In such an example implementation, recursive quadtree partitioning or the predefined partitioning patterns mentioned above (e.g.) can be used. Figure 9 and Figure 10The basic block (CB) is divided into coded blocks (CBs) using a pattern (as described in the example). At each level, whether further quadtree segmentation of a particular partition should be performed can be determined by the characteristics of the local video data. The resulting CBs can be at various quadtree segmentation levels and have various sizes. The decision on whether to use inter-frame (temporal) or intra-frame (spatial) prediction to encode the image region can be made at the CB level (or CU level, for all three color channels). Each CB can be further segmented into one, two, four, or other numbers of PBs according to a predefined PB segmentation type. Within a PB, the same prediction process can be applied, and relevant information can be sent to the decoder based on the PB. After obtaining the remaining blocks by applying the prediction process based on the PB segmentation type, the CB can be divided into TBs according to another quadtree structure similar to the CB. In this particular implementation, the CB or TB can be, but is not limited to, squares. Furthermore, in this particular example, for inter-frame prediction, the PB can be square or rectangular, while for intra-frame prediction, the PB can simply be square. A coded block can be divided into, for example, four square TBs. Each TB can be further recursively divided (using quadtrees) into smaller TBs, called a Residual Quadtree (RQT).

[0148] The following describes another example implementation for partitioning basic blocks into CB, PB, and / or TB. For example, a quadtree with an embedded multi-type tree employing binary and ternary partitioning structures (e.g., a QTBT or a QTBT with ternary partitioning as described above) could be used instead of methods such as... Figure 9 or Figure 10 The diagram illustrates several partitioning unit types. Separation of CB, PB, and TB (i.e., partitioning a CB into PB and / or TB, and vice versa) may be abandoned unless the size of the CB is too large for the maximum transform length, in which case further partitioning of the CB may be necessary. This example partitioning scheme can be designed to support greater flexibility in the shape of CB partitions, allowing both prediction and transform to be performed at the CB level without further partitioning. In such a coding tree structure, the CB can have a square or rectangular shape. Specifically, the coding tree block (CTB) can first be partitioned using a quadtree structure. Then, the leaf nodes of the quadtree can be further partitioned using an embedded multi-type tree structure. Figure 11 Examples of embedded multi-type tree structures using binary or ternary partitioning are shown. Specifically, Figure 11The exemplary multi-type tree structure includes four partition types, referred to as vertical binary partition (SPLIT_BT_VER) (1102), horizontal binary partition (SPLIT_BT_HOR) (1104), vertical ternary partition (SPLIT_TT_VER) (1106), and horizontal ternary partition (SPLIT_TT_HOR) (1108). The CB then corresponds to the leaf of the multi-type tree. In this example implementation, unless the CB is too large for the maximum transform length, the partition is used for both prediction and transform processing without any further division. This means that, in most cases, in a quadtree with an embedded multi-type tree coding block structure, the CB, PB, and TB have the same block size. Exceptions occur when the maximum supported transform length is less than the width or height of the color component of the CB. In some implementations, in addition to binary or ternary partitions, Figure 11 The embedded mode can also include quadtree partitioning.

[0149] Figure 12 A concrete example of a quadtree with an embedded multi-type tree-coded block structure is shown, which partitions a basic block (including quadtree, binary tree, and ternary partitioning options). More detailed... Figure 12 The diagram shows that basic block 1200 is divided into four square partitions 1202, 1204, 1206, and 1208 by a quadtree. For each quadtree partition, a decision is made regarding further use... Figure 11 Further segmentation is performed on various types of tree structures and quadtrees. Figure 12 In the example, partition 1204 was not further divided. Partitions 1202 and 1208 were each divided using another quadtree. For partition 1202, the top-left, top-right, bottom-left, and bottom-right partitions of the second-order quadtree were divided using quadtrees respectively. Figure 11 Horizontal binary partition 1104, non-partitioning and Figure 11 The horizontal ternary partition 1108 is a three-level partition. Partition 1208 uses another quadtree partition, with the upper left, upper right, lower left, and lower right partitions of the two-level quadtree partition using... Figure 11 Vertical ternary partition 1106, no partition, no partition Figure 11 The horizontal binary partition of 1104 is a tertiary partition. Based on... Figure 11 The horizontal binary partition 1104 and the horizontal ternary partition 1108 further divide 1208 into two sub-partitions in the upper left of the third level. Partition 1206 adopts a continuation... Figure 11 The second-level partitioning pattern after the vertical binary partitioning 1102 divides the data into two partitions, and then... Figure 11 The third-level segmentation is performed using the horizontal ternary segmentation 1108 and the vertical binary segmentation 1102. Based on... Figure 11The horizontal binary split 1104 is further applied to one of them, and the fourth-level split is applied to it.

[0150] For the specific example above, the maximum luminance transformation size can be 64×64, and the supported maximum chrominance transformation size can be different from the luminance, for example, 32×32. Even Figure 12 The example CB above is usually not further divided into smaller PB and / or TB. When the width or height of the luma-coded block or chroma-coded block is greater than the maximum transform width or height, the luma-coded block or chroma-coded block can be automatically divided in the horizontal and / or vertical directions to meet the transform size limit in that direction.

[0151] In the specific example above used to divide a basic block into CBs, as mentioned above, the coding tree scheme can support the ability for luma and chroma to have separate block tree structures. For example, for P-slices and B-slices, the luma and chroma CTBs in a CTU can share the same coding tree structure. For example, for I-slices, luma and chroma can have separate coding block tree structures. When using separate block tree structures, the chroma CTB can be divided into chroma CBs using one coding tree structure and the chroma CTB into chroma CBs using another coding tree structure. This means that a CU in an I-slice may consist of coding blocks for one luma component or two chroma components, while a CU in a P-slice or B-slice always consists of coding blocks for all three color components, unless the video is monochrome.

[0152] When a coded block is further divided into multiple transform blocks, these transform blocks can be ordered in the bitstream in various sequences or scanning methods. The following describes in more detail example implementations of dividing a coded block or prediction block into transform blocks and the encoding order of these transform blocks. In some example implementations, as described above, transform partitioning can support transform blocks of multiple shapes, such as 1:1 (square), 1:2 / 2:1, and 1:4 / 4:1, with transform block sizes ranging from 4×4 to 64×64. In some implementations, if the coded block is less than or equal to 64×64, the transform block partitioning can be applied only to the luma component, such that for the chroma block, the transform block size is the same as the coded block size. Otherwise, if the coded block width or height is greater than 64, both the luma and chroma coded blocks can be implicitly partitioned into multiples of min(W, 64) × min(H, 64) and min(W, 32) × min(H, 32), respectively.

[0153] In some example implementations of transform block partitioning, for intra-frame and inter-frame coding blocks, the coding block can be further divided into multiple transform blocks, with a partitioning depth of up to a predetermined number of levels (e.g., 2 levels). The transform block partitioning depth and size can be correlated. For some example implementations, the mapping from the transform block size at the current depth to the transform block size at the next depth is shown in Table 1 below.

[0154] Table 1: Settings for changing partition size

[0155]

[0156] Based on the example mapping in Table 1, for a 1:1 square block, the next-level transform partition can create four 1:1 square sub-transform blocks. Transform block partitioning can stop at, for example, 4×4. Therefore, the size of a transform block at the current depth of 4×4 corresponds to the same size 4×4 at the next depth. In the examples in Table 1, for a 1:2 / 2:1 non-square block, the next-level transform partition can create two 1:1 square sub-transform blocks, while for a 1:4 / 4:1 non-square block, the next-level transform partition can create two 1:2 / 2:1 sub-transform blocks.

[0157] In some exemplary implementations, additional constraints may be imposed on the luma component of an intra-frame coded block relative to the transform block partitioning. For example, for each level of transform partitioning, all sub-transform blocks may be constrained to have equal sizes. For instance, for a 32×16 coded block, a level 1 transform partition creates two 16×16 sub-transform blocks, and a level 2 transform partition creates eight 8×8 sub-transform blocks. In other words, the second-level partitioning must be applied to all level 1 sub-blocks to maintain equal transform unit sizes. Figure 15 The diagram illustrates an example of dividing an intra-frame coded square block into transform blocks according to Table 1, with the encoding order indicated by arrows. Specifically, 1502 shows a square coded block. 1504 shows a first-level division of the coded block into four equal-sized transform blocks according to Table 1, with the encoding order of these four transform blocks indicated by arrows. 1506 shows a second-level division of all first-level equal-sized blocks according to Table 1 into 16 equal-sized transform blocks, with the encoding order of these 16 transform blocks indicated by arrows.

[0158] In some exemplary implementations, the aforementioned restrictions on intra-frame coding may not apply to the luminance component of inter-frame coded blocks. For example, after the first-level transform segmentation, any sub-transform block can be further independently segmented into more than one level. Therefore, the resulting transform blocks may be of the same size or of different sizes. Figure 16 The image shows an example of dividing an inter-frame coded block into multiple transform blocks with their coding order. Figure 16In the example, according to Table 1, inter-frame coded block 1602 is divided into two levels of transform blocks. At the first level, the inter-frame coded block is divided into four transform blocks of equal size. Then, only one (not all) of the four transform blocks is further divided into four sub-transform blocks, resulting in a total of seven transform blocks with two different sizes, as shown in 1604. The example coding order of these seven transform blocks is... Figure 16 The arrow in 1604 is shown.

[0159] In some exemplary implementations, additional constraints may be applied to the transform block for one or more chroma components. For example, the transform block size may be the same as the coding block size for one or more chroma components, but not smaller than a predefined size, such as 8×8.

[0160] In some other exemplary embodiments, for width ( W ) or height ( H For coding blocks greater than 64, both the luma coding block and the chroma coding block can be implicitly divided into min( W , 64)×min( H ,64) and min( W , 32)×min( H The number of transformation units is a multiple of 32. In this document, in this disclosure, "min(a, b)" can return the smaller value between a and b.

[0161] Figure 17 Another alternative example scheme for dividing a coding block or prediction block into multiple transform blocks is further illustrated. For example... Figure 17 As shown, a set of predefined partition types can be applied to the coded block based on the transform type, instead of using recursive transform partitioning. Figure 17 In the specific example shown, one of the six example partitioning types can be applied to divide the coded block into various numbers of transform blocks. This scheme for generating transform block partitions can be applied to either coded blocks or prediction blocks.

[0162] To be more detailed, Figure 17 The partitioning scheme provides up to six example partition types for any given transform type (transform type refers to, for example, the type of the main transform, such as ADST, etc.). In this scheme, a transform partition type can be assigned to each coding block or prediction block based on, for example, rate-distortion cost. In the examples, the transform partition type assigned to a coding block or prediction block can be determined based on its transform type. A specific transform partition type can correspond to a transform block split size and mode, such as... Figure 17The six transform partition types are shown in the diagram. The correspondence between various transform types and various transform partition types can be predefined. An example is shown below, where uppercase labels indicate the transform partition type that can be assigned to a coding block or a prediction block based on rate-distortion cost:

[0163] PARTITION_NONE (No partitioning): Allocate a transformation size equal to the block size.

[0164] PARTITION_SPLI (Segmentation): Allocate a transform size whose width is half the width of the block size and whose height is half the height of the block size.

[0165] PARTITION_HORZ (Horizontal Partition): Allocate a transform size with the same width as the block size and a height that is half the height of the block size.

[0166] PARTITION_VERT (Vertical Partition): Allocate a transform size with a width that is half the width of the block size and a height that is the same as the height of the block size.

[0167] PARTITION_HORZ4 (Horizontal 4-partition): Allocate a transform size with the same width as the block size and a height that is one-quarter of the block size height.

[0168] PARTITION_VERT4 (Vertical 4-partition): Allocate a transform size with a width that is one-quarter the width of the block size and a height that is the same as the height of the block size.

[0169] In the example above, such as Figure 17 The transform partitioning type shown here uses a uniform transform size for all partitioned transform blocks. This is just an example and is not restrictive. In some other implementations, mixed transform block sizes may be used for partitioned transform blocks within a specific partitioning type (or pattern).

[0170] A PB (or CB, also called a PB when it is not further divided into prediction blocks) obtained according to any of the above partitioning schemes can become a single block for coding via intra-frame or inter-frame prediction. For inter-frame prediction of the current PB, a residual between the current block and the prediction block can be generated, encoded, and included in the encoded bitstream.

[0171] Inter-frame prediction can be implemented, for example, in single-reference mode or composite reference mode. In some implementations, a skip flag can be included in the bitstream of the current block (or higher) to indicate whether the current block is inter-coded and not skipped. If the current block is inter-coded, another flag can be further included in the bitstream as a signal to indicate whether the current block uses single-reference mode or composite reference mode. For single-reference mode, a single reference block can be used to generate the prediction block for the current block. For composite reference mode, two or more reference blocks can be used, for example, by weighted averaging to generate the prediction block. Composite reference mode can be referred to as more than one reference mode, two reference modes, or multiple reference modes. One or more reference blocks can be identified using one or more reference frame indices and additionally using one or more corresponding motion vectors indicating the positional (e.g., horizontal and vertical pixel) offset between the reference blocks and the current block. For example, the inter-frame prediction block for the current block can be generated from a single reference block identified by a motion vector in the reference frames as a prediction block in single-reference mode, while for composite reference mode, the prediction block can be generated by weighted averaging of two reference blocks in two reference frames indicated by two motion vectors. Motion vectors can be encoded and included in the bitstream in various ways.

[0172] In some implementations, the encoding or decoding system may have a decoded picture buffer (DPB). Some pictures / images can be stored in the DPB awaiting display (in the decoding system), and some pictures / images in the DPB can be used as reference frames for inter-frame prediction. In some implementations, the reference frames in the DPB can be labeled as short-term or long-term references to the current picture being encoded or decoded. For example, short-term reference frames may include frames for performing inter-frame prediction on blocks in the current frame or for performing inter-frame prediction on blocks in a predetermined number (e.g., 2) of subsequent video frames closest to the current frame in decoding order. Long-term reference frames may include frames in the DPB that can be used to predict picture blocks in more than a predefined number of frames that are far removed from the current frame in decoding order. This information regarding the labeling of short-term and long-term reference frames may be referred to as a Reference Picture Set (RPS), and this information may be added to the header of each frame in the encoded bitstream. Each frame in an encoded video stream can be identified by a Picture Order Counter (POC), which numbers the frames in an absolute manner or in order associated with a group of images, such as an I-frame, based on the playback sequence.

[0173] In some exemplary implementations, one or more reference image lists containing identifiers of short-term and long-term reference frames for inter-frame prediction can be formed based on information in the RPS. For example, a single image reference list, denoted as L0 reference (or reference list 0), can be formed for unidirectional inter-frame prediction, while two image reference lists can be formed for bidirectional inter-frame prediction, denoted as L0 (or reference list 0) and L1 (or reference list 1) for each prediction direction. The reference frames included in the L0 and L1 lists can be ordered in various predetermined ways. The lengths of the L0 and L1 lists can be written into the video bitstream. When multiple reference frames used to generate a prediction block by weighted averaging are located on the same side of the block to be predicted in a composite prediction mode, unidirectional inter-frame prediction can be in a single-reference mode or a composite reference mode. Bidirectional inter-frame prediction can be in composite mode only because bidirectional inter-frame prediction involves at least two reference blocks.

[0174] Block partitioning

[0175] Figure 18 An exemplary partitioning tree for block partitioning is shown. VP9 uses a 4-way partitioning tree ranging from 64×64 levels to 4×4 levels, with additional restrictions on blocks of 8×8 and below. This is in Figure 18 As shown in the diagram, a partition designated as R can be recursively divided as follows: the same partition tree is repeated at lower sizes until the lowest 4×4 level is reached. As illustrated, while AV1 expands the partition tree to a 10-way structure, it also increases the maximum size (called a superblock in VP9 / AV1) to start at 128×128. This includes 4:1 / 1:4 rectangular partitions that did not exist in VP9. No rectangular partition can be further subdivided. Furthermore, AV1 adds more flexibility for using partitions below 8×8 levels. For example, 2×2 chroma inter-frame prediction is now possible in some cases.

[0176] In HEVC, coding tree units (CTUs) are partitioned into multiple coding units (CUs) using a quadtree structure represented as a coding tree to accommodate various local features. A CU can also be considered a block, including prediction blocks or coding blocks. The decision of whether to use inter-frame (temporal) prediction or intra-frame (spatial) prediction to code an image region is made at the CU level. Depending on the PU partitioning type, each CU can be further divided into one, two, or four prediction units (PUs). Within a PU, the same prediction processing is applied, and relevant information is sent to the decoder based on the PU. After obtaining residual blocks by applying a prediction process based on the PU partitioning type, the CUs can be partitioned into multiple transform units (TUs) according to another quadtree structure similar to the coding tree of the CU. The HEVC structure has multiple partitioning concepts including CUs, PUs, and TUs. In HEVC, for inter-frame prediction blocks, CUs or TUs can only be squares, while PUs can be squares or rectangles. In HEVC, a coding block can be further partitioned into four square sub-blocks, and a transform is performed on each sub-block (i.e., TU). Each TU can be further recursively segmented (using quadtree segmentation) into smaller TUs, a process known as residual quadtree segmentation (RQT). At image boundaries, HEVC employs implicit quadtree segmentation, ensuring that blocks remain quadtree-segmented until they are sized to fit the image boundaries.

[0177] A block localization structure using a quadtree (QT) plus a binary tree (BT) can exist. In HEVC, the CTU can be partitioned into CUs using a quadtree structure represented as a coding tree to accommodate various local features. The decision on whether to use inter-image (temporal) prediction or intra-image (spatial) prediction to encode the image region is made at the CU level. In some embodiments, each CU can be further partitioned into one, two, or four PUs based on the PU partitioning type. Within a PU, the same prediction processing can be applied, and relevant information is sent to the decoder based on the PU. After obtaining the residual block by applying the prediction process based on the PU partitioning type, the CU can be partitioned into transform units (TUs) according to another quadtree structure similar to the coding tree of the CU. The HEVC structure can have multiple partitioning concepts, including CU, PU, ​​and TU.

[0178] Block partitioning can use a ternary tree (TT) structure. Figure 19 An example of ternary tree partitioning is shown. In VVC, the Multi-Type Tree (MTT) structure can add a horizontal center-side ternary tree and a vertical center-side ternary tree on top of the QTBT, as shown below. Figure 19As shown (a and b respectively). Ternary tree partitioning can complement quadtrees and binary trees. While quadtrees and binary trees are partitioned along the block center, ternary tree partitioning can capture objects located at the block center. For ternary tree partitioning, the width and height of the proposed ternary tree partitions can be powers of 2, thus requiring no additional transformations. The purpose of the two-level tree design is to reduce complexity; the complexity of traversing a tree is... ,in T Indicates the number of partition types. D It's the depth of the tree.

[0179] Merging Mode with Motion Vector Difference (MMVD)

[0180] Besides using implicitly derived motion information directly in the merging mode for generating prediction samples in the current CU, a Merge Mode with Motion Vector Difference (MMVD) can also be used. Immediately after sending the skip flag and merge flag, the MMVD flag is written to the bitstream to specify whether the MMVD mode is used for the CU. In MMVD, after selecting a merge candidate, the selected merge candidate can be further refined using the written MMVD information. This further information includes merge candidate flags, an index indicating motion amplitude, and an index indicating motion direction. In MMVD mode, one of the top two candidates in the merge list is selected as the MV basis. The merge candidate flag is written to the bitstream to indicate which candidate is used.

[0181] Figure 20a An example of a merged pattern (MMVD) search point with motion vector difference is shown. Figure 20b Another example is shown with a merged MMVD search point having motion vector differences. The distance index indicates motion amplitude information and indicates a predefined offset from the distance starting point. Figure 20a and Figure 20b As shown, the offset is added to the horizontal or vertical component of the starting MV. The relationship between the distance index and the predefined offset is shown in Table 2:

[0182]

[0183] Table 2 - Relationship between Distance Index and Predefined Offset

[0184] The direction index indicates the direction of the MVD relative to the starting point. As shown in Table 3, the direction index can represent four directions. It is worth noting that the meaning of the MVD symbol can vary depending on the information of the starting MV. When the starting MV is a unidirectional or bidirectional predictive MV, where both lists point to the same side of the current image (i.e., both reference POCs are greater than or less than the current image's POC), the symbols in Table 2 indicate the symbol for the MV offset added to the starting MV. The POC can be a value that identifies when an image is displayed. In some embodiments, a higher POC indicates that more frames have been displayed before that image.

[0185] When the starting MV is a bidirectional predicted MV, where the two MVs point to different sides of the current image (i.e., one reference POC is greater than the current image's POC, and the other reference POC is less than the current image's POC), and the difference in POCs in List 0 is greater than the difference in POCs in List 1, the signs in Table 3 indicate the sign of the MV offset added to the List 0 MV component of the starting MV, and the signs of the List 1 MVs have the opposite value. Otherwise, if the difference in POCs in List 1 is greater than the difference in POCs in List 0, then the signs in Table 3 specify the sign of the MV offset added to the List 1 MV component of the starting MV, and the signs of the List 0 MVs have the opposite value.

[0186]

[0187] Table 3 - Symbols for MV offsets indicated by direction indices.

[0188] MVD can be scaled based on the difference in POC in each direction. If the difference in POC is the same in both lists, no adjustment is needed. Otherwise, if the difference in POC in reference list 0 is greater than the difference in POC in reference list 1, the MVD of reference list 1 is adjusted. If the difference in POC in reference list 1 is greater than the difference in POC in list 0, the MVD of list 0 can be adjusted in the same way. If the initial MV is unidirectionally predicted, the MVD is added to the available MV.

[0189] Symmetric MVD coding

[0190] In one example, a symmetric MVD mode can be used in VVC. This can complement normal unidirectional predictive MVD signaling and bidirectional predictive MVD signaling. The symmetric MVD mode can be used for bidirectional MVD signaling. In one embodiment, in the symmetric MVD mode, instead of using signal transmission, motion information of the MVD of both list-0 and list-1 is derived, including reference image indices and list-1.

[0191] The decoding process of symmetric MVD mode can include:

[0192] At the slice level, the derivation of variables BiDirPredFlag, RefIdxSymL0, and RefIdxSymL1 is as follows:

[0193] If mvd_l1_zero_flag is 1, then BiDirPredFlag is set to 0.

[0194] Otherwise, if the most recent reference image in list-0 and the most recent reference image in list-1 form a forward and backward reference image pair or a backward and forward reference image pair, then BiDirPredFlag is set to 1, and both list-0 and list-1 reference images are short-term reference images. Otherwise, BiDirPredFlag is set to 0.

[0195] 2. At the CU level, if bidirectional predictive coding is performed on the CU and BiDirPredFlag is equal to 1, then the symmetric mode flag indicating whether to use symmetric mode is explicitly written into the bitstream.

[0196] When the symmetry mode flag is true, only mvp_l0_flag, mvp_l1_flag, and MVD are explicitly written to the bitstream. The reference indices of list-0 and list-1 are set to equal the reference image pair, respectively. MVD1 can be set to equal to ( MVD0).

[0197] MVD coding in CWG-B018 can be another example. In AV1, for each coded block between frames, if the current block's mode is not skip mode but inter-frame coding mode, another flag is written to the bitstream to indicate whether the current block uses a single reference mode or a composite reference mode. In single reference mode, a prediction block is generated from a single motion vector, while in composite reference mode, a prediction block is generated by a weighted average of two prediction blocks derived from two motion vectors.

[0198] For a single reference, the following patterns can be written into the bitstream:

[0199] NEARMV uses one of the motion vector predictors (MVPs) in a list indicated by the DRL (Dynamic Reference List) index.

[0200] NEWMV - Uses one of the motion vector predictors (MVPs) in a list written by the DRL index as a reference and applies an increment to the MVP (e.g., using MVD).

[0201] GLOBALMV - Uses motion vectors based on frame-level global motion parameters.

[0202] For composite reference mode, the following modes can be written into the bitstream:

[0203] NEAR_NEARMV - One of the motion vector predictors (MVPs) that use a list written by the DRL index.

[0204] NEAR_NEWMV - Uses one of the motion vector predictors (MVPs) in the list written by the DRL index as the reference MV and sends an incremental MV for the second MV.

[0205] NEW_NEARMV - Uses one of the motion vector predictors (MVPs) in the list written by the DRL index as the reference MV and sends the incremental MV for the first MV.

[0206] NEW_NEWMV - Uses one of the motion vector predictors (MVPs) in the list written by the DRL index as the reference MV and sends incremental MVs for both MVs.

[0207] GLOBAL_GLOBALMV - Uses the MV for each reference based on frame-level global motion parameters.

[0208] Inter-frame prediction mode signaling

[0209] To use the signal transmission inter-frame prediction mode, the skip_mode flag is first written to the bitstream to determine if skip mode is applicable to the current coding block. When skip mode is enabled, composite inter-frame prediction is performed using two reference frames and translation motion. Both the MV and the reference frames are implicitly derived (as the first entry in the DRL) and are not written to the bitstream. When the mode is not enabled, reference frame information is written to the bitstream.

[0210] When a reference frame is transmitted using signals, the `comp_mode` flag is first written to the bitstream to indicate whether one or two reference frames are used for inter-frame prediction. This flag can only be written to the bitstream if both the width and height of the coded block are greater than or equal to 8. In one example, for a coded block with a width or height less than 8, only a single reference frame is used for inter-frame prediction. When compound prediction is applied, the `comp_ref_type` syntax is further written to the bitstream to indicate whether the compound prediction is unidirectional or bidirectional. Furthermore, when applying unidirectional compound prediction mode, one of four predefined combinations of reference frames is written to the bitstream. When applying bidirectional compound prediction mode, the reference frame indices of the forward and backward reference frames are written to the bitstream, respectively. For single-reference prediction, only one of the seven possible reference frames is written to the bitstream.

[0211] After transmitting the reference frame using signals, the Y-mode (YMode) value is derived through multiple syntaxes to indicate which mode will be applied to MV prediction and signaling. Table 4 below lists the mapping between YMode and MV prediction.

[0212]

[0213] Table 4: Mapping between YMode and MV Prediction Mode

[0214] After deriving YMode, the Dynamic Reference List_Mode (drl_mode) syntax is further written into the bitstream to indicate which candidate in the DRL will be used for MV prediction. Then, when one of NEWMV, NEAREST_NEWMV, NEW_NEARESTMV, NEAR_NEWMV, NEW_NEARMV, or NEW_NEWMV is applied, MVD is further written into the bitstream. When YMode equals NEW_NEARESTMV, NEW_NEARMV, or NEW_NEWMV, MVD is written into the bitstream for the MV associated with the first reference frame. When YMode equals NEAREST_NEWMV, NEAR_NEWMV, or NEW_NEWMV, MVD is written into the bitstream for the MV associated with the second reference frame.

[0215] After using the signaling to transmit MV prediction mode, the composite inter-intra mode can be further written into the bitstream. In one example, the composite inter-intra mode can only be written into the bitstream when a single reference frame inter-prediction is applied, and when the block size is greater than or equal to 8×8 but less than or equal to 32×32. The signaling for the composite inter-intra mode includes an inter-intra flag indicating whether the composite inter-intra mode is applied.

[0216] When applying the composite inter-intra mode, the intra-mode index `interintra_mode` is further written into the bitstream to indicate which of the following modes—DC_PRED, V_PRED, H_PRED, and SMOOTH—will be used to perform intra-prediction. Subsequently, the `wedge_interintra` flag is written into the bitstream to indicate whether a wedge-based inter-intra prediction mode should be applied. When applying a Wedge-based inter-prediction mode, the `wedge_index` syntax is written into the bitstream, indicating the Wedge partitioning mode to be applied in Wedge-based inter-intra prediction. After writing the composite inter-intra mode into the bitstream, the `motion_mode` is further derived using signal transmission or implicitly to indicate which of the two modes—Overlapped Block Motion Compensation (OBMC) and Twisted Motion Compensation—should be applied. After sending the motion_mode signal, the composite prediction type is further written into the bitstream to indicate which composite prediction mode to apply. Composite prediction modes include Wedge-based composite prediction, differential modulation composite prediction, and distance-based composite prediction. Finally, if applicable, the interpolation filter index is written into the bitstream to indicate the selection of the interpolation filter when applying translational motion compensation.

[0217] When two reference indices are written to the bitstream, an inter-frame prediction mode called Joint_New Motion Vector (JOINT_NEWMV) can be used, i.e., composite inter-frame prediction. The inter-frame prediction mode embodiments described herein can also be applied to the JOINT_NEWMV mode. Incremental MVs for multiple reference lists (e.g., reference list 0 and reference list 1) are jointly written to the bitstream, and incremental MVs for reference list 0 and reference list 1 are also jointly written to the bitstream. Therefore, only one incremental MV called Joint_Incremental_Motion Vector (joint_delta_mv) is written to the bitstream and sent to the decoder, and the incremental MVs for reference list 0 and reference list 1 are derived from joint_delta_mv. However, when two reference indices are written to the bitstream, the inter-frame prediction mode JOINT_NEWMV can be applied, which may limit higher coding gain. The embodiments described below illustrate and demonstrate applications with a single reference index.

[0218] For composite reference mode, if the point of view (POC) of both reference frames of a motion vector pair is greater than or less than the POC of the current frame, then the directions of the two reference frames can be the same. Otherwise, if the POC of one reference frame is greater than the POC of the current frame, while the POC of the other reference frame is less than the POC of the current frame, then the directions of the two reference frames are different.

[0219] The proposed embodiments can be used individually or in any combination in any order. Furthermore, each of the method (or embodiment), encoder, and decoder can be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer-readable medium. Hereinafter, the term 'block' can be interpreted as a prediction block, coding block, or coding unit, i.e., a CU. The orientation of the reference frame can be determined by whether the reference frame precedes or follows the current frame in the display order.

[0220] For a single-reference example, only one reference frame is written to the bitstream. Exemplary inter-frame prediction modes (e.g., DERIVED_NEWMV and / or DERIVED_NEARMV) may exist. These modes are applied to indicate whether another motion vector pointing to a different reference frame can be derived using the motion vector written / predicted for the current reference frame. DERIVED_NEWMV could be an inter-frame prediction mode where the motion vector difference (MVD) is written to the bitstream. In this example, signaling for one reference frame is used for two reference frames. DERIVED_NEARMV could be an inter-frame prediction mode that selects the motion vector difference (MVD) based on what neighboring blocks are using. This may have less overhead but may also be less accurate. In this example, the neighboring block mode of one reference frame is used for two reference frames.

[0221] In one embodiment, a reference frame index indicating which reference frame is used for the current block can be written to the bitstream. This writing can occur before newly added inter-frame prediction modes. In one embodiment, DERIVED_NEWMV and / or DERIVED_NEARMV can be written to the bitstream along with NEWMV, NEARMV, and / or GLOBALMV. These are inter-frame prediction modes that use two reference frames instead of a single reference frame. NEWMV can write motion vectors to the bitstream, while NEARMV can depend on neighboring blocks.

[0222] In one embodiment, signaling for MVD is the same between DERIVED_NEWMV and NEWMV modes. In one embodiment, no MVD is written to the bitstream for DERIVED_NEARMV. In one embodiment, DERIVED_NEWMV and / or DERIVED_NEARMV is written to the bitstream only if there are at least two reference frames available for selection by the current image in the reference frame buffer, and at least one of the at least two reference frames is displayed before the current frame in order of appearance, and at least one of the at least two reference frames is displayed after the current frame in order of appearance.

[0223] The joint reference case will be further described below. NEARMV directly uses one of the motion vector predictors (MVPs) in the list indicated by the DRL (Dynamic Reference List) index, without using any MVD. NEWMV uses one of the motion vector predictors (MVPs) in the list written by the DRL index as a reference and applies an increment to the MVP (e.g., using MVD). GLOBALMV uses motion vectors based on frame-level global motion parameters. The term "NEAR" above can refer to MV prediction using a reference MV without using MVD as a general merging mode, while the term "NEW" refers to MV prediction involving using a reference MV and offsetting it with a written MVD as an MMVD mode. For composite inter-frame prediction, the reference-based motion vectors and motion vector increments described above, even if they can be correlated, can often be different or independent between the two references, and this correlation can be used to reduce the amount of information required to write the two motion vector increments. The dynamic reference list (DRL) described above can be used to store a set of indexed motion vectors that are dynamically stored and considered as candidate motion vector predictors. In some exemplary embodiments, optical flow-based methods can be used to refine motion vectors (MVs) at the sub-block level for composite prediction. Specifically, the optical flow equations can be applied to a formulaic least-squares problem, thereby deriving the refined motion from the gradients of composite inter-frame prediction samples. Utilizing these refined motions, the MVs of each sub-block can be refined within the prediction block, which can enhance inter-frame prediction quality. Some coding features can be extensions of the bidirectional optical flow (BDOF) concept, as BDOF supports MV refinement when there are arbitrary temporal distances between two reference blocks and the current block. In various embodiments of this disclosure, "block" can refer to a prediction block, coding block, transform block, or coding unit (CU).

[0224] Returning to the example of deriving references, which assists in deriving the MVD of multiple reference signals using a single reference signal. In one embodiment, when a block uses a single reference frame (i.e., comp_mode equals signal_reference(SINGLE_REFERENCE)) and the inter-frame prediction mode is DERIVED_NEARMV (or DERIVED_NEWMV), a new MV can be derived from the mirrored predicted or written MV. This derived MV can point to another reference frame in a different direction than the current reference frame. In other words, there may only be a signal for a single reference frame, but this signal is used to identify the MV of another reference frame.

[0225] In one embodiment, the derived MV may have the same magnitude but opposite sign as the predicted (or written) MV. In one embodiment, the time distance from the reference frame to the current frame pointed to by the derived MV may be the same as the time distance between the written reference frame and the current frame. In another embodiment, the derived MV can be used to generate another prediction block. For example, when there are two frames (e.g., a reference frame and a derived frame), there may be two prediction blocks. This prediction block, along with the original prediction block, can be derived from the predicted (or written) MV and the written reference frame. This can be used to derive the final prediction block using a weighted average.

[0226] In one embodiment, the reference frame pointed to by the derived MV may have a different orientation than the written reference frame. The time distance to the current frame may differ. The time distance can be modified. In one embodiment, when the time distance from the written reference frame to the current frame differs from the time distance from the reference frame associated with the derived MV to the current frame, the derived MV can be scaled based on the time distance.

[0227] Figure 21 A flowchart of a method according to an exemplary embodiment of the present disclosure is shown. In block 2102, an encoded video stream is received. The encoded video stream includes syntax flags indicating a specific inter-frame coding mode. The inter-frame coding mode may include DERIVED NEWMV and / or DERIVED_NEARMV. In block 2104, a single reference frame of the current frame in the encoded video stream is received. In block 2106, in response to the syntax flags indicating the application of a specific inter-frame coding mode. The specific inter-frame coding mode may include a determination regarding whether to use a first motion vector of the current reference frame to derive a second motion vector pointing to a different reference frame, as described in block 2108. In block 2108, in response to determining that the first motion vector of the reference frame of the current frame is used to derive the second motion vector of the different reference frame, a second motion vector is determined based on the syntax flags and the first motion vector. In block 2110, the encoded video stream is decoded based on the derived second motion vector.

[0228] Although described as a reference frame, a reference frame can also include reference frame pairs. In one embodiment, if multiple motion vectors point to a reference frame or a reference frame pair, the value with the smallest difference referenced by these motion vectors can be labeled as the score of the reference frame or the reference frame pair. In another embodiment, if no motion vector points to a reference frame or a reference frame pair, the score of that reference frame or reference frame pair is labeled as the maximum allowed value. In another embodiment, if multiple reference frames or multiple reference frame pairs have the same score, the sorting order of these reference frames or reference frame pairs is the same as the scanning order of the spatial reference motion information and / or the temporal reference motion information. In another embodiment, if multiple reference frames or multiple reference frame pairs have the same score, the sorting order of these reference frames or reference frame pairs depends on the frequency of occurrence of these reference frames or reference frame pairs used in the spatial reference motion information and / or the temporal reference motion information.

[0229] In one embodiment, all allowed unidirectional composite reference frame pairs and bidirectional composite reference frame pairs are sorted together using a template matching (TM) method, and the indices of the reference frame pairs of the current block are written to the bitstream in this sorting order. In another embodiment, all allowed individual reference frames are sorted together using a TM method, and the indices of the reference frames of the current block are written to the bitstream in this sorting order.

[0230] In one embodiment, a method for reordering reference frames for each block using template matching (TM) includes comparing a template of the current block with a template of a reference block to obtain motion information, wherein the motion information includes spatial reference motion information or temporal reference motion information; calculating the difference between the template of the current block and the template of the reference block; determining a score for an associated reference frame based on the calculated difference; and reordering the reference frames based on the determined scores. The reference frames also include pairs of reference frames. The TM includes motion vector derivation on the decoder side for refining the motion information of the current block. The reordering method further includes sorting the available reference frames based on the score of each reference frame in the reference frames. When multiple reference frames have equal scores, the sorting corresponds to the scanning order of the spatial reference motion information or temporal reference motion information. When multiple reference frames have equal scores, the sorting is based on the frequency of occurrence of these reference frames used in the spatial reference motion information or temporal reference motion information. The calculated differences include at least one of the following: Sum of Absolute Differences (SAD), Sum of Squared Differences (SSD), Mean Squared Error (MSE), or Sum of Absolute Transform Differences (SATD). The template includes either the top adjacent block or the left adjacent block. Spatial reference motion information includes one or more spatial motion vectors. Temporal reference motion information includes one or more temporal motion vectors. When multiple motion vectors point to one of the reference frames, the motion vector with the smallest difference is used to determine the score. When no motion vector points to one of the reference frames, that score is determined as the maximum allowed value. Allowed unidirectional composite reference frames and bidirectional composite reference frames are sorted together using a TM, where the index of the reference frame of the current block is written into the bitstream. Allowed individual reference frames are sorted together using a TM, where the index of the reference frame of the current block is written into the bitstream.

[0231] The embodiments in this disclosure can be used individually or in any combination in any order. Furthermore, each of the methods (or embodiments), encoders, and decoders can be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer-readable medium. The term "block" can include a prediction block, an encoding block, or an encoding unit, i.e., a CU. The embodiments in this disclosure can be applied to luma blocks or chroma blocks.

[0232] The above-described technology can be implemented as computer software that uses computer-readable instructions and is physically stored in one or more computer-readable media. For example, Figure 22 A computer system (2200) suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0233] Computer software can be coded using any suitable machine code or computer language. Any suitable machine code or computer language can be assembled, compiled, linked, or similarly processed to create code containing instructions that can be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or through decoding, microcode, etc.

[0234] The instructions can be executed on various types of computers or their components, including personal computers, tablets, servers, smartphones, gaming devices, and Internet of Things (IoT) devices.

[0235] Figure 22 The components of the computer system (2200) shown are exemplary in nature and are not intended to impose any limitation on the scope of use or functionality of the computer software implementing the embodiments of this disclosure. The configuration of the components should also not be construed as having any dependency or requirement relating to any one or a combination of components shown in the exemplary embodiments of the computer system (2200).

[0236] The computer system (2200) may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input from one or more human users through, for example, tactile input (e.g., keystrokes, swipes, movement of a data glove), audio input (e.g., speech, clapping), visual input (e.g., gestures), and olfactory input (not depicted). The human-machine interface device may also be used to capture certain media that are not necessarily directly related to human conscious input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographs acquired from a still image camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video), etc.

[0237] The input human-machine interface device may include one or more of the following (only one of each is shown): keyboard (2201), mouse (2202), touchpad (2203), touch screen (2210), data glove (not shown), joystick (2205), microphone (2206), scanner (2207), camera (2208).

[0238] The computer system (2200) may also include certain human-machine interface output devices. Such human-machine interface output devices may, for example, stimulate the senses of one or more human users through tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include tactile output devices (e.g., tactile feedback from touchscreens (2210), data gloves (not shown), or joysticks (2205), but may also be tactile feedback devices that are not input devices), audio output devices (e.g., speakers (2209), headphones (not shown)), visual output devices (e.g., screens (2210) including CRT screens, LCD screens, plasma screens, OLED screens, each screen may or may not have touchscreen input functionality, each screen may or may not have tactile feedback functionality, some of which are capable of outputting two-dimensional visual output or more than three-dimensional output through devices such as stereoscopic image output, virtual reality glasses (not depicted), holographic displays and smoke boxes (not depicted), and printers (not depicted).

[0239] The computer system (2200) may also include human-accessible storage devices and their associated media: such as optical media including CD / DVD ROM / RW (2220) having media such as CD / DVD (2221), finger drives (2222), removable hard disk drives or solid-state drives (2223), conventional magnetic media such as magnetic tapes and floppy disks (not shown), devices based on dedicated ROM / ASIC / PLD such as security dongles (not shown), etc.

[0240] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not cover transmission media, carrier waves, or other transient signals.

[0241] The computer system (2200) may also include an interface (2254) to one or more communication networks (2255). The network may be, for example, a wireless network, a wired network, or an optical network. The network may further be a local area network, a wide area network, a metropolitan area network, a vehicle and industrial network, a real-time network, a latency-tolerant network, etc. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., cable or wireless wide area digital television networks including cable television, satellite television, and terrestrial broadcast television, vehicle and industrial television including CANBus, etc. Some networks typically require an external network interface adapter (e.g., a USB port of the computer system (2200)) to connect to some general-purpose data port or peripheral bus (2249); other network interfaces are typically integrated into the core of the computer system (2200) by connecting to a system bus (e.g., an Ethernet interface connected to a PC computer system or a cellular network interface connected to a smartphone computer system). The computer system (2200) can use any of these networks to communicate with other entities. Such communication can be one-way receiving (e.g., broadcast television), one-way transmitting (e.g., CANbus connected to certain CANbus devices), or bidirectional, such as connecting to other computer systems using a local area network (LAN) or wide area network (WAN) digital network. As mentioned above, certain protocols and protocol stacks can be used on each of those networks and network interfaces.

[0242] The aforementioned human-machine interface device, human-machine accessible storage device, and network interface can be attached to the kernel (2240) of the computer system (2200).

[0243] The core (2240) may include one or more central processing units (CPU) (2241), graphics processing units (GPUs) (2242), dedicated programmable processing units in the form of field-programmable gate areas (FPGAs) (2243), hardware accelerators (2244) for certain tasks, graphics adapters (2250), etc. These devices, as well as read-only memory (ROM) (2245), random access memory (2246), and internal mass storage (2247) such as internal non-user-accessible hard disk drives, SSDs, etc., may be connected via a system bus (2248). In some computer systems, the system bus (2248) may be accessed in the form of one or more physical plugs to allow for expansion with additional CPUs, GPUs, etc. Peripheral devices may be directly connected to the core's system bus (2248) or connected to the core's system bus (2248) via a peripheral bus (2249). For example, a screen (2210) may be connected to a graphics adapter (2250). Peripheral bus architectures include PCI, USB, etc.

[0244] The CPU (2241), GPU (2242), FPGA (2243), and accelerator (2244) can execute certain instructions, which can be combined to form the aforementioned computer code. This computer code can be stored in ROM (2245) or RAM (2246). Transient data can also be stored in RAM (2246), while permanent data can be stored, for example, in internal mass storage (2247). Fast storage and retrieval to any storage device can be achieved using a cache, which can be closely associated with one or more CPUs (2241), GPUs (2242), mass storage (2247), ROM (2245), RAM (2246), etc.

[0245] Computer-readable media may have computer code thereon that performs various computer-implemented operations. The media and computer code may be media and computer code specifically designed and constructed for the purposes of this disclosure, or the media and computer code may be of a type known and available to those skilled in the art of computer software.

[0246] As a non-limiting example, a computer system having an architecture (2200), particularly a kernel (2240), can be made functional by one or more processors (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software contained in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage as described above, and memory of some non-transitory kernel (2240), such as internal kernel mass storage (2247) or ROM (2245). Software implementing various embodiments of this application can be stored in such means and executed by the kernel (2240). Depending on specific needs, the computer-readable medium may include one or more storage devices or chips. The software can cause the kernel (2240), particularly the processors therein (including CPUs, GPUs, FPGAs, etc.), to execute a particular process or a particular portion of a particular process described herein, including defining data structures stored in RAM (2246) and modifying such data structures according to a software-defined process. Additionally or alternatively, the computer system may provide functionality through hard-wired or otherwise embodied logic in circuitry (e.g., the accelerator (2244)), which may replace or operate with the software to perform a particular process or a particular portion of a particular process described herein. Where appropriate, references to software may include logic, and vice versa. Where appropriate, references to computer-readable media may include circuitry (e.g., an integrated circuit (IC)) storing software for execution, circuitry embodying logic for execution, or both. This application includes any suitable combination of hardware and software.

[0247] Although several exemplary embodiments have been described in this application, modifications, substitutions, and various equivalent alternatives that fall within the scope of this application are possible. Therefore, it should be understood that those skilled in the art will be able to design numerous systems and methods that, while not expressly shown or described herein, embody the principles of this disclosure and thus fall within its spirit and scope.

[0248] Appendix A: Abbreviations

[0249] ALF: Adaptive Loop Filter

[0250] AMVP: Advanced Motion Vector Prediction

[0251] APS: Adaptation Parameter Set

[0252] ASIC: Application-Specific Integrated Circuit

[0253] AV1: Open Media Alliance Video 1

[0254] AV2: Open Media Alliance Video 2

[0255] BCW: CU-level weighted bidirectional prediction

[0256] BM: Two-sided matching

[0257] BMS: Benchmark Set

[0258] CANBus: Controller Area Network Bus

[0259] CC-ALF: Cross-component adaptive loop filter

[0260] CCSO: Cross-component sample offset

[0261] CD: Compressed optical disc

[0262] CDEF: Constrained Direction Enhancement Filter

[0263] CDF: Cumulative Density Function

[0264] CfL: Predicting chromaticity from luminance

[0265] CIIP: Combined Intra-Inter-Frame Prediction

[0266] CPUs: Multiple central processing units

[0267] CRT: Cathode Ray Tube

[0268] CTB: Coded Tree Block

[0269] CTU: Coding Tree Unit

[0270] CTUs: Multiple Coding Tree Units

[0271] CU: Encoding Unit

[0272] DMVR: Decoder-side motion vector refinement

[0273] DPB: Decoded Image Buffer

[0274] DPS: Decoding Parameter Set

[0275] DVD: Digital Video Disc

[0276] FPGA: Field Programmable Gate Domain

[0277] GBI: Generalized Two-Way Forecasting

[0278] GOPs: Multiple Image Groups

[0279] GPUs: Multiple graphics processing units

[0280] GSM: Global System for Mobile Communications

[0281] HDR: High Dynamic Range

[0282] HEVC: High-Efficiency Video Coding

[0283] HRD: Hypothetical Reference Decoder

[0284] IBC (or IntraBC): Intra-block copy

[0285] IC: Integrated Circuit

[0286] ISP: Intra-Frame Sub-Partition

[0287] JEM: Joint Exploration Model

[0288] JVET: Joint Video Exploration Team

[0289] LAN: Local Area Network

[0290] LCD: Liquid Crystal Display

[0291] LR: Loop Recovery Filter

[0292] LSO: Local Sampling Offset

[0293] LTE: Long Term Evolution

[0294] MMVD: Merging Mode with Motion Vector Difference

[0295] MPM: Most Likely Pattern

[0296] MV: Motion Vector

[0297] MVD: Motion Vector Difference

[0298] MVP: Motion Vector Predictor

[0299] OLED: Organic Light Emitting Diode

[0300] PBs: Multiple prediction blocks

[0301] PCI: Peripheral Component Interconnect

[0302] PDPC: Location-Related Prediction Combination

[0303] PLD: Programmable Logic Device

[0304] POC: Image Sequence Counting

[0305] PPS: Image Parameter Settings

[0306] PU: Prediction Unit

[0307] PUs: Multiple prediction units

[0308] ROM: Random Access Memory

[0309] ROM: Read-Only Memory

[0310] RPS: Reference Image Set

[0311] SAD: Sum of Absolute Differences

[0312] SAO: Sample Adaptive Offset

[0313] SB: Superblock

[0314] SCC: Screen Content Encoding

[0315] SDP: Semi-decoupling partitioning

[0316] SDR: Standard Dynamic Range

[0317] SDT: Semi-decoupled tree

[0318] SEI: Supplemental Enhancement Information

[0319] SNR: Signal-to-noise ratio

[0320] SPS: Sequence Parameter Settings

[0321] SSD: Solid State Drive

[0322] SST: Semi-separated tree

[0323] TM: Template Matching

[0324] TU: Transformation Unit

[0325] TUs: Multiple transformation units

[0326] USB: Universal Serial Bus

[0327] VPS: Video Parameter Settings

[0328] VUI: Video Availability Information

[0329] VVC: Multi-functional Video Coding

[0330] WAIP: Wide-angle Intra-frame Prediction

Claims

1. A method for decoding video streams, characterized in that, The method includes: The video stream is received, the video stream includes a syntax flag, the syntax flag characterizes the inter-frame coding mode, and indicates whether the current block in the current frame within the video stream is encoded in a single reference frame inter-frame prediction mode, and when the single reference frame inter-frame prediction mode is used for encoding, indicates whether a first motion vector of a first reference frame is used to derive a second motion vector of a second reference frame. Receive the first reference frame of the current frame in the video stream; Based on the value of the syntax flag, determine whether the first motion vector of the first reference frame is used to derive the second motion vector of the second reference frame; In response to the value of the syntax flag indicating that the current block adopts the single-reference-frame inter-prediction mode coding, and the first motion vector of the first reference frame is used to derive the second motion vector of the second reference frame, wherein the derivation is based on the value of the syntax flag and the first motion vector of the first reference frame; and The current block is decoded based on video blocks in the first reference frame and the second reference frame, which are respectively pointed to by the first motion vector and the derived second motion vector.

2. The method according to claim 1, characterized in that, The first motion vector points to the first reference frame, the second motion vector points to a second reference frame different from the first reference frame, and the motion vector difference (MVD) provides a prediction based on the first motion vector and the second motion vector.

3. The method according to claim 1, characterized in that, The derivation includes inter-frame coding modes, wherein the inter-frame coding modes include the Derived_New Motion Vector (DERIVED_NEWMV) mode or the Derived_Nearby Motion Vector (DERIVED_NEARMV) mode.

4. The method according to claim 3, characterized in that, The DERIVED_NEWMV mode includes deriving the second motion vector directly from the first reference frame.

5. The method according to claim 3, characterized in that, The DERIVED_NEARMV mode includes deriving the second motion vector from adjacent frames of the first reference frame.

6. The method according to claim 5, characterized in that, For the DERIVED_NEARMV mode, motion vector difference (MVD) is not used for signal transmission.

7. The method according to claim 3, characterized in that, Also includes: In response to the syntax flag, it indicates which of the inter-frame coding modes to apply, wherein the second motion vector is derived from the first motion vector when it indicates that the first motion vector of the first reference frame can derive the second motion vector.

8. The method according to claim 7, characterized in that, Also includes: Based on the indicated inter-frame coding mode, motion vector difference (MVD) is used to transmit signals.

9. The method according to claim 1, characterized in that, Also includes: A reference frame index is sent using a signal, the reference frame index indicating whether the current block uses the first reference frame or the second reference frame for the inter-frame prediction mode.

10. The method according to claim 9, characterized in that, The reference frame index is sent using a signal when there are at least two reference frames available for the current frame in the reference frame buffer, and one of the at least two reference frames is displayed before the current frame in the display order, while the other of the at least two reference frames is displayed after the current frame in the display order.

11. The method according to claim 1, characterized in that, The orientation of the reference frame is based on whether the reference frame is displayed before or after the current frame in terms of display order.

12. The method according to claim 11, characterized in that, The direction of the first reference frame is different from the direction of the second reference frame.

13. The method according to claim 1, characterized in that, The two reference frames are in the same direction when the image sequence count (POC) of both reference frames of a motion vector pair is greater than or less than the POC of the current frame.

14. The method according to claim 1, characterized in that, The pair of reference frames are in different orientations when the image sequence count (POC) of one reference frame in a pair is greater than the POC of the current frame and the POC of the other reference frame in the pair is less than the POC of the current frame.

15. The method according to claim 1, characterized in that, The direction of the second reference frame to which the derived second motion vector points is different from the direction of the reference frame transmitted using the signal or the direction of the first reference frame.

16. An apparatus for decoding video streams, characterized in that, The device includes: Memory, used to store instructions; and A processor that communicates with the memory, wherein when the processor executes the instructions, the processor is configured to cause the apparatus to perform the method for decoding a video stream as described in any one of claims 1 to 15.

17. A non-transitory computer-readable storage medium having instructions stored thereon, characterized in that, When the instructions are executed by a processor, the instructions are configured to cause the processor to perform the method for decoding a video stream as described in any one of claims 1 to 15.

Images