Methods, apparatus and computer-readable storage media for encoding and decoding video data

Abstract

This application relates to methods, apparatus, and computer-readable storage media for encoding and decoding video data. The method includes: determining whether a first prediction mode can be applied based on a picture sequence count (POC) parameter associated with a picture of a current video block, in which the predicted samples of the current video block are refined at least based on gradients in different directions; performing a conversion between the current video block and a bitstream based on the determination; and associating a first reference picture (R0) and a second reference picture (R1) with the current picture including the current video block, where τ0 is the difference between the picture sequence count of the current picture and the picture sequence count of the first reference picture (τ0 = ...). POC(current) - POC(R0)), where τ1 is the difference between the image sequence count of the second reference image and the image sequence count of the current image (τ1 = POC(R1) - POC(current)). In response to abs(abs(τ0) - abs(τ1)) ≥ T0, the first prediction mode is avoided, where T0 is an integer and greater than 0. Alternatively, in response to abs(τ0) > abs(τ1) × T0 or abs(τ1) > abs(τ0) × T0, the first prediction mode is avoided, where T0 is an integer and T0 = 1.

Description

[0001] This application is a divisional application of the invention patent application with application number 201980072654.6, application date November 5, 2019, and invention title "Refined Inter-Frame Prediction in Video Processing".

[0002] Cross-references to related applications

[0003] Pursuant to applicable patent law and / or the rules applicable to the Paris Convention, this application promptly claims priority and benefit to International Patent Application No. PCT / CN2018 / 113928, filed November 5, 2018. For all purposes under U.S. law, the entire disclosure of the foregoing application is incorporated herein by reference as part of this application disclosure. Technical Field

[0004] This patent document relates to video encoding and decoding technologies, devices, and systems. Background Technology

[0005] Despite advancements in video compression, digital video still accounts for the largest share of bandwidth usage on the internet and other digital communication networks. As the number of connected user devices capable of receiving and displaying video increases, the bandwidth required for digital video usage is expected to continue to grow. Summary of the Invention

[0006] This disclosure relates to apparatus, systems, and methods related to digital video coding and decoding, and specifically describes coordination for linear mode prediction in video coding and decoding. The described methods can be applied to existing video coding and decoding standards (e.g., High Efficiency Video Codec (HEVC)) and future video coding and decoding standards or codecs.

[0007] In one representative aspect, the disclosed technology can be used to provide a method for encoding and decoding video data, comprising: determining whether a first prediction mode can be applied based on one or more picture order count (POC) parameters associated with a picture of a current video block of the video, wherein, in the first prediction mode, the prediction samples of the current video block are refined at least based on gradients in different directions; and, based on the determination, performing a conversion between the current video block and the bitstream of the video; wherein a first reference picture (R0) and a second reference picture (R1) are associated with a current picture including the current video block, wherein τ0 is the picture order count of the current picture and the... The difference between the image sequence counts of the first reference image (τ0 = POC(current) - POC(R0)), and where τ1 is the difference between the image sequence count of the second reference image and the image sequence count of the current image (τ1 = POC(R1) - POC(current)), in response to abs(abs(τ0) - abs(τ1)) ≥ T0, the first prediction mode is avoided, where T0 is an integer and T0 is greater than 0, or in response to abs(τ0) > abs(τ1) × T0 or abs(τ1) > abs(τ0) × T0, the first prediction mode is avoided, where T0 is an integer and T0 = 1.

[0008] In another representative aspect, the disclosed technology can be used to provide an apparatus in a video system, including a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to implement the methods described above.

[0009] In another representative aspect, the disclosed technology can be used to provide a computer program product stored on a non-transitory computer-readable medium, the computer program product including program code for performing the methods described above.

[0010] In another representative aspect, the disclosed technology can be used to provide an apparatus for encoding and decoding video data, including a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to: determine, based on one or more picture sequence count (POC) parameters associated with a picture of a current video block of the video, whether a first prediction mode can be applied, wherein, in the first prediction mode, the prediction samples of the current video block are refined based at least on gradients in different directions; and, based on the determination, perform a conversion between the current video block and the bitstream of the video; wherein a first reference picture (R0) and a second reference picture (R1) are associated with the current picture including the current video block. Where τ0 is the difference between the image sequence count of the current image and the image sequence count of the first reference image (τ0 = POC(current) - POC(R0)), and where τ1 is the difference between the image sequence count of the second reference image and the image sequence count of the current image (τ1 = POC(R1) - POC(current)), the first prediction mode is avoided in response to abs(abs(τ0) - abs(τ1)) ≥ T0, where T0 is an integer and T0 is greater than 0, or the first prediction mode is avoided in response to abs(τ0) > abs(τ1) × T0 or abs(τ1) > abs(τ0) × T0, where T0 is an integer and T0 = 1.

[0011] In another representative aspect, the disclosed technology can be used to provide a non-transitory computer-readable storage medium storing instructions that cause a processor to: determine, based on one or more picture sequence count (POC) parameters associated with a picture of a current video block of a video, whether a first prediction mode can be applied, wherein, in the first prediction mode, the prediction samples of the current video block are refined at least based on gradients in different directions; and, based on the determination, perform a conversion between the current video block and the bitstream of the video; wherein a first reference picture (R0) and a second reference picture (R1) are associated with a current picture including the current video block, wherein τ0 is a graph of the current picture. The difference between the image sequence count and the image sequence count of the first reference image (τ0 = POC(current) - POC(R0)), and where τ1 is the difference between the image sequence count of the second reference image and the image sequence count of the current image (τ1 = POC(R1) - POC(current)), in response to abs(abs(τ0) - abs(τ1)) ≥ T0, the first prediction mode is avoided, where T0 is an integer and T0 is greater than 0, or in response to abs(τ0) > abs(τ1) × T0 or abs(τ1) > abs(τ0) × T0, the first prediction mode is avoided, where T0 is an integer and T0 = 1.

[0012] The above and other aspects and features of the disclosed technology are described in more detail in the accompanying drawings, description and claims. Attached Figure Description

[0013] Figure 1 An example of constructing a Merge candidate list is shown.

[0014] Figure 2 An example of a spatial candidate location is shown.

[0015] Figure 3 An example of a candidate pair that has undergone redundant checks of spatial Merge candidates is shown.

[0016] Figure 4A and Figure 4B An example of the position of a second prediction unit (PU) based on the size and shape of the current block is shown.

[0017] Figure 5 An example of motion vector scaling for time merge candidates is shown.

[0018] Figure 6 An example of candidate positions for the time-merge candidate is shown.

[0019] Figure 7 An example of generating bidirectional prediction Merge candidates using a combination is shown.

[0020] Figure 8 An example of constructing motion vector prediction candidates is shown.

[0021] Figure 9 An example of motion vector scaling for spatial motion vector candidates is shown.

[0022] Figure 10 An example of motion prediction using the Optional Time Motion Vector Prediction (ATMVP) algorithm for the codec unit (CU) is shown.

[0023] Figure 11 An example is shown with a codec unit (CU) for sub-blocks and adjacent blocks used by the Space-Time Motion Vector Prediction (STMVP) algorithm.

[0024] Figure 12A and Figure 12B An example snapshot of a sub-block is shown when the Overlapping Block Motion Compensation (OBMC) algorithm is used.

[0025] Figure 13 An example of adjacent samples used to derive the parameters of the Local Illumination Compensation (LIC) algorithm is shown.

[0026] Figure 14 An example of a simplified affine motion pattern is shown.

[0027] Figure 15 An example of the affine motion vector field (MVF) for each sub-block is shown.

[0028] Figure 16 An example of motion vector prediction (MVP) for the AF_INTER affine motion pattern is shown.

[0029] Figure 17A and Figure 17B Example candidates for the AF_MERGE affine motion mode are shown.

[0030] Figure 18 An example of bilateral matching in the Pattern Matching Motion Vector Derivation (PMMVD) mode is shown, which is a special Merge mode based on the Frame Rate Upconversion (FRUC) algorithm.

[0031] Figure 19 An example of template matching in the FRUC algorithm is shown.

[0032] Figure 20 An example of one-sided motion estimation in the FRUC algorithm is shown.

[0033] Figure 21 An example of the optical flow trajectory used by the bidirectional optical flow (BDOF) algorithm is shown.

[0034] Figure 22A and Figure 22B An example snapshot is shown using the Bidirectional Optical Flow (BDOF) algorithm without block expansion.

[0035] Figure 23 An example of interpolation samples used in BDOF is shown.

[0036] Figure 24 An example of a decoder-side motion vector refinement (DMVR) algorithm based on bilateral template matching is shown.

[0037] Figure 25 An example of the location of sample points for the parameters (α and β) used to derive the linear model in a linear prediction mode is shown.

[0038] Figure 26 An example of a straight line between the maximum and minimum brightness values ​​is shown (representing a linear mode).

[0039] Figure 27 Another example of interpolation samples used in BDOF is shown.

[0040] Figures 28A to 28I A flowchart of an example method for video processing is shown.

[0041] Figure 29 This is a block diagram of an example hardware platform used to implement the visual media decoding or visual media encoding techniques described herein.

[0042] Figure 30 This is a block diagram of an example video processing system that can implement the disclosed technology. Detailed Implementation

[0043] Due to the increasing demand for higher resolution video, video encoding and decoding methods and technologies are ubiquitous in modern technology. Video codecs typically consist of electronic circuitry or software that compresses or decompresses digital video and are constantly being improved to provide higher encoding and decoding efficiency. A video codec converts uncompressed video into a compressed format and vice versa. There is a complex relationship between video quality, the amount of data used to represent the video (determined by the bit rate), the complexity of the encoding and decoding algorithms, sensitivity to data loss and errors, ease of editing, random access, and end-to-end latency. Compression formats typically conform to standard video compression specifications, such as the High Efficiency Video Codec (HEVC) standard (also known as H.265 or MPEG-H Part 2), the Universal Video Codec Standard to be Completed, or other current and / or future video codec standards.

[0044] Embodiments of the disclosed techniques can be applied to existing video codec standards (e.g., HEVC, H.265) and future standards to improve compression performance. Section headings are used in this document to improve readability of the description and do not in any way limit the discussion or embodiments (and / or implementations) to the relevant sections only.

[0045] 1. Example of inter-frame prediction in HEVC / H.265

[0046] Over the years, video codec standards have improved significantly and now offer, in part, high encoding and decoding efficiency and support for higher resolutions. Latest standards such as HEVC and H.265 are based on a hybrid video codec architecture, which utilizes time prediction plus transform encoding and decoding.

[0047] 1.1. Examples of Predictive Patterns

[0048] Each inter-frame prediction PU (prediction unit) has motion parameters for one or two lists of reference images. In some embodiments, the motion parameters include motion vectors and reference image indices. In other embodiments, inter_pred_idc can also be used to signal the use of one of the two lists of reference images. In yet another embodiment, the motion vectors can be explicitly encoded into increments relative to the predictor.

[0049] When encoding and decoding a CU using skip mode, a PU is associated with a CU, and there are no significant residual coefficients, no encoded / decoded motion vector increments, or reference picture indices. A Merge mode is specified to obtain the motion parameters of the current PU from neighboring PUs, including spatial and temporal candidates. Merge mode can be applied to any PU for inter-frame prediction, not just skip mode. An alternative to Merge mode is explicit transmission of motion parameters, where, for each PU, the motion vectors, the corresponding reference picture index for each reference picture list, and the reference picture list usage are explicitly communicated via signaling.

[0050] A PU is generated from a sample block when the signaling indicates that one of two reference image lists will be used. This is called "uni-prediction". Uni-prediction can be used for both P-strips and B-strips.

[0051] When the signaling indicates that two lists of reference images will be used, a PU is generated from two sample blocks. This is called "bi-prediction". Bi-prediction is only applicable to B-strips.

[0052] 1.1.1.1 Constructing candidate implementations for the Merge pattern

[0053] When predicting the PU using the Merge pattern, the indices pointing to entries in the Merge candidate list are parsed from the bitstream and used to retrieve motion information. The construction of this list can be summarized in the following steps:

[0054] Step 1: Derivation of the original candidates

[0055] Step 1.1: Spatial Candidate Derivation

[0056] Step 1.2: Redundancy check of spatial candidates

[0057] Step 1.3: Derivation of Time Candidates

[0058] Step 2: Insert additional candidates

[0059] Step 2.1: Create bidirectional prediction candidates

[0060] Step 2.2: Insert zero-motion candidates

[0061] Figure 1 An example of constructing a Merge candidate list based on the sequence of steps summarized above is shown. For spatial Merge candidate derivation, up to four Merge candidates are selected from candidates located at five distinct positions. For temporal Merge candidate derivation, up to one Merge candidate is selected from two candidates. Since a constant number of candidates is assumed for each PU at the decoder, additional candidates are generated when the number of candidates does not reach the maximum number of Merge candidates (MaxNumMergeCand) signaled in the stripe header. Because the number of candidates is constant, truncated unary binarization (TU) is used to encode the index of the best Merge candidate. If the CU size is equal to 8, all PUs of the current CU share a single Merge candidate list, which is the same as the Merge candidate list of a 2N×2N prediction unit.

[0062] 1.1.2 Construction Space Merge Candidates

[0063] In the derivation of the space Merge candidate, in the position of Figure 2 Up to four merge candidates are selected from the candidates for the described positions. The derivation order is A1, B1, B0, A0, and B2. Position B2 is considered only if any PU at positions A1, B1, B0, or A0 is unavailable (e.g., because it belongs to another stripe or block) or if it is intra-frame encoding / decoding. After adding the candidate at position A1, redundancy checks are performed on the addition of the remaining candidates to ensure that candidates with the same motion information are excluded from the list, thereby improving encoding / decoding efficiency.

[0064] To reduce computational complexity, not all possible candidate pairs were considered in the mentioned redundancy check. Instead, only... Figure 3 Pairs connected by arrows are added to the list only if the corresponding candidate used for redundancy checking has different motion information. Another source of duplicate motion information is a "second PU" associated with partitions different from 2N×2N. As an example, Figure 4A and 4B The second PU is depicted for the N×2N and 2N×N cases, respectively. When the current PU is partitioned into N×2N, the candidate at position A1 is not considered for list construction. In some embodiments, adding this candidate may result in two prediction units with the same motion information, which is redundant for an encoding / decoding unit with only one PU. Similarly, when the current PU is partitioned into 2N×N, position B1 is not considered.

[0065] 1.1.1.3 Construction Time Merge Candidates

[0066] In this step, only one candidate is added to the list. Specifically, in the derivation of the time-merge candidate, the scaling motion vector is derived based on the co-located PU, which belongs to the image within a given list of reference images that has the smallest POC difference with the current image. The list of reference images to be used for deriving the co-located PU is explicitly notified by signaling in the strip header.

[0067] Figure 5 An example of the derivation for scaling motion vectors for temporal merge candidates (e.g., dashed lines) is shown, which is scaled from the motion vectors of co-located PUs using POC distances tb and td, where tb is defined as the POC difference between the current image and the reference image of the current image, and td is defined as the POC difference between the reference image of the co-located image and the co-located image. The reference image index for the temporal merge candidate is set to zero. For the B-strip, two motion vectors are obtained, one for reference image list 0 and the other for reference image list 1, and these two motion vectors are combined to obtain bidirectional predicted merge candidates.

[0068] In the co-occurring PU(Y) of the reference frame, a time candidate position is selected between candidate C0 and C1, such as... Figure 6 As shown. If the PU at position C0 is unavailable, is intra-frame encoded, or is outside the current CTU, then position C1 is used. Otherwise, position C0 is used for the derivation of the time merge candidate.

[0069] 1.1.4 Constructing Additional Types for Merge Candidates

[0070] In addition to the space-time merge candidates, two additional types of merge candidates exist: combined bidirectional predictive merge candidates and zero merge candidates. Combined bidirectional predictive merge candidates are generated by utilizing space-time merge candidates. These combined bidirectional predictive merge candidates are only used for B-strips. Combined bidirectional predictive candidates are generated by combining the motion parameters of the first reference image list of the original candidate with the motion parameters of the second reference image list of another candidate. If these two tuples provide different motion hypotheses, they will form a new bidirectional predictive candidate.

[0071] Figure 7 An example of the process is shown, where two candidates in the original list (710 on the left) have mvL0 and refIdxL0 or mvL1 and refIdxL1, which are used to create a bidirectional prediction of the Merge candidate combination added to the final list (720 on the right).

[0072] Zero-motion candidates are inserted to populate the remaining entries in the Merge candidate list, thus achieving the MaxNumMergeCand capacity. These candidates have null spatial displacements and reference image indices that start at zero and increase whenever a new zero-motion candidate is added to the list. The number of reference frames used for these candidates is 1 and 2, respectively, for unidirectional and bidirectional prediction. In some embodiments, redundancy checks are not performed on these candidates.

[0073] 1.1.5 Example of motion estimation region for parallel processing

[0074] To accelerate encoding processing, motion estimation can be performed in parallel, thereby simultaneously deriving the motion vectors of all prediction units within a given region. Deriving merge candidates from spatial neighborhoods can interfere with parallel processing because a prediction unit cannot derive motion parameters from neighboring PUs until its associated motion estimation is complete. To mitigate the trade-off between encoding / decoding efficiency and processing latency, a Motion Estimation Region (MER) can be defined. The size of the MER is signaled in the Picture Parameter Set (PPS) using the syntax element "log2_parallel_merge_level_minus2". When defining the MER, merge candidates falling into the same region are marked as unavailable and therefore not considered in the list construction.

[0075] 1.2 Examples of Advanced Motion Vector Prediction (AMVP)

[0076] AMVP utilizes the spatiotemporal correlation of motion vectors with adjacent PUs for explicit transmission of motion parameters. It constructs a motion vector candidate list by first verifying the availability above and to the left of temporally adjacent PU locations, removing redundant candidates, and adding zero vectors to make the candidate list a constant length. The encoder can then select the best predictor from the candidate list and send the corresponding index indicating the selected candidate. Similar to Merge index signaling, a truncated unary is used to encode the index of the best motion vector candidate. In this case, the maximum value to be encoded is 2 (see [link to documentation]). Figure 8 The following sections provide details of the derivation process for the motion vector prediction candidates.

[0077] 1.2.1 Example of constructing motion vector prediction candidates

[0078] Figure 8 The derivation process of motion vector prediction candidates is summarized, and it can be implemented by taking the index as input for each list of reference images.

[0079] In motion vector prediction, two types of motion vector candidates are considered: spatial motion vector candidates and temporal motion vector candidates. For the derivation of spatial motion vector candidates, the final result is based on the previously established... Figure 2 The motion vector derivation for each PU at the five different locations shown in the figure derives two motion vector candidates.

[0080] For temporal motion vector candidate derivation, a motion vector candidate is selected from two candidates, based on the derivation of two distinct co-located positions. After generating the first list of spatiotemporal candidates, duplicate motion vector candidates are removed from the list. If the number of potential candidates is greater than 2, motion vector candidates with a reference image index greater than 1 from the associated reference image list are removed from the list. If the number of spatiotemporal motion vector candidates is less than 2, additional zero motion vector candidates are added to the list.

[0081] 1.2.2 Constructing Spatial Motion Vector Candidates

[0082] In the derivation of the spatial motion vector candidates, at most two candidates are considered from the five potential candidates, which are located as previously stated. Figure 2 The derivation in the PU at the positions shown is the same as the position of the motion merge. The derivation order on the left side of the current PU is defined as A0, A1, and scaled A0, scaled A1. The derivation order on the upper side of the current PU is defined as B0, B1, B2, and scaled B0, scaled B1, scaled B2. Therefore, for each side, there are four cases that can be used as motion vector candidates, two of which do not require spatial scaling, and two of which do. The four different cases are summarized as follows:

[0083] - No space scaling

[0084] -(1) Same list of reference images, and same reference image index (same POC) -(2) Different list of reference images, but the same reference images (same POC)

[0085] - Space scaling

[0086] -(3) Same list of reference images, but different reference images (different POCs)

[0087] -(4) Different lists of reference images, and different reference images (different POCs)

[0088] First, we examine the case without spatial scaling, then we examine the case where spatial scaling is allowed. Spatial scaling is considered when the reference image of the POC differs between the reference image of the adjacent PU and the reference image of the current PU, regardless of the reference image list. If all PUs of the left candidate are unavailable or are intra-frame encoded / decoded, scaling of the above motion vectors is allowed to aid in the parallel derivation of the left and top MV candidates. Otherwise, spatial scaling of the above motion vectors is not allowed.

[0089] like Figure 9 As shown in the example, for spatial scaling, the motion vectors of adjacent PUs are scaled in a similar manner to temporal scaling. One difference is that a list of reference images and the index of the current PU are given as input; the actual scaling process is the same as the temporal scaling process.

[0090] 1.2.3 Constructing Time Motion Vector Candidates

[0091] Except for the derivation of the reference image index, all the procedures used to derive the temporal Merge candidate are the same as those used to derive the spatial motion vector candidate (e.g., ...). Figure 6 (As shown in the example). In some embodiments, the reference image index is signaled to the decoder.

[0092] 2. Example of inter-frame prediction method in Joint Exploration Mode (JEM)

[0093] In some embodiments, reference software called Joint Exploration Mode (JEM) is used to explore future video codec techniques. In JEM, sub-block-based predictions are employed across several codec tools, such as affine prediction, optional temporal motion vector prediction (ATMVP), space-time motion vector prediction (STMVP), bidirectional optical flow (BDOF or BIO), frame rate upconversion (FRUC), local adaptive motion vector resolution (LAMVR), overlapping block motion compensation (OBMC), local illumination compensation (LIC), and decoder-side motion vector refinement (DMVR).

[0094] 2.1 Example of motion vector prediction based on sub-CU

[0095] In a JEM with a quadtree plus binary tree (QTBT), each CU can have at most one set of motion parameters for each prediction direction. In some embodiments, two sub-CU level motion vector prediction methods are considered in the encoder by dividing the large CU into sub-CUs and deriving the motion information of all sub-CUs of the large CU. The Alternative Temporal Motion Vector Prediction (ATMVP) method allows each CU to extract multiple sets of motion information from multiple blocks smaller than the current CU in the juxtaposed reference image. In the Spatial-Temporal Motion Vector Prediction (STMVP) method, the motion vectors of the sub-CUs are recursively derived using temporal motion vector prediction values ​​and spatially adjacent motion vectors. In some embodiments, motion compression of the reference frame may be disabled to preserve a more accurate motion field for sub-CU motion prediction.

[0096] 2.1.1 Example of Optional Time Motion Vector Prediction (ATMVP)

[0097] In the ATMVP method, the temporal motion vector prediction (TMVP) method is modified by extracting multiple sets of motion information (including motion vectors and reference indices) from blocks smaller than the current CU.

[0098] Figure 10 An example of the ATMVP motion prediction process for CU 1000 is shown. The ATMVP 1000 method predicts the motion vectors of sub-CUs 1001 within CU 1000 in two steps. The first step is to identify the corresponding block 1051 in reference image 1050 using time vectors. Reference image 1050 is also referred to as the motion source image. The second step is to divide the current CU 1000 into sub-CUs 1001 and obtain the motion vector and reference index of each sub-CU from the block corresponding to each sub-CU.

[0099] In the first step, reference image 1050 and corresponding blocks are determined based on the motion information of spatially adjacent blocks of the current CU 1000. To avoid repeated scanning of adjacent blocks, the first Merge candidate in the Merge candidate list of the current CU 1000 is used. The first available motion vector and its associated reference index are set to the index of the time vector and the motion source image. In this way, corresponding blocks can be identified more accurately compared to TMVP, where the corresponding block (sometimes called the juxtaposed block) is always located in the lower right or center position relative to the current CU.

[0100] In the second step, the corresponding block of sub-CU 1051 is identified by adding a time vector to the coordinates of the current CU, using the time vector in the motion source image 1050. For each sub-CU, the motion information of its corresponding block (e.g., the smallest motion grid covering the center sample point) is used to derive the sub-CU's motion information. After identifying the motion information of the corresponding N×N block, it is converted into the reference index and motion vector of the current sub-CU in the same manner as the TMVP of HEVC, where motion scaling and other processes are also applied. For example, the decoder checks whether a low latency condition is met (e.g., the POC of all reference images of the current image is less than the POC of the current image) and may use motion vector MV. x (For example, the motion vector corresponding to the reference image list X) to predict the motion vector MV of each sub-CU. y (For example, where X equals 0 or 1 and Y equals 1-X).

[0101] 2.1.2 Example of Space-Time Motion Vector Prediction (STMVP)

[0102] In the STMVP method, the motion vector of the sub-CU is recursively derived according to the raster scan order. Figure 11 An example of a CU with four sub-blocks and adjacent blocks is shown. Consider an 8×8 CU1100 containing four 4×4 sub-CUs A(1101), B(1102), C(1103), and D(1104). The adjacent 4×4 blocks in the current frame are labeled a(1111), b(1112), c(1113), and d(1114).

[0103] Motion derivation of sub-CU A begins by identifying its two spatial neighbors. The first neighbor is the N×N block above sub-CU A1101 (block c 1103). If block c (1113) is unavailable or intra-frame encoded, the other N×N blocks above sub-CU A (1101) are checked (starting from block c 1113, from left to right). The second neighbor is the block to the left of sub-CU A1101 (block b 1112). If block b (1112) is unavailable or intra-frame encoded, the other blocks to the left of sub-CU A1101 are checked (starting from block b 1112, from top to bottom). Motion information obtained from adjacent blocks in each list is scaled to the first reference frame of the given list. Next, the Temporal Motion Vector Predictor (TMVP) of sub-block A1101 is derived by following the same procedure as the TMVP derivation specified in HEVC. Motion information of the juxtaposed block at D 1104 is extracted and scaled accordingly. Finally, after retrieving and scaling the motion information, all available motion vectors are averaged separately for each reference list. The averaged motion vector is then designated as the motion vector for the current sub-CU.

[0104] 2.1.3 Example of motion prediction mode signaling for sub-CU

[0105] In some embodiments, a sub-CU mode is enabled as an additional Merge candidate, and no additional syntax elements are required to signal this mode. Two additional Merge candidates are added to the Merge candidate list of each CU to represent ATMVP and STMVP modes. In some embodiments, up to seven Merge candidates can be used if the sequence parameter set indicates that ATMVP and STMVP are enabled. The encoding logic for the additional Merge candidates is the same as that for the Merge candidates in HM, meaning that for each CU in a P or B stripe, the two additional Merge candidates may require two additional RD checks. In some embodiments, such as JEM, all Merge index bins are context-coded using CABAC (Context-Based Adaptive Binary Arithmetic Codec). In other embodiments, such as HEVC, only the first bin is context-coded, while the remaining bins are context-bypass coded.

[0106] 2.2 Adaptive Motion Vector Differential Resolution

[0107] In some embodiments, when the use_integer_mv_flag in the stripe header is equal to 0, the motion vector difference (MVD) between the PU's motion vector and the predicted motion vector is signaled in units of quarter-luminance samples. In JEM, Locally Adaptive Motion Vector Resolution (LAMVR) is introduced. In JEM, MVD can be encoded and decoded in units of quarter-luminance samples, integer luminance samples, or four luminance samples. The MVD resolution is controlled at the codec unit (CU) level, and the MVD resolution flag is conditionally signaled for each CU having at least one non-zero MVD component.

[0108] For a CU with at least one non-zero MVD component, signaling notifies a first flag to indicate whether quarter-luminance sample MV precision is used in the CU. When the first flag (equal to 1) indicates that quarter-luminance sample MV precision is not used, signaling notifies another flag to indicate whether integer luminance sample MV precision or four-luminance sample MV precision is used.

[0109] When the first MVD resolution flag of the CU is zero or not encoded for the CU (meaning all MVDs in the CU are zero), a quarter-lumen lumen sample MV resolution is used for the CU. When the CU uses integer lumen sample MV precision or four-lumen sample MV precision, the MVPs in the CU's AMVP candidate list are rounded to the corresponding precision.

[0110] In the encoder, CU-level RD checks are used to determine which MVD resolution to use for the CU. That is, for each MVD resolution, CU-level RD checks are performed three times. To speed up the encoder, the following encoding scheme is applied in JEM.

[0111] - During the RD verification of a CU with a normal quarter-luminance sample MVD resolution, the motion information (integer luminance sample accuracy) of the current CU is stored. The stored motion information (after rounding) is used as the starting point for further small-range motion vector refinement for the same CU with integer luminance sample and 4-luminance sample MVD resolutions during the RD verification, so that the time-consuming motion estimation process is not repeated three times.

[0112] - Conditionally invoke the RD check for the CU with a 4-luminance sample MVD resolution. For the CU, skip the RD check for the 4-luminance sample MVD resolution if the RD cost integer luminance sample MVD resolution is much greater than a quarter luminance sample MVD resolution.

[0113] 2.3 Examples of Higher Motion Vector Storage Accuracy

[0114] In HEVC, motion vector accuracy is one-quarter of a pixel (one-quarter of a luma sample and one-eighth of a chroma sample for a 4:2:0 video). In JEM, the accuracy of internal motion vector storage and merge candidates is increased to 1 / 16 pixel. This higher motion vector accuracy (1 / 16 pixel) is used for motion-compensated inter-frame prediction in CUs encoded in skip / merge mode. For CUs encoded in normal AMVP mode, integer pixel or one-quarter pixel motion is used.

[0115] The SHVC upsampling interpolation filter, with the same filter length and normalization factor as the HEVC motion compensation interpolation filter, is used as the motion compensation interpolation filter for additional fractional pixel locations. In JEM, the chroma component motion vector accuracy is 1 / 32 sample, and the additional interpolation filter for the 1 / 32 pixel fractional location is derived by averaging the filters for two adjacent 1 / 16 pixel fractional locations.

[0116] 2.4 Example of Overlapped Block Motion Compensation (OBMC)

[0117] In JEM, OBMC can be enabled and disabled using CU-level syntax. When using OBMC in JEM, OBMC is performed on all Motion Compensation (MC) block boundaries except for the right and bottom boundaries of the CU. Furthermore, it is applied to the luma and chroma components. In JEM, MC blocks correspond to codec blocks. When encoding and decoding a CU using sub-CU modes (including sub-CUEngine, affine, and FRUC modes), each sub-block of the CU is an MC block. To handle CU boundaries uniformly, OBMC is performed at the sub-block level for all MC block boundaries, where the sub-block size is set to equal to 4×4, such as... Figure 12A and 12B As shown.

[0118] Figure 12A The sub-block at the CU / PU boundary is shown; the shaded sub-block is where the OBMC is applied. Similarly, Figure 12B The sub-blocks in the ATMVP pattern are shown.

[0119] When OBMC is applied to the current sub-block, in addition to the current motion vector, the motion vectors of the four connected adjacent sub-blocks (if available and different from the current motion vector) are also used to derive the prediction block for the current sub-block. These multiple prediction blocks based on multiple motion vectors are combined to generate the final prediction signal for the current sub-block.

[0120] The prediction block based on the motion vectors of neighboring sub-blocks is represented as P. N , where N indicates the indices of the adjacent upper, lower, left, and right sub-blocks, and the predicted block based on the motion vector of the current sub-block is represented as P. C When P N When the motion information is based on the motion information of adjacent sub-blocks that contain the same motion information as the current sub-block, it does not start from P. N Execute OBMC. Otherwise, execute each P. N Samples added to P C In the same point in the middle, P is about to be N Add four rows / columns to P C The weighting factors {1 / 4, 1 / 8, 1 / 16, 1 / 32} are used for P. N And the weighting factors {3 / 4, 7 / 8, 15 / 16, 31 / 32} are applied to the PC. An exception is small MC blocks (i.e., when the height or width of the codec block is equal to 4 or when the CU is encoded / decoded using sub-CU mode), for which only the P is applied. N Add two rows / columns to P C In this case, the weighting factors {1 / 4, 1 / 8} are used for P. N And the weighting factors {3 / 4, 7 / 8} are used for P.C For P generated based on the motion vectors of vertically (horizontally) adjacent sub-blocks N , will P N Samples in the same row (column) are added to P with the same weighting factor. C .

[0121] In JEM, for CUs with a size of 256 lumen samples or less, signaling informs a CU-level flag to indicate whether OBMC should be applied to the current CU. For CUs with a size exceeding 256 lumen samples or those not encoded / decoded using AMVP mode, OBMC is applied by default. At the encoder, when OBMC is applied to the CU, its effects are considered during the motion estimation phase. The predicted signal formed by OBMC using motion information from the upper and left adjacent blocks is used to compensate for the upper and left boundaries of the original signal of the current CU, and then normal motion estimation processing is applied.

[0122] 2.5 Example of Local Illumination Compensation (LIC)

[0123] Illumination compensation (LIC) is based on a linear model for illumination variations, using a scaling factor a and an offset b. It is adaptively enabled or disabled for each inter-frame mode encoding / decoding unit (CU).

[0124] When LIC is applied to CU, the least squares error method is used to derive parameters a and b by using the neighboring samples of the current CU and their corresponding reference samples. Figure 13 An example of neighboring samples used to derive the parameters of the IC algorithm is shown. More specifically, as... Figure 13 As shown, neighboring samples of the CU's subsamples (2:1 subsamples) and corresponding samples in the reference image (identified by the motion information of the current CU or subCU) are used. The IC parameters are derived and applied to each prediction direction.

[0125] When encoding and decoding a CU in Merge mode, the LIC flag is copied from the adjacent block in a manner similar to motion information copying in Merge mode; otherwise, the LIC flag is signaled to the CU to indicate whether LIC should be applied.

[0126] When LIC is enabled for an image, additional CU-level RD checks are required to determine whether LIC should be applied to the CU. When LIC is enabled for the CU, Mean-Removed Sum Of Absolute Difference (MR-SAD) and Mean-Removed Sum Of Absolute Hadamard-Transformed Difference (MR-SATD) are used instead of SAD and SATD for integer pixel motion search and fractional pixel motion search, respectively.

[0127] To reduce coding complexity, the following coding scheme is applied in JEM:

[0128] - When there is no significant lighting change between the current image and its reference images, LIC is disabled for the entire image. To identify this situation, histograms of the current image and each reference image of the current image are calculated at the encoder. If the histogram difference between the current image and each reference image of the current image is less than a given threshold, LIC is disabled for the current image; otherwise, LIC is enabled for the current image.

[0129] 2.6 Examples of Affine Motion Compensation Prediction

[0130] In HEVC, only the translational motion model is applied to motion compensation prediction (MCP). However, the camera and object may exhibit various motions, such as zooming in / out, rotation, perspective motion, and / or other irregular motions. On the other hand, JEM applies a simplified affine transformation motion compensation prediction. Figure 14 An example of the affine motion field of block 1400 described by two control point motion vectors V0 and V1 is shown. The motion vector field (MVF) of block 1400 is described by the following equation:

[0131]

[0132] like Figure 14 As shown, (v 0x v 0y (v) is the motion vector of the top-left control point. 1x v 1y ) is the motion vector of the upper right control point.

[0133] To further simplify motion compensation prediction, sub-block-based affine transformation prediction can be applied. The sub-block size M×N is derived as follows:

[0134]

[0135] Here, MvPre is the fractional accuracy of the motion vector (e.g., 1 / 16 in JEM), (v 2x v 2y ) is the motion vector of the lower left control point calculated according to Equation 1. If necessary, M and N can be adjusted downwards to make them the divisors of w and h, respectively.

[0136] Figure 15 An example of the affine MVF for each sub-block of block 1500 is shown. To derive the motion vector for each M×N sub-block, the motion vector of the center sample point of each sub-block is calculated according to Equation 1 and rounded to the fractional accuracy of the motion vector (e.g., 1 / 16 in JEM). A motion-compensated interpolation filter is then applied to generate a prediction for each sub-block using the derived motion vector.

[0137] After MCP, the high-accuracy motion vector of each sub-block is rounded and saved with the same accuracy as the normal motion vector.

[0138] 2.6.1 Implementation Examples of AF_INTER Mode

[0139] In JEM, there are two affine motion modes: AF_INTER mode and AF_MERGE mode. AF_INTER mode can be applied to CUs with both width and height greater than 8. The affine flag at the CU level is signaled in the bitstream to indicate whether AF_INTER mode is used. In AF_INTER mode, adjacent blocks are used to construct motion vector pairs {(v0, v1)|v0={v...}. A v B v c}, v1={v D v E The candidate list of}}.

[0140] Figure 16 An example of motion vector prediction (MVP) for block 1600 in AF_INTER mode is shown. Figure 16As shown, v0 is selected from the motion vectors of sub-blocks A, B, or C. Motion vectors from neighboring blocks can be scaled according to a reference list. The scaling can also be based on the relationship between the Picture Order Count (POC) for the reference of the neighboring block, the POC for the reference of the current CU, and the POC of the current CU. The method for selecting v1 from neighboring sub-blocks D and E is similar. If the number of candidates in the candidate list is less than two, the list can be populated by motion vector pairs formed by copying each AMVP candidate. When the candidate list is greater than two, the candidates can first be sorted according to adjacent motion vectors (e.g., based on the similarity of two motion vectors in a candidate pair). In some embodiments, the first two candidates are retained. In some embodiments, a rate distortion (RD) cost check is used to determine which motion vector pair candidate is selected as the Control Point Motion Vector Prediction (CPMVP) for the current CU. The index indicating the position of the CPMVP in the candidate list can be signaled in the bitstream. After determining the CPMVP of the current affine CU, affine motion estimation is applied and the Control Point Motion Vector (CPMV) is found. Then, the difference between CPMV and CPMVP is communicated via signaling in the bitstream. 2.6.3 Example of AF_MERGE mode

[0141] When CU is applied in AF_MERGE mode, it obtains the first block encoded and decoded in affine mode from the valid adjacent reconstructed blocks. Figure 17A This shows an example of the current candidate block selection order for the CU 1700. (Example:) Figure 17A As shown, the selection order can be from the left (1701), top (1702), top right (1703), bottom left (1704) to top left (1705) of the current CU 1700. Figure 17B Another example of a candidate block for the current CU 1700 in AF_MERGE mode is shown. If the adjacent lower-left block 1701 is encoded / decoded in affine mode, such as... Figure 17B As shown, the motion vectors v2, v3, and v4 of the upper left, upper right, and lower left corners of the CU containing block A are derived. Furthermore, the motion vector v0 of the upper left corner of the current CU 1700 is calculated based on v2, v3, and v4. The motion vector v1 of the upper right corner of the current CU can be calculated accordingly.

[0142] After calculating the CPMV v0 and v1 of the current CU according to the affine motion model in equation (1), the MVF of the current CU can be generated. In order to identify whether the current CU is encoded and decoded in AF_MERGE mode, when at least one adjacent block is encoded and decoded in affine mode, the affine flag can be notified by signaling in the bit stream.

[0143] 2.7 Example of Motion Vector Derivation (PMMVD) for Pattern Matching

[0144] PMMVD mode is a special merge mode based on the Frame-Rate Up Conversion (FRUC) method. This mode uses the decoder to derive block motion information instead of sending signaling to notify the block of its motion.

[0145] When the Merge flag of the CU is true, the FRUC flag can be signaled to the CU. When the FRUC flag is false, the Merge index can be signaled and the regular Merge mode can be used. When the FRUC flag is true, additional FRUC mode flags can be signaled to indicate which method (e.g., bilateral matching or template matching) will be used to derive the block's motion information.

[0146] On the encoder side, the decision on whether to use the FRUC Merge mode for the CU is based on the RD cost selection made for normal Merge candidates. For example, multiple matching modes for the CU (e.g., bilateral matching and template matching) are validated using RD cost selection. The matching mode that results in the minimum cost is further compared with other CU modes. If the FRUC matching mode is the most efficient mode, the FRUC flag is set to true for the CU, and the relevant matching mode is used.

[0147] Typically, the motion derivation process in the FRUC Merge model has two steps: first, a CU-level motion search is performed, followed by sub-CU-level motion refinement. At the CU level, the original motion vector of the entire CU is derived based on bilateral matching or template matching. First, a candidate MV list is generated, and the candidate that causes the minimum matching cost is selected as the starting point for further CU-level refinement. Then, a local search based on bilateral matching or template matching is performed near the starting point. The MV result with the minimum matching cost is taken as the MV of the entire CU. Subsequently, using the derived CU motion vector as the starting point, the motion information is further refined at the sub-CU level.

[0148] For example, the following derivation process is performed for the motion information derivation of a W×HCU. In the first stage, the MV of the entire W×HCU is derived. In the second stage, the CU is further divided into M×M sub-CUs. The value of M is calculated as shown in (3), and D is a predefined partitioning depth, which is set to 3 by default in JEM. Then the MV of each sub-CU is derived.

[0149]

[0150] Figure 18An example of bilateral matching used in the Frame Rate Upconversion (FRUC) method is shown. Bilateral matching is used to derive motion information of the current CU (1800) by finding the closest match between two blocks along the motion trajectory of the current CU (1800) in two different reference images (1810, 1811). Under the assumption of continuous motion trajectories, the motion vectors MV0 (1801) and MV1 (1802) pointing to the two reference blocks are proportional to the temporal distances between the current image and the two reference images (e.g., TD0 (1803) and TD1 (1804)). In some embodiments, bilateral matching becomes a mirror-based bidirectional MV when the current image 1800 is temporally between the two reference images (1810, 1811) and the temporal distances from the current image to the two reference images are the same.

[0151] Figure 19 An example of template matching used in the Frame Rate Upconversion (FRUC) method is shown. Template matching is used to deduce motion information for the current CU 1900 by finding the closest match between a template in the current image 1910 (the top and / or left adjacent block of the current CU) and a block in the reference image (e.g., of the same size as the template). In addition to the FRUCMerge mode described above, template matching can also be applied to the AMVP mode. As done in both JEM and HEVC, AMVP has two candidates. New candidates are deduced using the template matching method. If a newly deduced candidate by template matching differs from the first existing AMVP candidate, it is inserted at the beginning of the AMVP candidate list, and the list size is then set to 2 (e.g., by removing the second existing AMVP candidate). When applied to AMVP mode, only CU-level search is performed.

[0152] The MV candidates set at the CU level may include the following: (1) the original AMVP candidate if the current CU is in AMVP mode, (2) all Merge candidates, (3) several MVs in the interpolated MV field (described later), and the top and left adjacent motion vectors.

[0153] When using bilateral matching, each valid MV of the Merge candidate can be used as input to generate MV pairs assuming bilateral matching. For example, in reference list A, a valid MV of the Merge candidate is (MVa, ref a Then, find the reference image ref of its paired bilateral MV in other reference list B. b , making ref a and ref b It is located on a different side of the current image in time. If such a ref b If it is not available in reference list B, then ref bDetermined to be related to ref a Different references, and their time distance to the current image is the minimum value in list B. In determining the reference... b Then, based on the current image ref a and ref b The time distance between them is used to scale MVA to derive MVb.

[0154] In some embodiments, four MVs from the interpolated MV field can also be added to the CU-level candidate list. More specifically, the interpolated MVs at the current CU positions (0,0), (W / 2,0), (0,H / 2), and (W / 2,H / 2) are added. When FRUC is applied to the AMVP pattern, the original AMVP candidates are also added to the CU-level MV candidate set. In some embodiments, at the CU level, 15 MVs are added to the candidate list for AMVPCU and 13 MVs are added to the candidate list for MergeCU.

[0155] The MV candidates set at the sub-CU level include: (1) MVs determined from the CU level search, (2) adjacent MVs at the top, left, top-left, and top-right corners, (3) scaled versions of the juxtaposed MVs from the reference image, (4) one or more ATMVP candidates (maximum 4), and (5) one or more STMVP candidates (e.g., maximum 4). The scaled MVs from the reference image are derived as follows. The reference images in both lists are traversed. The MVs at the juxtaposed positions of the sub-CUs in the reference images are scaled to the reference of the starting CU level MV. The ATMVP and STMVP candidates may be limited to the first four. At the sub-CU level, one or more MVs (e.g., up to 17) are added to the candidate list.

[0156] Generation of interpolated MV fields. Before encoding and decoding the frames, an interpolated motion field is generated for the entire image based on a one-sided ME. The motion field can then be used later as a CU-level or sub-CU-level MV candidate.

[0157] In some embodiments, the motion field of each reference image in the two reference lists is traversed in a 4×4 block-level manner. Figure 20 An example of one-sided motion estimation (ME)2000 in the FRUC method is shown. For each 4×4 block, if the motion associated with the block passes through 4×4 blocks in the current image and the block is not assigned any interpolated motion, the motion of the reference block is scaled to the current image according to the temporal distances TD0 and TD1 (in the same way as the MV scaling of TMVP in HEVC), and the scaled motion is assigned to the block in the current frame. If no scaled MV is assigned to the 4×4 block, the block's motion is marked as unavailable in the interpolated motion field.

[0158] Interpolation and matching costs.When the motion vector points to the fractional sample point location, motion-compensated interpolation is required. To reduce complexity, bilinear interpolation can be used for bilateral matching and template matching instead of conventional 8-tap HEVC interpolation.

[0159] The calculation of matching cost differs slightly at different steps. When selecting candidates from the candidate set at the CU level, the matching cost can be the absolute sum difference (SAD) of bilateral matching or template matching. After determining the starting MV, the matching cost C of bilateral matching in the sub-CU level search is calculated as follows:

[0160]

[0161] Here, w is the weighting factor. In some embodiments, w can be set to 4. MV and MV S These indicate the current MV and the starting MV, respectively. SAD can still be used as the matching cost for template matching in sub-CU level searches.

[0162] In FRUC mode, the motion signature (MV) is derived solely using luma samples. The derived motion is then used for both luma and chroma predictions in the inter-frame prediction (MC). After determining the MV, the final MC is performed using an 8-tap interpolation filter for luma and a 4-tap interpolation filter for chroma.

[0163] MV refinement is a pattern-based MV search, categorized by bilateral matching cost or template matching cost. JEM supports two search modes—Unrestricted Center-Biased Diamond Search (UCBDS) and Adaptive Cross Search—performing MV refinement at the CU and sub-CU levels, respectively. For both CU and sub-CU level MV refinement, the MV is directly searched with a quarter-luminance sample MV precision, followed by an eighth-luminance sample MV refinement. The search range for MV refinement in the CU and sub-CU steps is set to equal 8 luminance samples.

[0164] In the bilateral matching Merge mode, bidirectional prediction is applied because the motion information of the CU is derived based on the nearest match between two blocks along the current CU's motion trajectory in two different reference images. In the template matching Merge mode, the encoder can select for the CU from unidirectional predictions in list 0, unidirectional predictions in list 1, or bidirectional predictions. The template matching cost can be selected based on the following:

[0165] If costBi <= factor * min(cost0, cost1)

[0166] Then use bidirectional prediction;

[0167] Otherwise, if cost0 <= cost1

[0168] Then use the one-way prediction from list 0;

[0169] otherwise,

[0170] Use the one-way prediction from List 1;

[0171] Here, cost0 is the SAD of template matching in list 0, cost1 is the SAD of template matching in list 1, and costBi is the SAD of bidirectional prediction template matching. For example, when the value of factor equals 1.25, this means that the selection process is biased towards bidirectional prediction. Inter-frame prediction direction selection can be applied to CU-level template matching processes.

[0172] 2.8 Examples of Generalized Bidirectional Prediction Improvement (GBi)

[0173] In VTM-3.0, a generalized bidirectional prediction improvement (GBi) is employed. GBi applies unequal weights to the predictors from L0 and L1 in bidirectional prediction mode. In inter-frame prediction mode, multiple weight pairs, including equal weight pairs (1 / 2, 1 / 2), are evaluated based on rate distortion optimization (RDO), and the GBi index of the selected weight pair is signaled to the decoder. In Merge mode, the GBi index is inherited from the neighboring CU. The predictor generation formula is shown in Equation (5).

[0174] P GBi =(w0×P L0 +w1×P L1 +RoundingOffset)>>shiftNum Gbi Equation (5)

[0175] Here, P GBi This is the final predictor for GBi, and w0 and w1 are the selected GBi weights applied to the predictors (PL0 and PL1) for list 0 (L0) and list 1 (L1), respectively. RoundingOffset GBi and shiftNum Gbi This is the final predictor used in the standardized GBi. The supported w1 weight set is {-1 / 4, 3 / 8, 1 / 2, 5 / 8, 5 / 4}, where five weights correspond to one equal weight pair and four unequal weight pairs. The mixed gain, the sum of w1 and w0, is fixed at 1.0. Therefore, the corresponding w0 weight set is {5 / 4, 5 / 8, 1 / 2, 3 / 8, -1 / 4}. Weight pair selection is at the CU level.

[0176] For non-low latency images, the weight set size is reduced from 5 to 3, where the w1 weight set is {3 / 8, 1 / 2, 5 / 8} and the w0 weight set is {5 / 8, 1 / 2, 3 / 8}. This reduction in weight set size for non-low latency images applies to BMS2.1 GBi and all GBi tests in this disclosure.

[0177] 2.8.1 GBi encoder bug fixes

[0178] To reduce GBi encoding time, the current encoder design stores the unidirectional predicted motion vector estimated based on a GBi weight of 4 / 8 and reuses it for unidirectional prediction searches with other GBi weights. This fast encoding method is applied to both translational and affine motion modes. In VTM2.0, both 6-parameter and 4-parameter affine models are used. When the GBi weight is 4 / 8, the BMS2.1 encoder does not distinguish between the 4-parameter and 6-parameter affine models when storing the unidirectional predicted affine MV. Therefore, after encoding with a GBi weight of 4 / 8, the 4-parameter affine MV can be overwritten with the 6-parameter affine MV. For GBi weights of other values, the stored 6-parameter affine MV can be used for the 4-parameter affine ME, or vice versa. The proposed GBi encoder error correction involves storing the 4-parameter and 6-parameter affine MVs separately. When the GBi weight is equal to 4 / 8, the encoder will store these affine MVs based on the affine mode type, and for GBi with other weights, it will reuse the corresponding affine MVs based on the affine mode type.

[0179] 2.8.2GBi encoder acceleration

[0180] In this existing implementation, five encoder acceleration methods are proposed to reduce encoding time when GBi is enabled.

[0181] (1) Conditional skipping affine motion estimation for certain GBi weights

[0182] In BMS2.1, affine ME, including 4-parameter and 6-parameter affine MEs, is performed on all GBi weights. For some unequal GBi weights (weights not equal to 4 / 8), we recommend conditionally skipping affine ME. Specifically, affine ME is performed on other GBi weights only if the affine mode is selected as the current best mode and it is not the affine merge mode after evaluating the GBi with weights of 4 / 8. If the current image is not a low-latency image, the bidirectional prediction ME for the translation mode will be skipped when performing affine ME for unequal GBi weights. If the affine mode is not selected as the current best mode, or if the affine merge mode is selected as the current best mode, then affine ME will be skipped for all other GBi weights.

[0183] (2) Reduce the weights of RD cost checks for low-latency images used in encoding for 1-pixel and 4-pixel MVD precision.

[0184] For low-latency images, there are five weights used for RD cost checking across all MVD precisions, including 1 / 4 pixel, 1 pixel, and 4 pixel. The encoder will first check the RD cost for the 1 / 4 pixel MVD precision. We recommend skipping a portion of the GBi weights for RD cost checking of the 1 pixel and 4 pixel MVD precisions. We order these unequal weights for the 1 / 4 pixel MVD precision based on their RD costs. When encoding at 1 pixel and 4 pixel MVD precisions, only the top two weights with the lowest RD costs, along with the GBi weight of 4 / 8, will be evaluated. Therefore, for low-latency images, a maximum of three weights will be evaluated for both 1 pixel and 4 pixel MVD precisions.

[0185] (3) When the L0 and L1 reference images are the same, skip the bidirectional prediction search conditionally.

[0186] For some images in the RA, the same image may appear in two reference image lists (list-0 and list-1). For example, for the random access codec configuration in CTC, the reference image structure for the first group of images (GOP) is listed below.

[0187] POC: 16, TL: 0, [L0: 0] [L1: 0]

[0188] POC: 8, TL: 1, [L0: 0 16] [L1: 16 0]

[0189] POC: 4, TL: 2, [L0: 0 8] [L1: 8 16]

[0190] POC: 2, TL: 3, [L0: 0 4] [L1: 4 8]

[0191] POC: 1, TL: 4, [L0: 0 2] [L1: 2 4]

[0192] POC: 3, TL: 4, [L0: 2 0] [L1: 4 8]

[0193] POC: 6, TL: 3, [L0: 4 0] [L1: 8 16]

[0194] POC: 5, TL: 4, [L0: 4 0] [L1: 6 8]

[0195] POC: 7, TL: 4, [L0: 6 4] [L1: 8 16]

[0196] POC: 12, TL: 2, [L0: 8 0] [L1: 16 8]

[0197] POC: 10, TL: 3, [L0: 8 0] [L1: 12 16]

[0198] POC: 9, TL: 4, [L0: 8 0] [L1: 10 12]

[0199] POC: 11, TL: 4, [L0: 10 8] [L1: 12 16]

[0200] POC: 14, TL: 3, [L0: 12 8] [L1: 12 16]

[0201] POC: 13, TL: 4, [L0: 12 8] [L1: 14 16]

[0202] POC: 15, TL: 4, [L0: 14 12] [L1: 16 14]

[0203] Note that images 16, 8, 4, 2, 1, 12, 14, and 15 have the same reference images in both lists. For bidirectional prediction of these images, the L0 and L1 reference images may be the same. We propose that the encoder skips bidirectional prediction MEs for unequal GBi weights when 1) the two reference images in bidirectional prediction are the same, 2) the temporal layer is greater than 1, and 3) the MVD precision is 1 / 4 pixel. For affine bidirectional prediction MEs, this fast skipping method only applies to 4-parameter affine MEs.

[0204] (4) Based on the time layer and the POC distance between the reference image and the current image, skip the RD cost check for unequal GBi weights.

[0205] We recommend skipping RD cost evaluations for those with unequal GBi weights when the time layer is 4 (the highest time layer in RA) or the POC distance between the reference image (list 0 or list 1) and the current image is 1 and the encoding / decoding QP is greater than 32.

[0206] (5) During ME, for unequal GBi weights, change floating-point calculations to fixed-point calculations.

[0207] For existing bidirectional predictive search, the encoder will fix the MV of one list and optimize the MV of the other list. The objective is modified before ME to reduce computational complexity. For example, if the MV of list 1 is fixed and the encoder will refine the MV of list 0, the objective of the MV of list 0 is modified using equation (6). O is the original signal, P1 is the predicted signal of list 1, and w is the GBi weight of list 1.

[0208] T=((O<<3)-w*P1)*(1 / (8-w)) (6)

[0209] Here, the term (1 / (8-w)) is stored with floating-point precision, which increases computational complexity. We suggest changing Equation (6) to fixed-point, similar to Equation (7).

[0210] T=(O*a1-P1*a2+round)>>N (7)

[0211] Where a1 and a2 are scaling factors, and their calculation formulas are as follows:

[0212] γ=(1<<N) / (8-w); a1=γ<<3; a2=γ*w; round=1<<(N-1)

[0213] 2.8.3 GBi CU Dimension Constraints

[0214] In this approach, GBi is disabled for small CUs. In inter-frame prediction mode, if bidirectional prediction is used and the CU area is less than 128 luminance samples, GBi will be disabled without sending any signaling.

[0215] 2.9 Example of Bidirectional Optical Flow (BDOF)

[0216] In bidirectional optical flow (BDOF or BIO), motion compensation is first performed to generate a first prediction for the current block (in each prediction direction). This first prediction is used to derive the spatial gradient, temporal gradient, and optical flow for each sub-block / pixel within the block, and is then used to generate a second prediction, such as the final prediction for the sub-block / pixel. Details are described below.

[0217] BDOF is a sample-level motion refinement that is performed on top of block-by-block motion compensation used for bidirectional prediction. In some embodiments, sample-level motion refinement does not use signaling.

[0218] Let I (k) The brightness value of reference k (k = 0, 1) after block motion compensation, and I (k) The horizontal and vertical components of the gradient. Assuming optical flow is effective, the motion vector field (v) x v y The following formula is given:

[0219]

[0220] Combining this optical flow equation with the Hermitian interpolation of the motion trajectory of each sample point yields a unique third-order polynomial, which ultimately matches the function value I. (k) and its derivative Both. The value of the third-order polynomial at t=0 is the BDOF prediction:

[0221]

[0222] Figure 24 An example optical flow trajectory in the Two-Way Optical Flow (BDOF) method is shown. Here, τ0 and τ1 represent the distances to the reference frame, as... Figure 21 As shown. Distances τ0 and τ1 are calculated based on the POC of Ref0 and Ref1: τ0 = POC(current) - POC(Ref0), τ1 = POC(Ref1) - POC(current). If both predictions originate from the same time direction (both from the past or both from the future), then the sign is different, i.e., τ0·τ1 < 0. In this case, BDOF is applied only when the predictions do not originate from the same time (i.e., τ0 ≠ τ1), both reference regions have non-zero motion (MVx0, MVy0, MVx1, MVy1 ≠ 0), and the block motion vector is proportional to the time distance (MVx0 / MVx1 = MVy0 / MVy1 = -τ0 / τ1).

[0223] The motion vector field (v) is determined by minimizing the difference Δ between the values ​​at points A and B. x v y ). Figure 9 An example of the intersection of the motion trajectory and the reference frame plane is shown. The model uses only the first linear term of the local Taylor expansion of Δ:

[0224]

[0225] All values ​​in the above equation depend on the sample point location, denoted as (i′, j′). Assuming the motion is consistent in the local surrounding region, minimize Δ within a square window Ω of (2M+1)×(2M+1) centered on the current prediction point, where M equals 2:

[0226]

[0227] For this optimization problem, JEM uses a simplification method, first minimizing in the vertical direction and then minimizing in the horizontal direction. This results in the following:

[0228]

[0229] in,

[0230]

[0231] To avoid division by zero or very small values, regularization parameters r and m can be introduced in equations (9) and (10).

[0232] r = 500·4 d-8 (12)

[0233] m = 700·4 d-8 (13)

[0234] Here, d is the bit depth of the video sample.

[0235] To keep BDOF memory access the same as regular bidirectional predictive motion compensation, all predictions and gradient values ​​I are computed only for the position within the current block. (k) , Figure 22A An example of an access location outside block 2200 is shown. Figure 22A As shown, in equation (9), a (2M+1)×(2M+1) square window Ω centered on the current prediction point on the boundary of the prediction block needs to access locations outside the block. In JEM, the I outside the block... (k) , The value is set to be equal to the nearest available value within the block. For example, this could be implemented to fill region 2201, such as... Figure 22B As shown.

[0236] Using BDOF, the motion field can be refined for each sample point. To reduce computational complexity, a block-based BDOF design is used in JEM. Motion refinement can be calculated based on 4×4 blocks. In block-based BDOF, the s in equation (9) of all samples in a 4×4 block can be aggregated. n The value of s, then s nThe aggregated values ​​are used to derive the BDOF motion vector offset for a 4×4 block. More specifically, the following formula can be used for block-based BDOF derivation:

[0237]

[0238] Here b k Let represent the sample set belonging to the k-th 4×4 block of the prediction block. Then, in equations (9) and (10), s... n Replace with ((s) n，bk (>>4) to derive the associated motion vector offset.

[0239] In some scenarios, the MV registrar in BDOF may be unreliable due to noise or irregular motion. Therefore, the size of the MV registrar in BDOF is cropped by a threshold. The threshold is determined based on whether all reference images of the current image come from the same direction. For example, if all reference images of the current image come from the same direction, the threshold value is set to 12×2. 14-d Otherwise, set it to 12×2 13-d .

[0240] The gradient of BDOF can be simultaneously computed using motion-compensated interpolation that operates in accordance with the HEVC motion compensation process (e.g., 2D separable finite impulse response (FIR)). In some embodiments, the input to the 2D separable FIR is a reference frame sample point that is identical to the motion compensation process and the fractional position (fracX, fracY) based on the fractional portion of the block motion vector. For horizontal gradients... First, the vertically interpolated signal is used using BIOfilterS, corresponding to the fractional position fracY with a descaling offset of d-8. Then, a gradient filter BIOfilterG is applied in the horizontal direction, corresponding to the fractional position fracX with a descaling offset of 18-d. For the vertical gradient... First, a gradient filter is applied vertically using BIOfilterG, corresponding to the fractional position fracY with a descaling offset of d-8. Then, signal displacement is performed horizontally using BIOfilterS, corresponding to the fractional position fracX with a descaling offset of 18-d. The lengths of the interpolation filter BIOfilterG for gradient calculation and the interpolation filter BIOfilterS for signal displacement can be relatively short (e.g., 6 taps) to maintain reasonable complexity. Table 1 shows examples of filters that can be used for gradient calculation at different fractional positions of the block motion vector in BDOF. Table 2 shows examples of interpolation filters that can be used for predictive signal generation in BIO.

[0241] Table 1: Exemplary filters for gradient computation in BDOF (or BIO)

[0242] Fractional pixel position Gradient interpolation filter (BIOfilterG) 0 {8,-39,-3,46,-17,5} 1 / 16 {8,-32,-13,50,-18,5} 1 / 8 {7,-27,-20,54,-19,5} 3 / 16 {6,-21,-29,57,-18,5} 1 / 4 {4,-17,-36,60,-15,4} 5 / 16 {3,-9,-44,61,-15,4} 3 / 8 {1,-4,-48,61,-13,3} 7 / 16 {0,1,-54,60,-9,2} 1 / 2 {-1,4,-57,57,-4,1}

[0243] Table 2: Exemplary interpolation filters used for predictive signal generation in BDOF (or BIO)

[0244] Fractional pixel position Interpolation filters for predicted signals (BIOfilters) 0 {0,0,64,0,0,0} 1 / 16 {1,-3,64,4,-2,0} 1 / 8 {1,-6,62,9,-3,1} 3 / 16 {2,-8,60,14,-5,1} 1 / 4 {2,-9,57,19,-7,2} 5 / 16 {3,-10,53,24,-8,2} 3 / 8 {3,-11,50,29,-9,2} 7 / 16 {3,-11,44,35,-10,3} 1 / 2 {3,-10,35,44,-11,3}

[0245] In JEM, BDOF can be applied to all dual-prediction blocks when the two predictions come from different reference images. BDOF can be disabled when Local Illumination Compensation (LIC) is enabled for the Cu.

[0246] In some embodiments, OBMC is applied to blocks after the normal MC process. To reduce computational complexity, BDOF may not be applied during the OBMC process. This means that BDOF is only applied to the MC process of a block when its own MV is used, and is not applied to the MC process when the MV of an adjacent block is used during the OBMC process.

[0247] 2.9.1 BDOF Example in VTM-3.0

[0248] Step 1 : Determine if BIO is applicable (W / H is the width / height of the current block)

[0249] BIO is not applicable if the following conditions are met:

[0250] • Is the current video block encoded in affine or ATMVP?

[0251] ·(iPOC-iPOC0)×(iPOC-iPOC1)≥0

[0252] • H = 4 or (W = 4 and H = 8)

[0253] • Weighted prediction

[0254] The GBi weight is not (1,1).

[0255] If the total SAD (denoted as R0 and R1) between two reference blocks is less than a threshold, then BDOF is not used.

[0256]

[0257] Step 2 Data preparation

[0258] For a W×H block, interpolate (W+2)×(H+2) sample points.

[0259] Similar to normal motion compensation, the internal W×H samples are interpolated using an 8-tap interpolation filter.

[0260] The four lateral outlines of the sample point ( Figure 23 The black circle in the image is interpolated using a bilinear filter.

[0261] For each location, the gradient is calculated on two reference blocks (R0 and R1).

[0262] Gx0(x,y)=(R0(x+1,y)-R0(x-1,y))>>4

[0263] Gy0(x,y)=(R0(x,y+1)-R0(x,y-1))>>4

[0264] Gx1(x,y)=(R1(x+1,y)-R1(x-1,y))>>4

[0265] Gy1(x,y)=(R1(x,y+1)-R1(x,y-1))>>4

[0266] For each location, the internal value is calculated as follows:

[0267] T1=(R0(x,y)>>6)-(R1(x,y)>>6), T2=(Gx0(x,y)+Gx1(x,y))>>3,

[0268] T3 = (Gy0(x,y) + Gy1(x,y)) >> 3; and

[0269] B1(x,y)=T2*T2, B2(x,y)=T2*T3, B3(x,y)=-T1*T2, B5(x,y)=T3*T3,

[0270] B6(x,y)=-T1*T3

[0271] Step 3 Calculate the prediction for each block.

[0272] If the SAD between two 4×4 reference blocks is less than the threshold, then skip BDOF for the 4×4 block.

[0273] Calculate Vx and Vy.

[0274] Calculate the final prediction for each position in the 4×4 block:

[0275] b(x,y)=(Vx(Gx 0 (x,y)-Gx 1 (x,y))+Vy(Gy 0 (x,y)-Gy 1(x,y)+1)>>1

[0276] P(x,y)=(R 0 (x,y)+R 1 (x,y)+b(x,y)+offset)>>shift

[0277] Here, b(x, y) is called the correction term.

[0278] 2.9.2 Alternative Examples of BDOF (or BIO) in VTM-3.0

[0279] 8.3.4 Decoding process of inter-frame blocks

[0280] --If predFlagL0 and predFlagL1 are equal to 1, then DiffPicOrderCnt(currPic,refPicList0[refIdx0])*DiffPicOrderCnt(currPic,refPicList1[refIdx1])<0, MotionModelIdc[xCb][yCb] is equal to 0, and MergeModeList[merge_idx[xCb][yCb]] is not equal to SbCol, so set the value of bioAvailableFlag to TRUE.

[0281] --Otherwise, set the value of bioAvailableFlag to FALSE.

[0282] --If bioAvailableFlag equals TRUE, then apply the following:

[0283] --Shift the variable to be equal to Max(2, 14-bitDepth).

[0284] -- The variables cuLevelAbsDiffThres and subCuLevelAbsDiffThres are set to equal (1<<(bitDepth–8+shift))*cbWidth*cbHeight and 1<<(bitDepth–3+shift). The variable cuLevelSumAbsoluteDiff is set to 0.

[0285] --For xSbIdx = 0..(cbWidth>>2)-1 and ySbIdx = 0..(cbHeight>>2)-1, the variables subCuLevelSumAbsoluteDiff[xSbIdx][ySbIdx] and the bidirectional optical flow utilization flag bioUtilizationFlag[xSbIdx][ySbIdx] of the current sub-block are derived as follows:

[0286] subCuLevelSumAbsoluteDiff[xSbIdx][ySbIdx]=∑ i ∑ j Abs(predSamplesL0L[(xSbIdx<<2)+1+i][(ySbIdx<<2)+1+j]-predSamplesL1L[(xSbIdx<<2)+1+i][(ySbIdx<<2)+1+j])with i,j=0..3

[0287] bioUtilizationFlag[xSbIdx][ySbIdx]=subCuLevelSumAbsoluteDiff[xSbIdx][ySbIdx]>=subCuLevelAbsDiffThres

[0288] cuLevelSumAbsoluteDiff+=subCuLevelSumAbsoluteDiff[xSbIdx][ySbIdx]

[0289] --If cuLevelSumAbsoluteDiff is less than cuLevelAbsDiffThres, then set bioAvailableFlag to FALSE.

[0290] --If bioAvailableFlag equals TRUE, then the predicted samples within the current luminance codec subblock are derived by calling the bidirectional optical flow sample prediction process specified in Section 8.3.4.5 using the luminance codec subblock width sbWidth, luminance codec subblock height sbHeight, sample arrays predSamplesL0L and predSamplesL1L, and variables predFlagL0, predFlagL1, refIdxL0, refIdxL1.

[0291] 8.3.4.3 Decimal Sample Interpolation Process

[0292] 8.3.4.3.1 Overview

[0293] The input to this process is:

[0294] – Luminance position (xSb, ySb), used to specify the top-left sample of the current codec sub-block relative to the top-left luminance sample of the current image.

[0295] – The variable sbWidth specifies the width of the current codec sub-block in the luminance sample.

[0296] – The variable sbHeight specifies the height of the current codec sub-block in the luminance sample.

[0297] – The luminance motion vector mvLX is given in 1 / 16th of a luminance sample unit.

[0298] – The chromaticity motion vector mvCLX given in 1 / 32 chromaticity sample units.

[0299] – Selected reference image sample arrays refPicLXL, refPicLXCb, and refPicLXCr.

[0300] – Bidirectional optical flow, enabled by the bioAvailableFlag.

[0301] The output of this process is:

[0302] – When bioAvailableFlag is FALSE, it is a (sbWidth) x (sbHeight) array predSamplesLXL of predicted luminance sample values, or when bioAvailableFlag is TRUE, it is a (sbWidth+2) x (sbHeight+2) array predSamplesLXL of predicted luminance sample values.

[0303] – Two (sbWidth / 2)×(sbHeight / 2) arrays, predSamplesLXCb and predSamplesLXCr, are used to predict the chromaticity sample values.

[0304] Let (xIntL, yIntL) be the luminance position given in full sample units, and (xFracL, yFracL) be the offset given in 1 / 16 sample units. These variables are used only in this clause to specify the fractional sample positions within the reference sample arrays refPicLXL, refPicLXCb, and refPicLXCr.

[0305] When bioAvailableFlag equals TRUE, the derivation of the corresponding predicted luminance sample value predSamplesLXL[xL][yL] for each luminance sample position (xL = -1..sbWidth, yL = -1..sbHeight) within the predicted luminance sample array predSamplesLXL is as follows:

[0306] The derivation of variables xIntL, yIntL, xFracL, and yFracL is as follows:

[0307] xIntL=xSb-1+(mvLX[0]>>4)+xL

[0308] yIntL=ySb-1+(mvLX[1]>>4)+yL

[0309] xFracL=mvLX[0]&15

[0310] yFracL=mvLX[1]&15

[0311] The value of –bilinearFiltEnabledFlag is derived as follows:

[0312] – If xL equals -1 or sbWidth, or yL equals -1 or sbHeight, set the value of bilinearFiltEnabledFlag to TRUE.

[0313] Otherwise, set the value of bilinearFiltEnabledFlag to FALSE.

[0314] – The predicted luminance sample values ​​predSamplesLXL[xL][yL] are derived by calling the procedure specified in Section 8.3.4.3.2 with (xIntL, yIntL), (xFracL, yFracL), refPicLXL and bilinearFiltEnabledFlag as inputs.

[0315] When bioAvailableFlag equals FALSE, the derivation of the corresponding predicted luminance sample value predSamplesLXL[xL][yL] for each luminance sample position (xL = 0..sbWidth-1, yL = 0..sbHeight-1) within the predicted luminance sample array predSamplesLXL is as follows:

[0316] The variables xIntL, yIntL, xFracL, and yFracL are derived as follows:

[0317] xIntL=xSb+(mvLX[0]>>4)+xL

[0318] yIntL=ySb+(mvLX[1]>>4)+yL

[0319] xFracL=mvLX[0]&15

[0320] yFracL=mvLX[1]&15

[0321] – The variable bilinearFiltEnabledFlag is set to FALSE.

[0322] – The predicted luminance sample values ​​predSamplesLXL[xL][yL] are derived by calling the procedure specified in Section 8.3.4.3.2 with (xIntL, yIntL), (xFracL, yFracL), refPicLXL and bilinearFiltEnabledFlag as inputs.

[0323] 8.3.4.5 Bidirectional Optical Flow (BDOF or BIO) Prediction Process

[0324] The input to this process is:

[0325] – Two variables, nCbW and nCbH, are used to specify the width and height of the current codec block.

[0326] – Two (nCbW+2)×(nCbH+2) brightness prediction sample arrays, predSamplesL0 and predSamplesL1,

[0327] – The prediction list utilizes the flags predFlagL0 and predFlagL1.

[0328] –Refer to indices refIdxL0 and refIdxL1,

[0329] – The bidirectional optical flow utilization flag bioUtilizationFlag[xSbIdx][ySbIdx], where xSbIdx = 0..(nCbW>>2)-1, ySbIdx = 0..(nCbH>>2)-1

[0330] The output of this process is an array of (nCbW)x(nCbH) sample values, pbSamples.

[0331] The variable bitDepth is set to be equal to BitDepthY.

[0332] The variable shift2 is set to equal Max(3, 15-bitDepth), and the variable offset2 is set to equal 1<<(shift2-1).

[0333] The variable mvRefineThres is set to equal 1 << (13-bit Depth).

[0334] For xSbIdx=0..(nCbW>>2)-1 and ySbIdx=0..(nCbH>>2)-1,

[0335] – If bioUtilizationFlag[xSbIdx][ySbIdx] is FALSE, then for x = xSb..xSb+3, y = ySb..ySb+3, the predicted sample values ​​of the current prediction unit are derived as follows:

[0336] pbSamples[x][y] = Clip3(0, (1< <bitDepth)-1,

[0337] (predSamplesL0[x][y]+predSamplesL1[x][y]+offset2)>>shift2)

[0338] Otherwise, the predicted sample value of the current prediction unit is derived as follows:

[0339] – The position (xSb, ySb) of the top-left sample point relative to the top-left sample points of the prediction sample point arrays predSamplesL0 and predSampleL1 is derived as follows:

[0340] xSb=(xSbIdx<<2)+1

[0341] ySb=(ySbIdx<<2)+1

[0342] – For x = xSb – 1..xSb + 4, y = ySb - 1..ySb + 4, the following will be applied:

[0343] – The position (hx, vy) of each corresponding sample point (x, y) in the sample point array is predicted as follows:

[0344] The derivation leads to:

[0345] hx = Clip3(1, nCbW, x)

[0346] vy = Clip3(1, nCbH, y)

[0347] – The derivation of variables gradientHL0[x][y], gradientVL0[x][y], gradientHL1[x][y] and gradientVL1[x][y] is as follows:

[0348] gradientHL0[x][y] = (predSamplesL0[hx + 1][vy] – predSampleL0[hx - 1][vy]) >> 4

[0349] gradientVL0[x][y] = (predSampleL0[hx][vy + 1] - predSampleL0[hx][vy - 1]) >> 4

[0350] gradientHL1[x][y] = (predSamplesL1[hx + 1][vy] – predSampleL1[hx - 1][vy]) >> 4

[0351] gradientVL1[x][y] = (predSampleL1[hx][vy + 1] – predSampleL1[hx][vy - 1]) >> 4

[0352] – The derivation of variables temp, tempX and tempY is as follows:

[0353] temp[x][y] = (predSamplesL0[hx][vy] >> 6) – (predSamplesL1[hx][vy] >> 6)

[0354] tempX[x][y] = (gradientHL0[x][y] + gradientHL1[x][y]) >> 3

[0355] empY[x][y] = (gradientVL0[x][y] + gradientVL1[x][y]) >> 3

[0356] – The derivation of variables sGx2, sGy2, sGxGy, sGxdI and sGydI is as follows:

[0357] sGx2 = ∑ x ∑ y (tempX[xSb + x][ySb + y] * tempX[xSb + x][ySb + y]), where x,y = -1..4

[0358] sGy2 = ∑ x ∑ y(tempY[xSb+x][ySb+y]*tempY[xSb+x][ySb+y]), where x,y=-1..4

[0359] sGxGy=∑ x ∑ y (tempX[xSb+x][ySb+y]*tempY[xSb+x][ySb+y]), where x,y=-1..4

[0360] sGxdI=∑ x ∑ y (-tempX[xSb+x][ySb+y]*temp[xSb+x][ySb+y]), where x,y=-1..4

[0361] sGydI=∑ x ∑ y (-tempY[xSb+x][ySb+y]*temp[xSb+x][ySb+y])withx,y=-1..4

[0362] –The derivation of the current sub-block's horizontal and vertical motion refinement is as follows:

[0363] vx=sGx2>0? Clip3(-mvRefineThres,mvRefineThres,-(sGxdI<<3)>>Floor(Log2(sGx2))):0

[0364] vy=sGy2>0? Clip3(-mvRefineThres,mvRefineThres,((sGydI<<3)-((vx*sGxGym)<<12+vx*sGxGys)>>1)>>Floor(Log2(sGy2))):0

[0365] sGxGym=sGxGy>>12;

[0366] sGxGys=sGxGy&((1<<12)-1)

[0367] For x = xSb-1..xSb+2, y = ySb-1..ySb+2, the following conditions apply:

[0368] sampleEnh=Round((vx*(gradientHL1[x+1][y+1]-gradientHL0[x+1][y+1]))>>1)+Round((vy*(gradientVL1[x+1][y+1]-gradientVL0[x+1][y+1]))>>1)

[0369] pbSamples[x][y] = Clip3(0, (1<<bitDepth)-1,(predSamplesL0[x+1][y+1]+predSamplesL1[x+1][y+1]+sampleEnh+offset2)> >shift2)

[0370] 2.10 Example of Decoder-Side Motion Vector Refinement (DMVR)

[0371] In bidirectional prediction, for the prediction of a block region, two prediction blocks formed by the motion vectors (MV) of list0 and list1 are combined to form a single prediction signal. In the decoder-side motion vector refinement (DMVR) method, the two motion vectors of the bidirectional prediction are further refined through a bilateral template matching process. Bilateral template matching is applied in the decoder to perform a distortion-based search between the bilateral templates and reconstructed samples in the reference image to obtain a refined MV without transmitting additional motion information.

[0372] In DMVR, the bilateral template is generated as a weighted combination (i.e., average) of the two prediction blocks from the original MV0 of list 0 and the MV1 of list 1, respectively, as follows: Figure 24 As shown. The template matching operation involves calculating a cost metric between the generated template and the sample region (around the original predicted block) in the reference image. For each of the two reference images, the MV that produces the minimum template cost is considered as the updated MV of that list to replace the original MV. In JEM, nine MV candidates are searched for each list. These nine MV candidates include the original MV and eight surrounding MVs that have a brightness sample offset from the original MV in the horizontal or vertical direction or in both directions. Finally, the two new MVs, i.e., as shown... Figure 24 MV0′ and MV1′ shown are used to generate the final bidirectional prediction result. The sum of absolute differences (SAD) is used as the cost metric. Note that when calculating the cost of the prediction block generated from a wrapping MV, the rounded MV (to integer pixels) is actually used to obtain the prediction block, not the actual MV.

[0373] DMVR is applied to the Merge pattern of bidirectional prediction, where one MV comes from a past reference image and the other from a future reference image, without transferring additional syntax elements. In JEM, DMVR is not applied when LIC, affine motion, FRUC, or subCU Merge candidates are enabled for a CU.

[0374] 3. Example embodiments related to the disclosed technology

[0375] 3.1 Example of predictions from the Cross-Component Linear Model (CCLM)

[0376] To reduce cross-component redundancy, the CCLM prediction mode (also known as LM) is used in JEM. For this mode, chromaticity samples are predicted based on reconstructed luminance samples from the same CU using the following linear model:

[0377] pred C (i, j) = α·rec L ′(i,j)+β

[0378] In this article, pred C (i, j) represent the predicted chromaticity samples in the CU, rec L ′(i,j) represents the reconstructed luminance sample points of the same CU for downsampling in color format 4:2:0 or 4:2:2, while rec L ′(i,j) represents the reconstructed luminance sample for the same CU in color format 4:4:4. The CCLM parameters α and β are derived by minimizing the regression error between neighboring reconstructed luminance and chrominance samples around the current block, as follows:

[0379]

[0380] Here, L(n) represents the top and left adjacent reconstructed luminance samples of the downsampled (for color formats 4:2:0 or 4:2:2) or original (for color format 4:4:4) samples, C(n) represents the top and left adjacent reconstructed chroma samples, and the value of N is equal to twice the minimum of the width and height of the current chroma codec block. For square codec blocks, the above two formulas are applied directly. For non-square codec blocks, the adjacent samples of the longer boundary are first subsampled to have the same number of samples as the shorter boundary. Figure 25 The location of the reconstructed samples on the left and top, as well as the samples of the current block involved in the CCLM mode, are shown.

[0381] This regression error minimization calculation is performed as part of the decoding process, not just the encoder search operation, and therefore the α and β values ​​are not conveyed using syntax. The CCLM prediction mode also includes prediction between the two chrominance components, i.e., predicting the Cr component from the Cb component. The CCLM Cb-to-Cr prediction is applied in the residual domain, rather than using the reconstructed sampled signal. This is achieved by adding a weighted reconstructed Cb residual to the original Cr intra-frame prediction to form the final Cr prediction:

[0382]

[0383] Here, resi Cb ′(i,j) represents the reconstructed Cb residual sample at position (i,j).

[0384] The scaling factor α is derived in a similar manner to CCLM's luminance-to-chrominance prediction. The only difference is the addition of regression costs relative to the default α value to the error function, causing the derived scaling factor to bias towards the default value of -0.5, as shown below:

[0385]

[0386] Here, Cb(n) represents adjacent reconstructed Cb samples, Cr(n) represents adjacent reconstructed Cr samples, and λ equals ∑(Cb(n)·Cb(n))>>9.

[0387] The CCLM luma-to-chroma prediction mode is added as an additional intra-chroma prediction mode. On the encoder side, an additional RD cost check for the chroma component is added for selecting the intra-chroma prediction mode. When an intra-prediction mode other than CCLM luma-to-chroma prediction mode is used for the chroma component of the CU, CCLM Cb-to-Cr prediction will be used for Cr component prediction.

[0388] 3.2 Simplified CCLM Example

[0389] In some existing implementations, the LMS algorithm replaces the linear model parameters α and β with a linear equation. These two points (luminance and chrominance pair) (A, B) are the minimum and maximum values ​​within the set of adjacent luminance samples, such as... Figure 26 As shown. Here, the linear model parameters α and β are obtained according to the following formula:

[0390] And β = y A -αx A .

[0391] Division can be replaced and avoided by multiplication and shifting.

[0392] For the current VTM implementation, in order to derive the chroma predictor, multiplication is replaced with the following integer operations:

[0393]

[0394] However, since the shift S always has the same value, this implementation is also simpler than the current VTM implementation. In terms of complexity, the proposed algorithm reduces the number of computations, as shown in the table below:

[0395]

[0396] Although the number of calculations is reduced, the proposed method still requires several comparisons to obtain the minimum and maximum brightness values ​​of adjacent samples.

[0397] 4. Disadvantages of existing implementation methods

[0398] Some existing implementations have the following drawbacks:

[0399] (1) Both BIO and CCLM require linear regression, but the simplification methods are different.

[0400] (2) The BIO implementation in VTM does not take into account the POC distance.

[0401] (3) Express the velocity on Ref 0 as: and The velocity on Ref 1 is expressed as

[0402] and In the current BIO design, it is assumed that the speeds on the two reference images are mirror images, such that... and (or and However, this assumption may be incorrect.

[0403] (4) Velocity v in BIO x and v y The derivation is oversimplified.

[0404] (5) BIO in v x and v y The derivation can be combined with the motion derivation on the DMVR or decoder side.

[0405] (6) I know v x and v y Then, the predictions / gradients on Ref 0 / Ref 1 can be further refined.

[0406] (7) An additional interpolation filter (bilinear filter) was introduced to interpolate pixels and boundaries.

[0407] (8) Some rounding and trimming operations are missing in the BIO process.

[0408] (9) The precision of internal calculations can be further reduced.

[0409] (10) It can simplify the SAD used for BIO calculations.

[0410] (11) When applying GBi, the weighting value was not properly considered in the BIO derivation process.

[0411] 5. Example methods for refined inter-frame prediction

[0412] The embodiments of the currently disclosed technology overcome the shortcomings of existing implementations, harmonize the linear regression process in BIO and CCLM, and propose BIO modifications, thereby providing video codecs with higher encoding and decoding efficiency. Based on the disclosed technology, the harmonization of linear mode prediction can enhance existing and future video codec standards, as illustrated in the examples described below for various implementations. The examples of the disclosed technology provided below illustrate general concepts and are not intended to be construed as limiting. In the examples, the various features described in these examples can be combined unless explicitly indicated otherwise.

[0413] Let the reference images of the current image from list 0 and list 1 be Ref0 and Ref1, respectively, and let τ0 = POC(current) - POC(Ref0), τ1 = POC(Ref1) - POC(current). Let the reference blocks of the current block from Ref0 and Ref1 be refblk0 and refblk1, respectively. For a sub-block in the current block, the original MV of the corresponding sub-block in refblk0 pointing to refblk1 is represented as (v x ,v y The MV of the sub-blocks in Ref0 and Ref1 are respectively determined by (mvL0). x ,mvL0 y ) and (mvL1 x mvL1 y )express.

[0414] Define Shift(x,s) as Shift(x,s) = (x + off) >> s

[0415] SignShift(x,s) is defined as:

[0416]

[0417] Here, off represents an integer offset, such as 0 or 2. s-1 .

[0418] Clip3(x,min,max) is defined as:

[0419]

[0420] Example 1. It is suggested that the linear optimization module used in BIO can be used by other codec tools in the video codec system.

[0421] (a) The linear optimization module involves a set of sample points u k ,v k ,wk ), where k (from 0 to N-1) is the input and two parameters a and b are the outputs, such that It can be minimized or approximately minimized. From equations (9) and (10), we can obtain that in BIO:

[0422] u k =I (0) (x,y)-I (1) (x,y)

[0423]

[0424] a = v x

[0425] b = v y

[0426] Here, G x (x,y) and G y (x, y) represent the horizontal and vertical gradients, respectively.

[0427] (b) In one example, the linear optimization module used in BIO is used to derive the parameters of CCLM. From equations (18) and (19), the notation can be written as:

[0428] u k =-C(n)

[0429] v k =L(n)

[0430] w k =1

[0431] a=α

[0432] b = β

[0433] (c) Alternatively, the linear optimization module used in another encoding / decoding tool (such as CCLM) can be used to derive v in BIO. x and v y .

[0434] Example 2. Whether to apply the BIO process and how to apply it depends on the POC distance.

[0435] (a) If abs(τ0)≥T0 or abs(τ1)≥T1, then the BIO procedure is not applied. T0 and T1 are integers, for example, T0 = T1 = 4. T0 and T1 can be fixed numbers or sent from the encoder to the decoder in VPS / SPS / PPS / strip header / piece group header / piece / CTU / CU.

[0436] (b) If abs(τ0)≥T0 and abs(τ1)≥T1, then the BIO procedure is not applied. T0 and T1 are integers, for example, T0 = T1 = 4. T0 and T1 can be fixed numbers or sent from the encoder to the decoder in VPS / SPS / PPS / strip header / piece group header / piece / CTU / CU.

[0437] (c) If abs(τ0) + abs(τ1) ≥ T, then the BIO procedure is not applied. T is an integer, for example, T = 8. T can be a fixed number or a signal that the encoder informs the decoder in VPS / SPS / PPS / strip header / piece group header / piece / CTU / CU.

[0438] (d) If abs(abs(τ0)-abs(τ1))≥T, then the BIO procedure is not applied. T is an integer, for example, T=8. T can be a fixed number or a signal that the encoder informs the decoder in VPS / SPS / PPS / strip header / piece group header / piece / CTU / CU.

[0439] (e) If abs(τ0) ≥ T*abs(τ1) or abs(τ1) ≥ T*abs(τ0), then the BIO procedure is not applied. T is an integer, for example, T = 4. T can be a fixed number or a signal that the encoder informs the decoder in VPS / SPS / PPS / strip header / piece group header / piece / CTU / CU.

[0440] Example 3. The velocity vector used to refine the predicted values ​​during the BIO process is suggested to depend on the POC distance. The prediction calculation after the BIO process is as follows:

[0441]

[0442] Among them G (0) x and G (0) y G represents the horizontal and vertical gradients of Ref0, and G (0) x and G (0) y Represents the horizontal and vertical gradients of Ref0.

[0443] (i) In the BIO design of VTM-3.0, and Where v x and v y It is derived. Then equation (22) is the same as equation (9).

[0444] (ii) In one example:

[0445]

[0446] In this paper, in HEVC, division can be implemented as multiplication and shifting, as an MV scaling operation.

[0447] Example 4. The substitution always begins with the assumption that v y To derive v from 0 x It is recommended to export the v file first. y And based on v y To derive v x Alternatively, it is possible to first derive v. y or v x It can depend on encoding and decoding information, such as the values ​​of the horizontal and vertical components of the motion vector, block size, etc.

[0448] Example 5. It is recommended to derive v iteratively. x and v y .

[0449] a. In one example, v is derived in step i. x Used to derive v in step i y And the v derived in step i y Used to derive v in step i+1 x The derivation is performed iteratively.

[0450] b. Alternatively, v derived in step i y Used to derive v in step i x And the v derived in step i x v used to derive the derived value in step i+1 y The derivation is performed iteratively.

[0451] c. In the same step, v is used x To derive v y Still using v y To derive v x It may depend on encoding and decoding information, such as the values ​​of the horizontal and vertical components of the motion vector, block size, etc.

[0452] d. In one example, s2, s5, and s6 are defined as in (13). ε is an integer or a decimal, such as 1 / 2 or 1 / 4.

[0453] i. For example, division operations can be simplified. Replace it with the MSB shift in JEM-3.0.

[0454] ii. A cut operation can be performed after a division operation.

[0455] e. In one example, s1, s2, and s3 are defined as in (13). ε is an integer or a decimal, such as 1 / 2 or 1 / 4.

[0456] i. For example, division operations can be simplified. Replace it with the MSB shift in JEM-3.0.

[0457] ii. A cut operation can be performed after a division operation.

[0458] f. The derivation is iterative, continuing until i reaches a prefix number (such as 2).

[0459] g. Alternatively, the derivation is iterative, up to and including a round before and after v. x and / or v y The absolute difference between them is less than the threshold.

[0460] Example 6 You can use v in BIO x and v y The derivation is combined with DMVR, bilateral matching, or other decoder-side motion derivation methods.

[0461] a. The v derived in BIO for blocks or sub-blocks (such as 4×4 blocks) x and v y This can be used to derive the search starting point of DMVR, bilateral matching, or other decoder-side motion derivation methods. Assuming the original MVs of Ref0 and Ref1 are MV0 and MV1, then MV0' and MV1', denoted as the starting points of DMVR, can be calculated using bilateral matching or other decoder-side motion derivation methods:

[0462] i. In one example, MV0' = MV0 + (v x ,v y ),MV1'=MV1-(v x ,v y );

[0463] ii. In one example, MV0' = MV0 - (v x ,v y ),MV1'=MV1+(v x ,v y );

[0464] iii. In one example, MV0' = MV0 + Scale(v x ,v y ),MV1'=MV1-Scale(v x ,v y );

[0465] iv. In one example, MV0' = MV0 - Scale(v x ,v y ),MV1'=MV1+Scale(v x ,v y );

[0466] v. In one example, MV0' = Clip(MV0 + Scale(v x ,v y )),MV1'=Clip(MV1-Scale(v x ,v y ));

[0467] vi. In one example, MV0' = Clip(MV0 - Scale(v x ,v y )),MV1'=Clip(MV1+Scale(v x ,v y ));

[0468] b. Alternatively, the output MV of DMVR, bilateral matching, or other decoder-side motion derivation methods for blocks or sub-blocks (such as 4×4 blocks) can be used to derive v used in BIO. x and v y Assuming the original MVs of Ref0 and Ref1 are MV0 and MV1, and the output MVs are MV0' and MV1', then v x and v y Calculated as

[0469] i. In one example, (v x ,v y =Scale(MV0'-MV0);

[0470] ii. In one example, (v x ,v y =Scale(MV0-MV0');

[0471] iii. In one example, (v x ,v y =Scale(MV1'–MV1);

[0472] iv. In one example, (v x ,v y =Scale(MV1–MV1');

[0473] v. In one example, (v x ,v y)=(Scale(MV0'-MV0)+Scale(MV1'–MV1)) / 2;

[0474] vi. In one example, (v x ,v y )=(Scale(MV0-MV0')+Scale(MV1–MV1')) / 2;

[0475] Example 7 Suggested derivation of v x and v y This can be used to refine the predictions and gradients on Ref0 and Ref1. Then, the refined predictions and gradients are used to derive the new v. x and v y .

[0476] a. The derivation and refinement process can be completed iteratively, up to the v before and after a round of derivation. x or v y The absolute difference between them is less than a threshold. Alternatively, the derivation and refinement process can be performed iteratively until a predetermined number of iterations is reached. For example, the number of iterations is 2.

[0477] b. Assuming the original MVs of Ref0 and Ref1 are MV0 and MV1, then the MVs represented as MV0' and MV1' can be used to obtain refined predictions and gradients, which can be calculated as follows:

[0478] i. In one example, MV0' = MV0 + (v x ,v y ),MV1'=MV1-(v x ,v y );

[0479] ii. In one example, MV0' = MV0 - (v x ,v y ),MV1'=MV1+(v x ,v y );

[0480] iii. In one example, MV0' = MV0 + Scale(v x ,v y ),MV1'=MV1-Scale(v x ,v y );

[0481] iv. In one example, MV0' = MV0 - Scale(v x ,v y ),MV1'=MV1+Scale(v x ,v y );

[0482] v. In one example, MV0' = Clip(MV0 + Scale(v x ,v y )),MV1'=Clip(MV1-Scale(v x ,v y ));

[0483] vi. In one example, MV0' = Clip(MV0 - Scale(v x ,v y )),MV1'=Clip(MV1+Scale(v x ,v y ));

[0484] Example 8. It is recommended that BIO be applied to a subset of samples within a block or a sub-block. In one example, BIO is not applied to samples located in the first / last row / first / last column.

[0485] Example 9. It is recommended to use only one interpolation filter for a single color component when applying BIO.

[0486] a. In one example, when applying BIO, only an 8-tap interpolation filter is used on the luminance component.

[0487] b. Alternatively, in order to reduce memory bandwidth, it is recommended to limit the size / range of the samples to be extracted to be smaller than the size / range required by the interpolation filter.

[0488] c. Alternatively, if some sample points are located outside a given size / range, padding can be applied.

[0489] d. Assuming the current block is M×N, the required size of the subsample is (M+G)×(N+G), and the required size of the L-tap interpolation filter should be (M+G+L-1)×(N+G+L-1). It is recommended that the allowed size of the integer luminance samples extracted during the BIO process be (M+L-1+k)×(N+L-1+k). For example, k is 0 or 1, and k is less than G. If the interpolation process requires integer luminance samples but does not allow their extraction, the integer luminance samples will be filled by adjacent luminance samples.

[0490] i. Figure 27An example of interpolation filtering in the BIO process is shown. In the example, the current block size is M×N, where M=N=8. Due to gradient calculation, the required size of the subsamples is (M+2)×(N+2). Therefore, the integer samples required by the interpolation filter are (M+2+7)×(N+2+7), which in this example equals 17×17. However, only (M+7)×(N+7), which in this example equals 15×15, requires the extraction of integer samples. The other samples required by the interpolation filter (black circles in the filter) will be filled with adjacent extracted samples.

[0491] e. In one example, the gradients at positions (x, y) where x = -1, y = -1, x = W, or y = H are not calculated, and BIO is not applied to these positions. Assume the top-left corner of the current block is (0, 0), and the width / height of the current block is W / H.

[0492] f. In one instance, the gradient of the position (x, y) where x = -1, y = -1, x = W, or y = H is calculated in different ways. For example,

[0493] ○gradientHL0[x][y]=

[0494] (predSamplesL0[hx+1][vy]–predSampleL0[hx][vy])>>4, if hx==1, (predSamplesL0[hx][vy]–predSam pleL0[hx-1][vy])>>4, if hx==nCbW, (predSamplesL0[hx+1][vy]–predSampleL0[hx-1][vy])>>4, in other cases. ○gradientVL0[x][y]=

[0495] (predSampleL0[hx][vy+1]-predSampleL0[hx][vy])>>4, if vy==1,

[0496] (predSampleL0[hx][vy]-predSampleL0[hx][vy-1])>>4, if vy==nCbH,

[0497] (predSampleL0[hx][vy+1]-predSampleL0[hx][vy-1])>>4, in other cases. ○gradientHL1[x][y]=

[0498] (predSamplesL1[hx+1][vy]–predSampleL1[hx][vy])>>4, if hx==1,(predSamplesL1[hx][vy]–predSampleL1[hx-1][vy])>>4, if hx==nCbW,

[0499] (predSamplesL0[hx+1][vy]–predSampleL1[hx-1][vy])>>4,other situations。○gradientVL1[x][y]=

[0500] (predSampleL1[hx][vy+1]–predSampleL1[hx][vy])>>4, if vy==1,

[0501] (predSampleL1[hx][vy]–predSampleL1[hx][vy-1])>>4, if vy==nCbH,

[0502] (predSampleL1[hx][vy+1]–predSampleL1[hx][vy-1])>>4,other circumstances。

[0503] In other examples,

[0504] ○gradientHL0[x][y]=

[0505] (predSamplesL0[hx+1][vy]–predSampleL0[hx][vy])>>3, if hx==1,(predSamplesL0[hx][vy]–predSampleL0[hx-1][vy])>>3, if hx==nCbW,(predSamplesL0[hx+1][vy]–predSampleL0[hx-1][vy])>>4, otherwise. ○gradientVL0[x][y]=

[0506] (predSampleL0[hx][vy+1]-predSampleL0[hx][vy])>>3, if vy==1,

[0507] (predSampleL0[hx][vy]-predSampleL0[hx][vy-1])>>3, if vy==nCbH,

[0508] (predSampleL0[hx][vy+1]-predSampleL0[hx][vy-1])>>4, in other cases. ○gradientHL1[x][y]=

[0509] (predSamplesL1[hx+1][vy]–predSampleL1[hx][vy])>>3, if hx==1, (predSamplesL1[hx][vy]–predSampleL1[hx-1][vy])>>3, if hx==nCbW,

[0510] (predSamplesL0[hx+1][vy]–predSampleL1[hx-1][vy])>>4, in other cases. ○gradientVL1[x][y]=

[0511] (predSampleL1[hx][vy+1]–predSampleL1[hx][vy])>>3, if vy==1, (predSampleL1[hx][vy]–predSampleL1[hx][vy-1])>>3, if vy==nCbH,

[0512] (predSampleL1[hx][vy+1]–predSampleL1[hx][vy-1])>>4, in other cases.

[0513] g. In one example, before calculating the gradient, the external samples ( Figure 23 The black circles in the diagram are not interpolated using fill.

[0514] i. For example, for all valid hx and vy:

[0515] predSampleL0[0][vy]=predSampleL0[1][vy],

[0516] predSampleL0[nCbW+1][vy]=predSampleL0[nCbW][vy],

[0517] predSampleL0[hx][0]=predSampleL0[hx][1],

[0518] predSampleL0[hx][nCbH+1]=predSampleL0[hx][nCbH]and

[0519] predSampleL1[0][vy]=predSampleL1[1][vy],

[0520] predSampleL1[nCbW+1][vy]=predSampleL1[nCbW][vy],

[0521] predSampleL1[hx][0]=predSampleL1[hx][1],

[0522] predSampleL1[hx][nCbH+1]=predSampleL1[hx][nCbH].

[0523] h. In one example, the gradient calculation method in BIO and the gradient calculation method in Adaptive Loop Filter (ALF) are the same.

[0524] i. In one example, the gradient calculation method of BIO in VTM-3 is also used to calculate the gradient of ALF.

[0525] ii. In one example, the gradient calculation method of ALF in VTM-3 was also used to calculate the gradient of BIO.

[0526] Example 10: The following changes were proposed to the bidirectional optical flow prediction process.

[0527] a. The gradient value can be shifted using a value other than 4. In one example, the variables gradientHL0[X][Y], gradientVL0[X][Y], gradientHL1[X][Y], and gradientVL1[X][Y] are derived as follows:

[0528] gradientHL0[x][y]=SignShift(predSamplesL0[hx+1][vy]–predSampleL0[hx-1][vy],S)

[0529] gradientVL0[x][y]=SignShift(predSampleL0[hx][vy+1]-predSampleL0[hx][vy-1],S)

[0530] gradientHL1[x][y]=SignShift(predSamplesL1[hx+1][vy]–predSampleL1[hx-1][vy],S)

[0531] gradientVL1[x][y]=SignShift(predSampleL1[hx][vy+1]–predSampleL1[hx][vy-1],S)

[0532] i. In one example, S is a fixed number such as 4 or 5.

[0533] ii. In one example, S depends on the sample bit depth. For example, S equals BP, where B is a sample bit depth such as 8, 10, or 12, and P is an integer such as 6.

[0534] b. The gradient values ​​should be within a certain range. In one example, the variables gradientHL0[x][y], gradientVL0[x][y], gradientHL1[x][y], and gradientVL1[x][y] must be represented by K integers, such as K = 8 or 16.

[0535] i. For example, after the derivation, the gradient is clipped to

[0536] gradientHL0[x][y]=Clip3(-2 K-1 ,2 K-1 -1,gradientHL0[x][y])

[0537] gradientVL0[x][y]=Clip3(-2 K-1 ,2 K-1 -1,gradientVL0[x][y])

[0538] gradientHL1[x][y]=Clip3(-2 K-1 ,2 K-1 -1,gradientHL1[x][y])

[0539] gradientVL1[x][y]=Clip3(-2 K-1 ,2 K-1 -1,gradientVL1[x][y])

[0540] c. The derivation of the internal variables temp, tempX, and tempY is as follows:

[0541] temp[x][y]=SignShift(predSamplesL0[hx][vy]–predSamplesL1[hx][vy],S1)

[0542] tempX[x][y]=SignShift(gradientHL0[x][y]+gradientHL1[x][y],S2)

[0543] tempY[x][y]=SignShift(gradientVL0[x][y]+gradientVL1[x][y],S3)

[0544] i. In one example, S1, S2, and S3 are fixed numbers, such as S1 = 6, S2 = S3

[0545] =3.

[0546] ii. In one example, S1, S2, and S3 depend on the sample bit depth. For example, S1

[0547] =B-P1, S2=B-P2 and S3=B-P3, where B is a sample bit depth such as 8, 10 or 12, and P1, P2 and P3 are integers, for example, P1=4, P2=P3=7.

[0548] d. It must be ensured that the internal variables temp, tempX and tempY are represented by K1-bit integers, K2-bit integers and K3-bit integers, for example, K1 = 8 or 16, K2 = 8 or 16, K3 = 8 or 16.

[0549] i. For example, trimming the internal variables after derivation to

[0550] temp[x][y] = Clip3(-2) K1-1 ,2 K1-1 -1,gradientHL0[x][y])

[0551] tempX[x][y] = Clip3(-2) K2-1 ,2 K2-1 -1,gradientVL0[x][y])

[0552] tempY[x][y]=Clip3(-2 K3-1 ,2 K3-1 -1,gradientHL1[x][y])

[0553] e. The internal variables sGx2, sGy2, sGxGy, sGxdI, and sGydI should be within a certain range. In one example, these variables must be represented by K1, K2, K3, K4, and K5 integers, for example, K1 = 8 or 16, K2 = 8 or 16, and K3 = 8 or 16.

[0554] i. For example, right-shift the internal variables after derivation as follows

[0555] sGx2 = Shift(sGx2, S1)

[0556] sGy2 = Shift(sGy2, S2)

[0557] sGxGy = SignShift(sGxGy, S3)

[0558] sGxdI = SignShift(sGxdI, S4)

[0559] sGydI = SignShift(sGydI, S5)

[0560] In one example, S1, S2, S3, S4, and S5 are fixed numbers such as 4 or 5.

[0561] Alternatively, S1, S2, S3, S4, and S5 depend on the sample bit depth. For example, S1 = B - P1, S2 = B - P2, S3 = B - P3, S4 = B - P4, and S5 = B - P5, where B is a sample bit depth such as 8, 10, or 12, and P1, P2, P3, P4, and P5 are integers.

[0562] ii. For example, trimming the internal variables after derivation to

[0563] sGx2 = Clip3(0,2) K1-1 ,sGx2)

[0564] sGy2 = Clip3(0,2) K2-1 ,sGy2)

[0565] sGxGy=Clip3(-2 K3-1 ,2 K3-1 -1,sGxGy)

[0566] sGxdI=Clip3(-2 K4-1 ,2 K4-1 -1,sGxdI)

[0567] sGydI=Clip3(-2 K5-1 ,2 K5-1 -1,sGydI)

[0568] f. Derive the variables gradientHL0[x][y], gradientVL0[x][y], gradientHL1[x][y], and gradientVL1[x][y] only for the selected positions.

[0569] i. In one example, these samples are calculated only for the points (x, y) where x%Q==0. For example, Q=2

[0570] ii. In one example, these samples are calculated only for the points at (x,y) where x%Q==1. For example, Q=2

[0571] iii. In one example, these samples are calculated only for the points (x, y) where y%Q==0. For example, Q=2

[0572] iv. In one example, these are calculated only for the sample points at the location (x,y) where y%Q==1. For example, Q=2

[0573] v. In one example, calculate the samples only for the points (x, y) at the location y%Q==0 or y%Q==3. For example, Q=4;

[0574] g. Derive the internal variables temp[x[y], tempX[x][y], and tempY[x][y] only for selected locations:

[0575] i. In one example, these samples are calculated only for the points (x, y) where x%Q == 0. For example, Q = 2;

[0576] ii. In one example, these samples are calculated only for the points at (x,y) where x%Q==1. For example, Q=2;

[0577] iii. In one example, calculation is performed only for the sample point at the location (x,y) where y%Q==0.

[0578] They. For example, Q = 2;

[0579] iv. In one example, calculation is performed only for the sample point at the position (x,y) where y%Q==1.

[0580] They. For example, Q = 2;

[0581] v. In one example, calculate the samples only for the points (x, y) at the location y%Q==0 or y%Q==3. For example, Q=4;

[0582] vi. In one example, calculate temp[x[y], tempX[x][y] and tempY[x][y], gradientHL0[x][y], gradientVL0[x][y], gradientHL1[x][y], gradientVL1[x][y] for samples at the same location (e.g., the locations in example gi~gv).

[0583] h. Calculate the internal variables sGx2, sGy2, sGxGy, sGxdI, and sGydI only for the cumulative sample points at selected locations. The formula is as follows:

[0584] sGx2=∑ x ∑ y (tempX[xSb+x][ySb+y]*tempX[xSb+x][ySb+y]), and x,y∈S

[0585] sGy2=∑ x ∑ y (tempY[xSb+x][ySb+y]*tempY[xSb+x][ySb+y]), and x,y∈S

[0586] sGxGy=∑ x ∑ y (tempX[xSb+x][ySb+y]*tempY[xSb+x][ySb+y]), and x,y∈S

[0587] sGxdI=∑ x ∑ y (-tempX[xSb+x][ySb+y]*temp[xSb+x][ySb+y]), and x,y∈S

[0588] sGydI=∑ x ∑ y (-tempY[xSb+x][ySb+y]*temp[xSb+x][ySb+y]), and x,y∈S

[0589] Where S is the set of selected locations.

[0590] i. In one example, the selected positions are x = 0, 1, 2, 3 and y = 0, 1, 2, 3;

[0591] ii. In one example, the selected positions are x = 0, 2 and y = 0, 1, 2, 3;

[0592] iii. In one example, the selected positions are x = 1, 2 and y = 0, 1, 2, 3;

[0593] iv. In one example, the selected positions are x = 1, 3 and y = 0, 1, 2, 3;

[0594] v. In one example, the selected positions are x = 2, 3 and y = 0, 1, 2, 3;

[0595] vi. In one example, the selected positions are x = 0, 3 and y = 0, 1, 2, 3;

[0596] vii. In one example, the selected positions are y = 0, 2 and x = 0, 1, 2, 3;

[0597] viii. In one example, the selected positions are y = 1, 2 and x = 0, 1, 2, 3;

[0598] ix. In one example, the selected positions are y = 1, 3 and x = 0, 1, 2, 3;

[0599] x. In one example, the selected positions are y = 2, 3 and x = 0, 1, 2, 3;

[0600] xi. In one example, the selected positions are y = 0, 3 and x = 0, 1, 2, 3;

[0601] xii. In one example, the selected positions are x = -1, 4 and y = -1, 0, 1, 2, 3, 4;

[0602] xiii. In one example, the selected positions are x = 0, 3 and y = -1, 0, 1, 2, 3, 4;

[0603] xiv. In one example, the selected positions are x = 1, 2 and y = -1, 0, 1, 2, 3, 4;

[0604] xv. In one example, the selected positions are x = -1, 1, 3 and y = -1, 0, 1, 2, 3, 4;

[0605] xvi. In one example, the selected positions are x = 0, 2, 4 and y = -1, 0, 1, 2, 3, 4;

[0606] xvii. In one example, the selected positions are x = -1, 1, 2, 4 and y = -1, 0, 1, 2, 3, 4;

[0607] xviii. In one example, the selected positions are x = 0, 1, 2, 3 and y = -1, 0, 1, 2, 3, 4;

[0608] xix. In one example, the selected positions are y = -1, 4 and x = -1, 0, 1, 2, 3, 4;

[0609] In one example, the selected positions are y = 0, 3 and x = -1, 0, 1, 2, 3, 4;

[0610] xxi. In one example, the selected positions are y = 1, 2 and x = -1, 0, 1, 2, 3, 4;

[0611] xxii. In one example, the selected positions are y = -1, 1, 3 and x = -1, 0, 1, 2, 3, 4;

[0612] xxiii. In one example, the selected positions are y = 0, 2, 4, and x = -1, 0, 1, 2, 3, 4;

[0613] xxiv. In one example, the selected positions are y = -1, 1, 2, 4 and x = -1, 0, 1, 2, 3, 4;

[0614] xxv. In one example, the selected positions are y = 0, 1, 2, 3 and x = -1, 0, 1, 2, 3, 4;

[0615] i. Used to derive v x and v y The division operation is replaced in a more complex way.

[0616] i. In one example, vx = sGx2 > 0? Clip3(-mvRefineThres,mvRefineThres,-(sGxdI<<3)>>M):0. Based on the value of sGx2, M could be...

[0617] Floor(Log2(sGx2)) or Ceiling(Log2(sGx2)). For example, if 3*sGx2 is greater than 2. Floor(Log2 (sGx2))+2 If so, then M is Ceiling(Log2(sGx2)); otherwise, M is...

[0618] Floor(Log2(sGx2)). In another example, if sGx2 is greater than T, then M is...

[0619] Ceiling(Log2(sGx2)) is used; otherwise, M is Floor(Log2(sGx2)). For example...

[0620] T = (Floor(Log2(sGx2)) + Ceiling(Log2(sGx2))) / 2. In another example,

[0621] If sGx2*sGx2 is greater than 2 2*Floor(Log2(sGx2))+1 Then M is Ceiling(Log2(sGx2)).

[0622] Otherwise, M is Floor(Log2(sGx2)).

[0623] 1. Alternatively, vx = sGx2 > 0? Clip3(-mvRefineThres,mvRefineThres,-((sGxdI<<3)+Offset)>>M):0. Offset is an integer, such as 1<<(M-1).

[0624] a.Offset can depend on sGx2.

[0625] ii. In one example, vy = sGy2 > 0? Clip3(-mvRefineThres,mvRefineThres,((sGydI<<3)-((vx*sGxGym)<<12+vx*sGxGys)>>1)>>M:0. Depending on the value of sGy2, M can be Floor(Log2(sGy2)) or Ceiling(Log2(sGy2)). For example, if 3*sGy2 is greater than 2 Floor(Log2(sGy2))+2 If sGy2 is greater than T, then M is Ceiling(Log2(sGy2)); otherwise, M is Floor(Log2(sGy2)). In another example, if sGy2 is greater than T, then M is Ceiling(Log2(sGy2)); otherwise, M is Floor(Log2(sGy2)). For example, T = (Floor(Log2(sGy2)) + Ceiling(Log2(sGy2))) / 2. In another example, if sGy2 * sGy2 is greater than 2... 2*Floor(Log2(sGy2))+1 If so, then M is Ceiling(Log2(sGy2)); otherwise, M is Floor(Log2(sGy2)).

[0626] 1. Alternatively, vy = sGy2 > 0? Clip3(-mvRefineThres,mvRefineThres,(((sGydI<<3)-((vx*sGxGym)<<12+vx*sGxGys)>>1)+Offset)>>M:0. Offset is an integer, such as 1<<(M-1).

[0627] a.Offset can depend on sGy2.

[0628] iii. sGxGym and sGxGys are calculated based on the sign of sGxGy. Assume that if x>=0, then sign(x)=1; if x<0, then sign(x)=-1. Therefore, sGxGym=sign(sGxGy)*|sGxGy|>>W; sGxGys=sign(sGxGy)*(|sGxGy|&((1< <W)-1))。

[0629] 1.W can be a fixed number such as 12. Or it can depend on the sample bit depth.

[0630] iv. Division operations in the BIO process are performed using a lookup table.

[0631] (i) The same lookup table is also used in CCLM to replace division operations.

[0632] Example 11. The decision to enable or disable BIO based on the SAD calculations of the entire block and sub-blocks can be simplified by calculating only the SAD calculation of sub-blocks. Alternatively, SAD calculations can be replaced by other rules such as MR-SAD.

[0633] Example 12. In BIO, SAD calculations are performed only on samples at selected locations.

[0634] a.subCuLevelSumAbsoluteDiff[xSbIdx][SbIdx]=∑ i ∑ j Abs(predSamplesL0L[(xSbIdx<<2)+1+i][(ySbIdx<<2)+1+j]-predSamplesL1L[(xSbIdx<<2)+1+i][(ySbIdx<<2)+1+j]), and i, j∈S

[0635] Where S is the set of selected locations.

[0636] i. In one example, the selected positions are i = 0, 2 and j = 0, 1, 2, 3;

[0637] ii. In one example, the selected positions are i = 1, 2 and j = 0, 1, 2, 3;

[0638] iii. In one example, the selected positions are i = 1, 3 and j = 0, 1, 2, 3;

[0639] iv. In one example, the selected positions are i = 2, 3 and j = 0, 1, 2, 3;

[0640] v. In one example, the selected positions are i = 0, 3 and j = 0, 1, 2, 3;

[0641] vi. In one example, the selected positions are j = 0, 2 and i = 0, 1, 2, 3;

[0642] vii. In one example, the selected positions are j = 1, 2 and i = 0, 1, 2, 3;

[0643] viii. In one example, the selected positions are j = 1, 3 and i = 0, 1, 2, 3;

[0644] ix. In one example, the selected positions are j = 2, 3 and i = 0, 1, 2, 3;

[0645] x. In one example, the selected positions are j = 0, 3 and i = 0, 1, 2, 3;

[0646] xi. In one example, the selected positions are i = 0, 3 and j = 0, 3;

[0647] xii. In one example, the selected positions are i = 1, 2 and j = 1, 2;

[0648] b. Alternatively, subCuLevelSumAbsoluteDiff[xSbIdx][ySbIdx] = max i,j Abs(predSamplesL0L[(xSbIdx<<2)+1+i][(ySbIdx<<2)+1+j]-predSamplesL1L[(xSbIdx<<2)+1+i][(ySbIdx<<2)+1+j]), and i, j∈S

[0649] Where S is the set of selected locations.

[0650] c. Alternatively, subCuLevelSumAbsoluteDiff[xSbIdx][ySbIdx] = min i,j Abs(predSamplesL0L[(xSbIdx<<2)+1+i][(ySbIdx<<2)+1+j]-predSamplesL1L[(xSbIdx<<2)+1+i][(ySbIdx<<2)+1+j]) with i, j∈S

[0651] Where S is the set of selected locations.

[0652] d. The threshold subCuLevelAbsDiffThres can be adaptive.

[0653] i. It can depend on encoding and decoding information such as QP and POC distance.

[0654] ii. It can be sent from the encoder to the decoder in VPS / SPS / PPS / strip header / piece group header / piece / CTU CU.

[0655] Example 13. In BIO, SAD calculations are performed only on samples in selected sub-blocks.

[0656] a. Alternatively, the SAD calculation for each sub-block may involve only a subset of samples within a single sub-block.

[0657] b.cuLevelSumAbsoluteDiff=∑ xSbIdx ∑ ySbIdx subCuLevelSumAbsoluteDiff[xSbIdx][ySbIdx]with xSbIdx,ySbIdx∈S

[0658] Where S is the set of selected sub-blocks.

[0659] i. In one example, the selected position is xSbIdx%2==0;

[0660] ii. In one example, the selected position is xSbIdx%2==1;

[0661] iii. In one example, the selected position is xSbIdx%4==0;

[0662] iv. In one example, the selected position is either xSbIdx == 0 or xSbIdx == (cbWidth >> 2) - 1;

[0663] v. In one example, the selected position is xSbIdy%2==0;

[0664] vi. In one example, the selected position is xSbIdy%2==1;

[0665] vii. In one example, the selected position is xSbIdy%4==0;

[0666] viii. In one example, the selected position is either xSbIdy == 0 or xSbIdy == (cbHeight >> 2) - 1;

[0667] ix. In one example, the selected position is (xSbIdy == 0 or xSbIdy == (cbHeight >> 2) - 1), and (xSbIdy == 0 or xSbIdy == (cbHeight >> 2) - 1);

[0668] c.cuLevelSumAbsoluteDiff=Max xSbIdx,ySbIdx subCuLevelSumAbsoluteDiff[xSbIdx][ySbIdx], and xSbIdx, ySbIdx∈S

[0669] d.cuLevelSumAbsoluteDiff=Min xSbIdx,ySbIdxsubCuLevelSumAbsoluteDiff[xSbIdx][ySbIdx], and xSbIdx, ySbIdx∈S

[0670] e. The threshold cuLevelAbsDiffThres can be adaptive.

[0671] i. It can depend on encoding / decoding information such as QP and POC distance.

[0672] ii. It can depend on the encoding / decoding mode, such as AMVP mode, Merge mode, or MMVD (Merge with MV differences) mode.

[0673] iii. It can be sent from the encoder to the decoder in VPS / SPS / PPS / strip header / piece group header / piece / CTU CU.

[0674] Example 14 For Examples 12 and 13, the proposed method can also be applied to other cases when SAD is replaced by other rules. That is, the use of BIO can be determined by considering only a portion of the samples within a sub-block and / or a portion of the sub-block.

[0675] Example 15. How to deduce the variables in the BIO process can differ depending on whether GBi is applied or not.

[0676] a. In addition, alternatively, the derivation may differ for GBi with different weights.

[0677] b. Assuming that the weights of Ref0 and Ref1 in the GBi process are W0 and W1, the prediction blocks are weighted first before deriving the variables used in the BIO process (such as SAD calculation, gradient calculation).

[0678] i. Let predSamplesL0[x][y] and predSamplesL1[x][y] represent two prediction blocks. W0*predSamplesL0[x][y], predSamplesL1[x][y], and W1*predSamplesL1[x][y] are used as inputs to BIO.

[0679] c. Alternatively, before deriving the variables used in the BIO process, predSamplesL0[x][y] is pre-computed as Shift(W0*predSamplesL0[x][y],S0) and predSamplesL1[x][y] is pre-computed as Shift(W1*predSamplesL1[x][y],S1).

[0680] i.S0 and S1 can depend on the sample bit depth.

[0681] ii. S0 and S1 can depend on W0 and W1.

[0682] iii. S0 and S1 can be fixed numbers such as 2.

[0683] d. Alternatively, when generating the final prediction block for a block, the values ​​of the correction terms can also be weighted.

[0684] i. In the prior art, when deriving the parameters used for the BIO process, two unweighted prediction blocks are used as inputs to the BIO. However, the final prediction block is generated by weighting the prediction blocks and the correction terms.

[0685] ii. The weights applied to the correction term can be the same as the weights applied to the prediction block. Alternatively, different weights can be applied to the correction term.

[0686] Example 16. It is recommended that the corrected item values ​​be within a specific range.

[0687] (a) For example, trim the correction term to [minCorrection, maxCorrection].

[0688] i.minCorrection and maxCorrection can be fixed, for example, -32 and 32;

[0689] ii. minCorrection and maxCorrection can depend on the sample bit depth. For example, minCorrection = -(32 << (BitDepth - 8)), maxCorrection = 32 << (BitDepth - 8).

[0690] iii. minCorrection and maxCorrection can depend on the width and / or height of the block.

[0691] iv.minCorrection and maxCorrection can be signaled from the encoder to the decoder in VPS / SPS / PPS / strip header / piece group header / piece / CTU line / CTU / CU.

[0692] (b) For example, if the correction term is out of range, it is divided by a factor or shifted to the right.

[0693] Example 17. It is recommended that the final prediction output in BIO should be within a specific range.

[0694] (a) For example, the final prediction output in BIO is clipped to [minPred, maxPred].

[0695] i.minPred and maxPred can be fixed numbers, such as -32 and 32;

[0696] ii. minPred and maxPred can depend on the sample bit depth. For example, minPred = -

[0697] (32<<(BitDepth-8)), maxCorrection=32<<(BitDepth-8).

[0698] iii. minPred and maxPred can depend on the width and / or height of the block.

[0699] iv.minPred and maxPred can be signaled from the encoder to the decoder in VPS / SPS / PPS / strip header / piece group header / piece / CTU line / CTU / CU.

[0700] The above examples can be incorporated into the context of methods described below, such as methods 2810, 2820, 2830, 2840, 2850, and 2860, which can be implemented at a video decoder or video encoder.

[0701] Figure 28A A flowchart of an exemplary method for video processing is shown. Method 2810 includes, in step 2812, determining a first linear optimization model for conversion between a current block of video and a bitstream representation of video, for conversion using a first codec mode, the first linear optimization model being derived from a second linear optimization model for conversion using a second codec mode.

[0702] Method 2810 includes performing a conversion based on the determination in step 2814.

[0703] Figure 28B A flowchart of an exemplary method for video processing is shown. Method 2820 includes, in step 2822, enabling a first prediction mode or a second prediction mode different from the first prediction mode based on one or more picture sequence count (POC) parameters associated with a picture of the current block of the video, wherein the first prediction mode is a codec mode using optical flow.

[0704] Method 2820 includes, in step 2824, performing a conversion between the current block and the bitstream representation of the video based on a first mode or a second mode.

[0705] Figure 28C A flowchart of an exemplary method for video processing is shown. Method 2830 includes, in step 2832, continuously deriving one or more velocity vectors (v) associated with a reference image of the current block, based on encoding / decoding information associated with the current block of the video. x ,v y ).

[0706] Method 2830 includes, in step 2834, performing a conversion between the current block and a bitstream representation of the video based on one or more velocity vectors, the encoding / decoding information including the value of the horizontal component of the motion vector of the current block, the value of the vertical component of the motion vector of the current block, or the size of the current block.

[0707] Figure 28D A flowchart of an exemplary method for video processing is shown. Method 2840 includes, in step 2842, filtering each color component of the current block using a single type of interpolation filter when it is determined that an optical flow-based encoding / decoding mode has been enabled for the current block of the video.

[0708] Method 2840 includes, in step 2844, performing a conversion between the current block and the bitstream representation of the video based on a filtering operation.

[0709] Figure 28E A flowchart of an exemplary method for video processing is shown. Method 2850 includes, in step 2852, determining a gradient value calculation algorithm using an optical flow tool for the conversion between a current block of video and a bitstream representation of the video.

[0710] Method 2850 includes performing a conversion based on the determination in step 2854.

[0711] Figure 28F A flowchart of an exemplary method for video processing is shown. Method 2860 includes, in step 2862, making a decision on selectively enabling a codec mode for using optical flow for the current block based on a sum of absolute differences (SAD) calculation of one or more sub-blocks of the current block of the video.

[0712] Method 2860 includes, in step 2864, performing a conversion between the current block and the bitstream representation of the current block based on the decision.

[0713] Figure 28G A flowchart of an exemplary method for video processing is shown. Method 2870 includes, in step 2872, selectively enabling a general bidirectional prediction improvement (GBi) process based on the current block of the video, deriving one or more parameters of the encoding / decoding mode of the current block using optical flow.

[0714] Method 2870 includes, in step 2874, performing a conversion between the current block and the bitstream representation of the video based on one or more parameters of the encoding / decoding mode using optical flow.

[0715] Figure 28H A flowchart of an exemplary method for video processing is shown. Method 2880 includes, in step 2882, performing a cropping operation on the correction terms of the final predicted output of the optical flow encoding / decoding mode for a current block of video encoded / decoded using an optical flow encoding / decoding mode.

[0716] Method 2880 includes, in step 2884, performing a conversion between the current block and the bitstream representation of the video based on the final prediction output.

[0717] Figure 28I A flowchart of an exemplary method for video processing is shown. The method 2890 includes, in step 2892, filtering each color component of the current block using a single type of interpolation filter upon determining that an optical flow-based encoding / decoding mode has been enabled for the current block of the video.

[0718] The method 2890 includes, in step 2894, performing a filling operation when it is determined that at least one sample point of the current block is outside a predetermined range.

[0719] The method 2890 includes, in step 2896, converting between the current block and the bitstream representation of the video based on the filtering operation and the padding operation.

[0720] In some embodiments, the following technical solutions may be implemented:

[0721] A1. A video processing method, comprising: determining a conversion between a current block of video and a bitstream representation of video using a first linear optimization model for conversion using a first encoding / decoding mode, wherein the first linear optimization model is derived from a second linear optimization model for conversion using a second encoding / decoding mode; and performing the conversion based on the determination.

[0722] A2. According to the method of scheme A1, the second encoding / decoding mode is a bidirectional optical flow (BDOF) prediction mode.

[0723] A3. According to the method of scheme A2, the second linear optimization model includes N sets of sample points (u) as input. k ,v k ,w k The second linear optimization model minimizes or approximately minimizes two output parameters a and b.

[0724]

[0725] A4. According to the method of scheme A3, where u k =(I (0) (x,y)-I (1) (x,y)), a = v x And b = v y , where I (i) (x,y) represents the brightness sample at pixel position (x,y) in reference frame i, and where and These represent the horizontal and vertical gradients of the brightness sample points, respectively.

[0726] A5. According to the method of scheme A3, where u k =-C(n), v k =L(n), w k =1, a = α and b = β, where C(n) represents the reconstructed chromaticity sample points of the top and left neighbors, where L(n) represents the reconstructed luminance sample points of the top and left neighbors, and where α and β represent the parameters of the second linear optimization model.

[0727] A6. According to the method of scheme A1, the second encoding / decoding mode is a cross-component linear model prediction mode.

[0728] A7. A video processing method comprising: enabling a first prediction mode or a second prediction mode different from the first prediction mode based on one or more picture sequence count (POC) parameters associated with a picture of a current block of video, wherein the first prediction mode is a codec mode using optical flow; and performing a conversion between the current block and a bitstream representation of the video based on the first mode or the second mode.

[0729] A8. The method according to scheme A7 further includes: avoiding the use of optical flow encoding / decoding modes, wherein the first reference image (R0) and the second reference image (R1) are associated with the current image including the current block, wherein τ0 is the difference between the image sequence count of the current image and the image sequence count of the first reference image (τ0 = POC(current) - POC(R0)), and wherein τ1 is the difference between the image sequence count of the second reference image and the image sequence count of the current image (τ1 = POC(R1) - POC(current)).

[0730] A9. According to the method of scheme A8, where abs(τ0)≥T0 or abs(τ1)≥T1, where T0 and T1 are integers.

[0731] A10. According to the method of scheme A8, where abs(τ0)≥T0 and abs(τ1)≥T1, where T0 and T1 are integers.

[0732] A11. According to the method of scheme A9 or A10, where T0 = T1 = 4.

[0733] A12. According to the method of scheme A8, where abs(τ0)+abs(τ1)≥T0, where T0 is an integer.

[0734] A13. According to the method of scheme A12, where T0 = 8.

[0735] A14. According to the method of scheme A8, where abs(abs(τ0)-abs(τ1))≥T0, where T0 is an integer.

[0736] A15. According to the method of scheme A14, where T0 = 0.

[0737] A16. According to the method of scheme A8, where abs(τ0)≥abs(τ1)×T0 and abs(τ1)≥abs(τ0)×T0, where T0 is an integer.

[0738] A17. According to the method of scheme A16, where T0 = 8.

[0739] A18. The method according to any one of schemes A9 to A17, wherein T0 is notified by signaling in the video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), strip header, slice group header, slice, codec unit (CU), or codec tree unit (CTU).

[0740] A19. According to the method of scheme A7, wherein an encoding / decoding mode using optical flow is enabled, wherein one or more velocity vectors (v) of the current image of the current block are included. x ,v y Based on one or more POC distances, wherein a first reference image (R0) and a second reference image (R1) are associated with the current image.

[0741] A20. According to the method of scheme A19, where τ0 is the difference between the image sequence count of the current image and the image sequence count of the first reference image (τ0 = POC(current) - POC(R0)), and τ1 is the difference between the image sequence count of the second reference image and the image sequence count of the current image (τ1 = POC(R1) - POC(current)), where...

[0742]

[0743] in It is the velocity vector of the first reference image, and It is the velocity vector of the second reference image.

[0744] A21. According to the method of scheme A19, one or more velocity vectors are derived iteratively.

[0745] A22. According to the method of scheme A19, wherein one or more velocity vectors are based on a third prediction mode, and wherein the third prediction mode is DMVR or bilateral matching.

[0746] A23. According to the method of scheme A22, MV0 and MV1 are motion vectors for the first reference image and the second reference image, respectively, and MV0' and MV1' are initial motion vectors for the third prediction mode.

[0747] A24. According to the method in scheme A23, where MV0' = MV0 + (v x ,v y And MV1' = MV1 - (v x ,v y ).

[0748] A25. According to the method of scheme A23, where MV0' = MV0 - (v x ,v y And MV1' =

[0749] MV1+(v x ,v y ).

[0750] A26. According to the method of scheme A7, wherein the encoding / decoding mode using optical flow is enabled and applied to a subset of samples of the current block, wherein the subset of samples does not include all samples of the current block.

[0751] A27. According to the method of scheme A26, the subset of samples does not include samples in the first row, last row, first column, or last column of the current block.

[0752] A28. According to the method of scheme A7, wherein the encoding and decoding mode using optical flow is enabled and applied to a subset of samples of the sub-blocks of the current block, wherein the subset of samples is not equivalent to all samples of the sub-blocks of the current block.

[0753] A29. According to the method of scheme A28, wherein the subset of samples does not include samples in the first row, last row, first column or last column of the sub-block of the current block.

[0754] A30. The method according to any one of schemes A1-A29, wherein the encoding / decoding mode using optical flow is a bidirectional optical flow (BDOF) prediction mode.

[0755] A31. A method for video processing, comprising:

[0756] Based on the encoding and decoding information associated with the current block of the video, one or more velocity vectors (v) associated with the reference image of the current block are continuously derived. x ,v y );as well as

[0757] Based on one or more velocity vectors, a conversion is performed between the current block and the bitstream representation of the video.

[0758] The encoding / decoding information includes the value of the horizontal component of the motion vector of the current block, the value of the vertical component of the motion vector of the current block, or the size of the current block.

[0759] A32. Following the method in scheme A31, where v is first derived... y And based on v y Derivation of the value of v x .

[0760] A33. The method according to scheme A31 also includes:

[0761] Based on one or more velocity vectors, refine at least one prediction or at least one gradient of the reference image; and

[0762] After refinement, one or more velocity vectors are derived based on the reference image.

[0763] A34. According to the method of scheme A33, refinement and derivation are performed multiple times until the difference between consecutive values ​​of one or more velocity vectors is less than a predetermined threshold.

[0764] A35. The method according to any one of schemes A1 to A34, wherein the transformation generates the current block from the bitstream representation.

[0765] A36. The method according to any one of schemes A1 to A34, wherein the transformation generates a bitstream representation from the current block.

[0766] A37. An apparatus in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to implement a method according to one or more of schemes A1 to A36.

[0767] A38. A computer program product stored on a non-transitory computer-readable medium, the computer program product comprising program code for performing one or more methods according to schemes A1 to A36.

[0768] In some embodiments, the following technical solutions may be implemented:

[0769] B1. A video processing method comprising: upon determining that an optical flow-based encoding / decoding mode has been enabled for a current block of video, performing a filtering operation on each color component of the current block using a single type of interpolation filter; and based on the filtering operation, performing a conversion between the current block and a bitstream representation of the video.

[0770] B2. According to the method of scheme B1, wherein the color component includes a luminance component, and the single-type interpolation filter includes an 8-tap interpolation filter.

[0771] B3. According to the method of scheme B1, the first number of samples of the current block used for filtering operation is less than the second number of samples required by a single type of interpolation filter, thereby reducing the memory bandwidth of the filtering operation.

[0772] B4. According to the method of scheme B3, when the encoding / decoding mode using optical flow is not applied, the second number of samples is equal to the number of samples of the current block used for filtering operations.

[0773] B5. The method according to scheme B1, wherein multiple samples of the current block are used for filtering operations, and wherein the method further includes performing a padding operation when determining the encoding / decoding mode of the optical flow used by the application.

[0774] B6. According to the method of scheme B1, wherein the size of the current block is M×N, wherein the first number of samples required for gradient calculation is (M+G)×(N+G), wherein the single-type interpolation filter includes L taps, wherein the second number of samples required for filtering operation is (M+G+L-1)×(N+G+L-1), wherein the third number of samples used for filtering operation is (M+L-1+k)×(N+L-1+k), wherein M, N, G and L are positive integers, wherein k is an integer less than G, and wherein a fourth number of samples, including the difference between the second number of samples and the third number of samples, is filled.

[0775] B7. According to the method of scheme B6, where M = N = 8, L = 8 and G = 2.

[0776] B8. According to the method of scheme B6 or B7, where k = 0 or k = 1.

[0777] B9. According to the method of scheme B1, where the coordinates of the top-left pixel of the current block are (0,0), and the method further includes:

[0778] Avoid calculating gradient values ​​and apply the optical flow encoding / decoding mode to pixels at predetermined locations.

[0779] B10. According to the method of scheme B9, the x-coordinate of the predetermined position is -1 or W, where W is the width of the current block, and where W is a positive integer.

[0780] B11. According to the method of scheme B9, the y-coordinate of the predetermined position is -1 or H, where H is the height of the current block, and where H is a positive integer.

[0781] B12. According to the method of scheme B1, the coordinates of the top left pixel of the current block are (0,0), and the method further includes: modifying the calculation of gradient values ​​for pixels at predetermined positions.

[0782] B13. According to the method of scheme B12, the x-coordinate of the predetermined position is -1 or W, where W is the width of the current block, and where W is a positive integer.

[0783] B14. According to the method of scheme B12, the y-coordinate of the predetermined position is -1 or H, where H is the height of the current block, and where H is a positive integer.

[0784] B15. According to the method of scheme B13 or B14, the calculation of the gradient value includes:

[0785]

[0786] Where W is the width of the current block, predSampleL0 is an array of brightness prediction samples of the current block in the reference list L0, gradientHL0 is the horizontal component of the gradient value derived from the reference list L0, and hx and vy are integer coordinate indices.

[0787] B16. According to the method of scheme B13 or B14, the calculation of the gradient value includes:

[0788]

[0789] Where W is the width of the current block, predSampleL0 is an array containing brightness prediction samples from the current block, gradientVL0 is the vertical component of the gradient values ​​derived from the reference list L0, and hx and vy are integer coordinate indices.

[0790] B17. The method according to scheme B1 further includes: filling one or more external samples of the current block before calculating the gradient value, instead of interpolating one or more external samples.

[0791] B18. According to the method of scheme B1, one or more gradient calculations used in the encoding / decoding mode using optical flow are the same as those used in the adaptive loop filter (ALF).

[0792] B19. A video processing method comprising: filtering each color component of the current block using a single-type interpolation filter when it is determined that an optical flow-based encoding / decoding mode has been enabled for the current block of the video; performing a padding operation when it is determined that at least one sample of the current block is outside a predetermined range; and converting between the current block and a bitstream representation of the video based on the filtering and padding operations.

[0793] B20. According to the method of scheme B19, a padding operation is performed before gradient calculation.

[0794] B21. According to the method of scheme B19, the predetermined range is based on the height or width of the current block.

[0795] B22. The method according to any one of schemes B1 to B21, wherein the encoding / decoding mode using optical flow includes a bidirectional optical flow (BDOF) prediction mode.

[0796] B23. The method according to any one of schemes B1 to B22, wherein the transformation generates the current block from the bitstream representation.

[0797] B24. The method according to any one of schemes B1 to B22, wherein the transformation generates a bitstream representation from the current block.

[0798] B25. An apparatus in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to implement one or more of the methods described in schemes B1 to B24.

[0799] B26. A computer program product stored on a non-transitory computer-readable medium, the computer program product comprising program code for performing one or more methods such as schemes B1 to B24.

[0800] In some embodiments, the following technical solutions may be implemented:

[0801] C1. A video processing method, comprising: for a conversion between a current block of video and a bitstream representation of video, determining a gradient value calculation algorithm using an optical flow tool; and based on the determination, performing the conversion, wherein the gradient value calculation algorithm differs from conventional gradient value calculation algorithms, including:

[0802] gradientHL0[x][y]=(predSampleL0[hx+1][vy]–predSampleL0[hx-1][vy])>>4,

[0803] gradientVL0[x][y]=(predSampleL0[hx][vy+1]–predSampleL0[hx][vy–1])>>4,

[0804] gradientHL1[x][y]=(predSampleL1[hx+1][vy]–predSampleL1[hx–1][vy])>>4, and

[0805] gradientVL1[x][y]=(predSampleL1[hx][vy+1]–predSampleL1[hx][vy-1])>>4,

[0806] Where gradientHL0 is the horizontal component of the gradient value derived from reference list L0, gradientVL0 is the vertical component of the gradient value derived from reference list L0, gradientHL1 is the horizontal component of the gradient value derived from reference list L1, gradientVL1 is the vertical component of the gradient value derived from reference list L1, predSampleL0 is an array including the brightness prediction samples of the current block in reference list L0, predSampleL1 is an array including the brightness prediction samples of the current block in reference list L1, and hx and vy are integer coordinate indices.

[0807] C2. According to the method of scheme C1, wherein modifying the gradient value calculation includes: shifting the gradient value by a predetermined positive integer (S), and wherein S≠4.

[0808] C3. According to the method of scheme C2, where S = 6.

[0809] C4. According to the method of scheme C2, where S = BP, where B is the bit depth of the sample points of the current block, and where P is a positive integer.

[0810] C5. According to the method of scheme C4, where P = 6, and where B = 8 or 12.

[0811] C6. According to the method of scheme C1, the modification of gradient value calculation includes: pruning the gradient value so that the gradient value can be represented as a K-bit integer, where K is a positive integer.

[0812] C7. According to the method of scheme C6, where K = 8 or 16.

[0813] C8. According to the method of scheme C1, the gradient value calculation algorithm of the optical flow tool includes the calculation of a first internal variable (temp), a second internal variable (tempX), and a third internal variable (tempY), wherein the calculation is defined as:

[0814] temp[x][y]=SignShift(predSampleL0[hx][vy]–predSampleL1[hx][vy],S1),

[0815] tempX[x][y]=SignShift(gradientHL0[x][y]+gradientHL1[x][y],S2), and

[0816] tempY[x][y]=SignShift(gradientVL0[x][y]+gradientVL1[x][y],S3),

[0817] Where S1, S2, and S3 are integers, and SignShift(x,s) is defined as:

[0818]

[0819] Where off is an integer.

[0820] C9. According to the method of scheme C8, where S1 = 6 and S2 = S3 = 3.

[0821] C10. According to the method of scheme C8, S1, S2 and S3 are based on the sample bit depth of the current block (B).

[0822] C11. According to the method of scheme C10, where S1 = B - P1, S2 = B - P2 and S3 = B - P3, where P1, P2 and P3 are integers.

[0823] C12. According to the method of scheme C11, where B = 8, 10 or 12, and where P1 = 4 and P2 = P3 = 7.

[0824] C13. According to the method of scheme C1, the gradient value calculation algorithm of the optical flow tool includes: calculating the first internal variable (temp), the second internal variable (tempX), and the third internal variable (tempY) represented by K1-bit integers, K2-bit integers, and K3-bit integers, respectively, where K1, K2, and K3 are positive integers.

[0825] C14. According to the method of scheme C13, where K1 = 8 or 16, K2 = 8 or 16 and K3 = 8 or 16.

[0826] C15. According to the method of scheme C13 or C14, where, after calculating temp, tempX, and tempY, the following clipping operation is performed:

[0827] temp[x][y] = Clip3(-2) K1-1 ,2 K1-1 -1,gradientHL0[x][y]),

[0828] tempX[x][y] = Clip3(-2) K2-1 ,2 K2-1 -1, gradientVL0[x][y]), and

[0829] tempY[x][y]=Clip3(-2 K3-1 ,2 K3-1 -1,gradientHL1[x][y]),

[0830] Clip3(x,min,max) is defined as follows:

[0831]

[0832] C16. According to the method of Scheme 1, the gradient value calculation algorithm of the optical flow tool includes: calculation of multiple internal variables, the multiple internal variables including sGx2, sGy2, sGxGy, sGxdI and sGydI represented by K1-bit integers, K2-bit integers, K3-bit integers, K4-bit integers and K5-bit integers respectively, and wherein K1, K2, K3, K4 and K5 are positive integers.

[0833] C17. According to the method of scheme C16, where K1 = 8 or 16, K2 = 8 or 16, K3 = 8 or 16, K4 = 8 or 16 and K5 = 8 or 16.

[0834] C18. According to the methods of schemes C16 and C17, after calculating sGx2, sGy2, sGxGy, sGxdI, and sGydI, perform the shift operation as defined below:

[0835] sGx2 = Shift(sGx2, S1),

[0836] sGy2 = Shift(sGy2, S2),

[0837] sGxGy=SignShift(sGxGy,S3),

[0838] sGxdI=SignShift(sGxdI,S4), and

[0839] sGydI=SignShift(sGydI,S5),

[0840] Where S1, S2, S3, S4, and S5 are positive integers.

[0841] Where Shift(x,s) = (x+off) >> s, and SignShift(x,s) is defined as:

[0842]

[0843] Where off is an integer.

[0844] C19. According to the method of scheme C18, where S1, S2, S3, S4 and S5 are equal to 4 or 5.

[0845] C20. According to the method of scheme C18, S1, S2, S3, S4 and S5 are based on the sample bit depth of the current block (B).

[0846] C21. According to the method of scheme C8, where S1 = B - P1, S2 = B - P2, S3 = B - P3, S4 = B - P4 and S5 = B - P5, where P1, P2, P3, P4 and P5 are integers.

[0847] C22. According to the method of scheme C21, where B = 8, 10 or 12.

[0848] C23. According to the methods in schemes C16 and C17, after calculating sGx2, sGy2, sGxGy, sGxdI, and sGydI, perform the clipping operation as defined below:

[0849] sGx2 = Clip3(0,2) K1 -1,sGx2),

[0850] sGy2 = Clip3(0,2) K2 -1,sGy2),

[0851] sGxGy=Clip3(-2 K3-1 ,2K3-1 -1,sGxGy),

[0852] sGxdI=Clip3(-2 K4-1 ,2 K4-1 -1,sGxdI), and

[0853] sGydI=Clip3(-2 K5-1 ,2 K5-1 -1,sGydI),

[0854] Clip3(x,min,max) is defined as follows:

[0855]

[0856] C24. The method according to any one of schemes C1 to C23, wherein the optical flow tool includes a bidirectional optical flow (BDOF) tool.

[0857] C25. A video processing method comprising: calculating one or more sum of absolute differences (SAD) of sub-blocks of a current block of video; making a decision on selectively enabling a bidirectional optical flow (BDOF) prediction mode for the current block; and, based on the decision, performing a conversion between the current block and a bitstream representation of the current block.

[0858] C26. According to the method of scheme C25, the SAD calculation includes the calculation of the sum of absolute differences after removing the mean (MR-SAD).

[0859] C27. According to the method of scheme C25 or C26, wherein the SAD calculation is performed on sample points at predetermined positions in the current block.

[0860] C28. According to the method of scheme C25 or C26, wherein the SAD calculation is performed on sample points at predetermined positions in sub-blocks of the current block.

[0861] C29. A video processing method comprising: selectively enabling a general bidirectional prediction improvement (GBi) process based on a current block of video, deriving one or more parameters of a coding / decoding mode using optical flow for the current block; and performing a conversion between the current block and a bitstream representation of the video based on one or more parameters of the coding / decoding mode using optical flow.

[0862] C30. A video processing method, comprising: for a current block of a video encoded and decoded using an optical flow encoding and decoding mode, performing a cropping operation on a correction term of a final prediction output of the optical flow encoding and decoding mode; and performing a conversion between the current block and a bitstream representation of the video based on the final prediction output.

[0863] C31. According to the method of scheme C30, the correction term is clipped to the range [minCorrection, maxCorrection], where minCorrection and maxCorrection are integers.

[0864] C32. According to the method of scheme C31, where minCorrection = -32 and maxCorrection = 32.

[0865] C33. The method according to scheme C31, where minCorrection and maxCorrection are based on the sample bit depth.

[0866] C34. According to the method of scheme C31, where minCorrection and maxCorrection are based on the height or width of the current block of the video.

[0867] C35. According to the method of scheme C31, minCorrection and maxCorrection are notified by signaling in the video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), strip header, slice group header, slice, codec unit (CU) or codec tree unit (CTU).

[0868] C36. According to the method of scheme C30, the final predicted output is clipped to the range [minPred, maxPred], where minPred and maxPred are integers.

[0869] C37. According to the method of scheme C36, where minPred = -32 and maxPred = 32.

[0870] C38. According to the method of scheme C36, where midPred and maxPred are based on the sample bit depth.

[0871] C39. According to the method of scheme C36, where midPred and maxPred are based on the height or width of the current block of the video.

[0872] C40. According to the method of scheme C36, midPred and maxPred are notified by signaling in the video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), strip header, slice group header, slice, codec unit (CU) or codec tree unit (CTU).

[0873] C41. The method according to any one of schemes C30 to C40, wherein the correction term includes a predicted offset derived from samples based on the encoding / decoding mode using optical flow.

[0874] C42. The method according to any one of schemes C25 to C41, wherein the encoding / decoding mode using optical flow includes a bidirectional optical flow (BDOF) prediction mode.

[0875] C43. The method according to any one of schemes C1 to C42, wherein the transformation generates the current block from the bitstream representation.

[0876] C44. The method according to any one of schemes C1 to C42, wherein the transformation generates a bitstream representation from the current block.

[0877] C45. An apparatus in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to implement a method according to one or more of schemes C1 to C44.

[0878] C46. A computer program product stored on a non-transitory computer-readable medium, the computer program product comprising program code for performing one or more methods according to schemes C1 to C44.

[0879] 6. Example implementation of the disclosed technology

[0880] Figure 29 This is a block diagram of a video processing apparatus 2900. Apparatus 2900 can be used to implement one or more methods described herein. Apparatus 2900 can be embodied in a smartphone, tablet, computer, Internet of Things (IoT) receiver, etc. Apparatus 2900 may include one or more processors 2902, one or more memories 3704, and video processing hardware 2906. Processor 2902 can be configured to implement one or more methods described herein (including, but not limited to, method 2800). Although some embodiments may operate without memory, memory(s) 2904 can be used to store data and code for implementing the methods and techniques described herein. Video processing hardware 2906 can be used to implement some of the techniques described in this document in hardware circuitry.

[0881] In some embodiments, the video encoding / decoding method may use methods such as those described above. Figure 29 The device described is implemented on a hardware platform.

[0882] Some embodiments of the disclosed technology involve making a decision or determination to enable a video processing tool or mode. In one example, when a video processing tool or mode is enabled, the encoder will use or implement that tool or mode in the processing of video blocks, but will not necessarily modify the resulting bitstream based on the use of that tool or mode. That is, the conversion from video blocks to a bitstream representation of video will be performed using the video processing tool or mode when it is enabled based on the decision or determination. In another example, when a video processing tool or mode is enabled, the decoder will process the bitstream based on the fact that the video processing tool or mode already knows that the bitstream has been modified. That is, the conversion from the bitstream representation of video to video blocks will be performed using the video processing tool or mode enabled based on the decision or determination.

[0883] Some embodiments of the disclosed technology involve making a decision or determination to disable a video processing tool or mode. In one example, when a video processing tool or mode is disabled, the encoder will not use that tool or mode in the conversion from video block to bitstream representation of video. In another example, when a video processing tool or mode is disabled, the decoder will process the bitstream knowing that the bitstream has not been modified using a video processing tool or mode enabled based on the decision or determination.

[0884] Figure 30 This is a block diagram illustrating an example video processing system 3000 in which various techniques disclosed herein may be implemented. Various implementations may include some or all of the components of system 3000. System 3000 may include an input 3002 for receiving video content. The video content may be received in a raw or uncompressed format (e.g., 8 or 10-bit multi-component pixel values), or in a compressed or encoded format. Input 3002 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces (such as Ethernet, Passive Optical Networking (PON), etc.) and wireless interfaces (such as Wi-Fi or cellular interfaces).

[0885] System 3000 may include an encoding / decoding component 3004 capable of implementing the various encoding / decoding or coding methods described herein. Encoding / decoding component 3004 can reduce the average bit rate of the video from input 3002 to output to produce an encoded / decoded representation of the video. Therefore, encoding / decoding techniques are sometimes referred to as video compression or video transcoding techniques. As indicated by component 3006, the output of encoding / decoding component 3004 can be stored or transmitted via a connected communication. The stored or transmitted bitstream (or encoded / decoded) representation of the video received at input 3002 can be used by component 3008 to generate pixel values ​​or displayable video that is sent to display interface 3010. The process of generating user-visible video from the bitstream representation is sometimes referred to as video decompression. Furthermore, although some video processing operations are referred to as “encoding / decoding” operations or tools, it should be understood that encoding / decoding tools or operations are used at the encoder, and the corresponding decoding tools or operations that reverse the encoding / decoding results will be performed by the decoder.

[0886] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), or DisplayPort. Examples of storage interfaces include SATA (Serial Advanced Technology Accessory), PCI, IDE, etc. The technologies described herein can be embodied in a variety of electronic devices, such as mobile phones, laptops, smartphones, or other devices capable of performing digital data processing and / or video display.

[0887] As can be understood from the foregoing, specific embodiments of the disclosed technology have been described herein for illustrative purposes, but various modifications can be made without departing from the scope of the invention. Therefore, the disclosed technology is not limited except for the appended claims.

[0888] The implementation and functional operation of the subject matter described in this patent document can be implemented in various systems, digital electronic circuits, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations thereof. The subject matter described in this specification can be implemented as one or more computer program products, i.e., one or more transient and non-transitory computer program instruction modules encoded on a computer-readable medium for operation by a data processing apparatus or for controlling the operation of a data processing apparatus. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of material affecting machine-readable propagation signals, or a combination thereof. The terms "data processing unit" and "data processing apparatus" include all means, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the apparatus may include code that creates an operating environment for the computer program in question, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination thereof.

[0889] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any programming language (including compiled or interpreted languages) and can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinating files (e.g., a file storing one or more modules, subroutines, or portions of code). Computer programs can be deployed to execute on one or more computers located at a single site or distributed across multiple sites interconnected by a communication network.

[0890] The processing and logic flows described in this specification can be executed by one or more programmable processors that run one or more computer programs to perform functions by manipulating input data and generating output. The processing and logic flows can also be executed by dedicated logic circuitry, and the device can be implemented as dedicated logic circuitry, such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit).

[0891] For example, processors suitable for running computer programs include general-purpose and special-purpose microprocessors, as well as any one or more processors in any type of digital computer. Typically, a processor receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are a processor that executes instructions and one or more storage devices that store those instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or operatively coupled to one or more mass storage devices to receive data from or transfer data to, or both. However, a computer does not necessarily need to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices. Processors and memory may be supplemented by or incorporated into special-purpose logic circuitry.

[0892] The purpose is to combine the instruction manual with the accompanying Figure 1 The above is to be considered illustrative only, where illustrative means example. As used herein, unless the context clearly indicates otherwise, the use of "or" is intended to include "and / or".

[0893] While this patent document contains numerous details, it should not be construed as limiting the scope of any invention or claim, but rather as a description of features specific to particular embodiments of a particular invention. Certain features described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various functions described in the context of a single embodiment may also be implemented individually in multiple embodiments, or in any suitable sub-combination. Furthermore, while features may be described as functioning in certain combinations, or even initially claimed in this way, in some cases one or more features may be removed from the claimed combination, and the claimed combination may refer to a sub-combination or a variation of a sub-combination.

[0894] Similarly, although operations are described in a specific order in the accompanying drawings, this should not be construed as meaning that these operations must be performed in the specific order or sequence shown, or that all of the operations shown must be performed, in order to obtain the desired result. Furthermore, the separation of various system components in the embodiments described in this patent document should not be construed as requiring such separation in all embodiments.

[0895] Only some implementation methods and examples have been described. Other implementation methods, enhancements and variations can be made based on the content described and illustrated in this patent document.

Claims

1. A method for encoding and decoding video data, comprising: Based on one or more picture order count (POC) parameters associated with the pictures of the current video block of the video, it is determined whether a first prediction mode can be applied, wherein the prediction samples of the current video block are refined based at least on gradients in different directions. Based on the determination, perform the conversion between the current video block and the bitstream of the video; Wherein, the first reference image (R0) and the second reference image (R1) are associated with the current image including the current video block, where τ0 is the difference between the image sequence count of the current image and the image sequence count of the first reference image (τ0 = POC(current) - POC(R0)), and where τ1 is the difference between the image sequence count of the second reference image and the image sequence count of the current image (τ1 = POC(R1) - POC(current)). In response to abs(abs(τ0)-abs(τ1))≥T0, the first prediction mode is avoided, where T0 is an integer and T0 is greater than 0, or In response to abs(τ0)>abs(τ1)×T0 or abs(τ1)>abs(τ0)×T0, the first prediction mode is avoided, where T0 is an integer and T0=1.

2. The method according to claim 1, wherein, Signaling notification T0 is provided in the Video Parameter Set (VPS), Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Strip header, Slice header, Slice, Codec Unit (CU), or Codec Tree Unit (CTU).

3. The method according to claim 1, wherein, The first prediction mode is the bidirectional optical flow (BDOF) prediction mode.

4. The method according to claim 1, wherein, The conversion includes decoding the current video block from the bitstream.

5. The method according to claim 1, wherein, The conversion includes encoding the current video block into the bitstream.

6. The method according to claim 1, comprising: A first linear optimization model is determined to be used for the conversion between the current block of the video and the bitstream of the video, for conversion using a first codec mode, wherein the first linear optimization model is derived from a second linear optimization model used for conversion using a second codec mode; and Based on the determination, the conversion is performed.

7. The method according to claim 6, wherein, The second encoding / decoding mode is the bidirectional optical flow (BDOF) prediction mode.

8. The method according to claim 7, wherein, The second linear optimization model includes N sets of samples (u) as input. k ,v k ,w k The second linear optimization model minimizes or approximately minimizes two output parameters a and b.

9. The method according to claim 8, wherein u k =(I (0) (x,y)-I (1) (x,y)), a = v x And b = v y , where I (i) (x,y) represents the brightness sample at pixel position (x,y) in reference frame i, and where and These represent the horizontal and vertical gradients of the brightness sample points, respectively.

10. The method according to claim 8, wherein, u k =-C(n), v k =L(n), w k =1, a = α and b = β, where C(n) represents the reconstructed chromaticity sample points of the top and left neighbors, where L(n) represents the reconstructed luminance sample points of the top and left neighbors, and where α and β represent the parameters of the second linear optimization model.

11. The method according to claim 6, wherein, The second encoding / decoding mode is a cross-component linear model prediction mode.

12. The method according to claim 1, comprising: Based on one or more Picture Order Count (POC) parameters associated with the current block of the video, enable a first prediction mode or a second prediction mode different from the first prediction mode, wherein the first prediction mode is a codec mode using optical flow; and Based on the first prediction mode or the second prediction mode, perform the conversion between the current block and the bitstream of the video.

13. The method of claim 12, further comprising: Avoid enabling the aforementioned codec mode that uses optical flow. Wherein, the first reference image (R0) and the second reference image (R1) are associated with the current image including the current block, where τ0 is the difference between the image sequence count of the current image and the image sequence count of the first reference image (τ0 = POC(current) - POC(R0)), and τ1 is the difference between the image sequence count of the second reference image and the image sequence count of the current image (τ1 = POC(R1) - POC(current)).

14. The method according to claim 13, wherein, abs(τ0)≥T0 or abs(τ1)≥T1, where T0 and T1 are integers.

15. The method according to claim 13, wherein, abs(τ0)≥T0 and abs(τ1)≥T1, where T0 and T1 are integers.

16. The method according to claim 14 or 15, wherein, T0 = ​​T1 = 4.

17. The method according to claim 13, wherein, abs(τ0)+abs(τ1)≥T0, where T0 is an integer.

18. The method according to claim 17, wherein, T0＝8。 19. The method according to claim 13, wherein, abs(abs(τ0)-abs(τ1))≥T0, where T0 is an integer.

20. The method according to claim 19, wherein, T0＝0。 21. The method according to claim 13, wherein, abs(τ0)≥abs(τ1)×T0 and abs(τ1)≥abs(τ0)×T0, where T0 is an integer.

22. The method according to claim 21, wherein, T0＝8。 23. The method according to any one of claims 14-15 and 17 to 22, wherein, The T0 is notified by signaling in the Video Parameter Set (VPS), Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Strip header, Slice header, Slice, Codec Unit (CU), or Codec Tree Unit (CTU).

24. The method according to claim 12, wherein, The encoding / decoding mode using optical flow is enabled, wherein one or more velocity vectors (v) of the current image, including the current block, are used. x ,v y Based on the one or more POC distances, and wherein the first reference image (R0) and the second reference image (R1) are associated with the current image.

25. The method according to claim 24, wherein, τ0 is the difference between the image sequence count of the current image and the image sequence count of the first reference image (τ0 = POC(current) - POC(R0)), where τ1 is the difference between the image sequence count of the second reference image and the image sequence count of the current image (τ1 = POC(R1) - POC(current)). in It is the velocity vector of the first reference image. It is the velocity vector of the second reference image.

26. The method according to claim 24, wherein, The one or more velocity vectors are derived iteratively.

27. The method according to claim 24, wherein, The one or more velocity vectors are based on a third prediction mode, wherein the third prediction mode is DMVR or bilateral matching.

28. The method according to claim 27, wherein, MV0 and MV1 are motion vectors for the first reference image and the second reference image, respectively, and MV0' and MV1' are initial motion vectors for the third prediction mode.

29. The method according to claim 28, wherein, MV0' = MV0 + (v x ,v y And MV1' = MV1 - (v x ,v y ).

30. The method according to claim 28, wherein, MV0' = MV0 - (v x ,v y And MV1' = MV1 + (v x ,v y ).

31. The method according to claim 12, wherein, The optical flow encoding / decoding mode is enabled and applied to a subset of samples of the current block, wherein the subset of samples does not include all samples of the current block.

32. The method according to claim 31, wherein, The subset of the samples does not include the samples in the first row, last row, first column, or last column of the current block.

33. The method according to claim 12, wherein, The optical flow encoding / decoding mode is enabled and applied to a subset of samples of the sub-blocks of the current block, wherein the subset of samples is not equivalent to all samples of the sub-blocks of the current block.

34. The method according to claim 33, wherein, The subset of samples does not include samples in the first row, last row, first column, or last column of the current block's sub-blocks.

35. The method according to any one of claims 12-15, 17-22, and 24-34, wherein, The encoding / decoding mode using optical flow is the bidirectional optical flow (BDOF) prediction mode.

36. The method according to claim 1, comprising: Based on the encoding and decoding information associated with the current block of the video, one or more velocity vectors (v) associated with a reference image of the current block are continuously derived. x ,v y );as well as Based on the one or more velocity vectors, a conversion is performed between the current block and the video bitstream. The encoding / decoding information includes the value of the horizontal component of the motion vector of the current block, the value of the vertical component of the motion vector of the current block, or the size of the current block.

37. The method of claim 36, wherein v is first derived. y And based on v y Derivation of the value of v x .

38. The method of claim 37, further comprising: Based on the one or more velocity vectors, refine at least one prediction or at least one gradient of the reference image; as well as After refinement, one or more velocity vectors are derived based on the reference image.

39. The method according to claim 38, wherein, The refinement and derivation are performed multiple times until the difference between consecutive values ​​of one or more velocity vectors is less than a predetermined threshold.

40. The method according to any one of claims 6-15, 17-22, 24-34, and 36-39, wherein, The conversion includes decoding the current block from the bitstream.

41. The method according to any one of claims 6-15, 17-22, 24-34, and 36-39, wherein, The conversion includes encoding the current block into the bit stream.

42. An apparatus in a video system, comprising a processor and a non-transitory memory having instructions thereon, wherein, When executed by the processor, the instructions cause the processor to implement the method according to any one of claims 6-41.

43. A computer program product stored on a non-transitory computer-readable medium, the computer program product comprising program code executed by a processor according to any one of claims 6-41.

44. An apparatus for encoding and decoding video data, comprising a processor and a non-transitory memory having instructions thereon, wherein, When the instruction is executed by the processor, the processor: Based on one or more picture order count (POC) parameters associated with the pictures of the current video block of the video, it is determined whether a first prediction mode can be applied, wherein the prediction samples of the current video block are refined based at least on gradients in different directions. Based on the determination, perform the conversion between the current video block and the bitstream of the video; Wherein, the first reference image (R0) and the second reference image (R1) are associated with the current image including the current video block, where τ0 is the difference between the image sequence count of the current image and the image sequence count of the first reference image (τ0 = POC(current) - POC(R0)), and where τ1 is the difference between the image sequence count of the second reference image and the image sequence count of the current image (τ1 = POC(R1) - POC(current)). In response to abs(abs(τ0)-abs(τ1))≥T0, the first prediction mode is avoided, where T0 is an integer and T0 is greater than 0, or In response to abs(τ0)>abs(τ1)×T0 or abs(τ1)>abs(τ0)×T0, the first prediction mode is avoided, where T0 is an integer and T0=1.

45. The apparatus according to claim 44, wherein, Signaling notification T0 is provided in the Video Parameter Set (VPS), Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Strip header, Slice header, Slice, Codec Unit (CU), or Codec Tree Unit (CTU).

46. ​​The apparatus according to claim 44, wherein, The first prediction mode is the bidirectional optical flow (BDOF) prediction mode.

47. The apparatus according to claim 44, wherein, The conversion includes decoding the current video block from the bitstream.

48. The apparatus according to claim 44, wherein, The conversion includes encoding the current video block into the bitstream.

49. A non-transitory computer-readable storage medium for storing instructions, said instructions causing a processor to: Based on one or more image sequence count (POC) parameters associated with the images of the current video block, it is determined whether a first prediction mode can be applied, wherein, In the first prediction mode, the prediction samples of the current video block are refined based at least on gradients in different directions; Based on the determination, perform the conversion between the current video block and the bitstream of the video; Wherein, the first reference image (R0) and the second reference image (R1) are associated with the current image including the current video block, where τ0 is the difference between the image sequence count of the current image and the image sequence count of the first reference image (τ0 = POC(current) - POC(R0)), and where τ1 is the difference between the image sequence count of the second reference image and the image sequence count of the current image (τ1 = POC(R1) - POC(current)). In response to abs(abs(τ0)-abs(τ1))≥T0, the first prediction mode is avoided, where T0 is an integer and T0 is greater than 0, or In response to abs(τ0)>abs(τ1)×T0 or abs(τ1)>abs(τ0)×T0, the first prediction mode is avoided, where T0 is an integer and T0=1.

Images