METHOD FOR ENCODING VIDEO, ELECTRONIC DEVICE, COMPUTER-READABLE STORAGE MEDIUM, AND COMPUTER PROGRAM - Patent application

Abstract

To provide a method for deriving constructed affine merge candidates, an electronic device, a program, and a storage medium.SOLUTION: A computer-implemented method for motion prediction includes: acquiring, from a decoder, a regular merge flag for a coding unit (CU) which is coded as merge mode and merge related modes; when the regular merge flag is one, indicating that regular merge mode or merge mode with motion vector differences (MMVD) is used by the CU; constructing a motion vector merge list for the CU and using regular merge index to indicate which candidate is used; and when the regular merge flag is zero, indicating regular merge mode is not used by the CU, and further receiving mode flags to indicate associated merge related modes used when a mode flag's constraints are met.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is based on provisional application Ser. No. 62 / 787,230, filed on December 31, 2018. No. 6,313,636, filed on and claims the benefit of which, the entire contents of which are incorporated herein by reference.

[0002] The present application relates to video encoding and compression. More specifically, the present application relates to a method, an electronic device, a computer-readable storage medium and a computer program product for encoding video. [Background technology]

[0003] Various video coding techniques may be used to compress the video data. The encoding is performed in accordance with one or more video coding standards. The following are the general-purpose video coding (Versatile Video Coding: VVC) and joint search test model. Dell (Joint Exploration test Model: JEM), high performance video encoding (H.265 / H igh-Efficiency Video Coding:HEVC), Advanced Video Coding (H.264 / Advanced Video Coding:AVC, Moving Picture Experts Group (MP Video coding generally involves the process of extracting the information present in a video image or sequence. Use prediction methods that exploit redundancy (e.g., inter-prediction, intra-prediction, etc.) An important objective of video coding techniques is to avoid or minimize degradation of video quality while The primary goal of video compression is to compress video data into a format that uses a lower bit rate. Summary of the Invention [Means for solving the problem]

[0004] The present disclosure provides an example of a method for improving the efficiency of syntax signaling in a merge-related mode. The present invention provides a method for: According to a first aspect of the present disclosure, a computer-implemented method for motion prediction includes: and coding units (CUs) coded as merge-related modes Obtain the normal merge flag for from the decoder, and In this case, normal merge mode or differential motion vector merge mode (Merge mode with Indicates that Motion Vector Difference (MMVD) is used by the CU. Construct a motion vector merge list for each candidate, and use the normal merger to indicate which candidate is used. If the merge mode flag is set to 0, the merge mode is Indicates that the mode is not used by the CU and the mode flag constraints are met. receiving a mode flag indicating an associated merge-related mode to be used when nothing. According to a second aspect of the present disclosure, a computing device includes one or more processors. a non-transitory memory storing instructions executable by the one or more processors; and a computer readable memory, the one or more processors configured to: and normal merge for coding units (CUs) coded as merge-related modes. The merge mode flag is obtained from the decoder. If the normal merge flag is 1, the normal merge mode is selected. Indicates that the motion vector merge mode or differential motion vector merge mode (MMVD) is used by the CU. Then, build a motion vector merge list for the CU and send a notification indicating which candidate will be used. If the normal merge index is used and the normal merge flag is 0, the normal merge mode is used. Indicates that the mode is not used by the CU and the mode flag constraints are met. configured to receive a mode flag indicating the accompanying merge-related mode to be used when can be. According to a third aspect of the present disclosure, a non-transitory computer having instructions stored thereon is provided. A readable storage medium is provided, the instructions being executable by one or more processors of the device. The instructions are then encoded in the device as merge and merge-related modes. obtaining from the decoder a normal merge flag for the corresponding coding unit (CU); If the normal merge flag is 1, the normal merge mode or the differential motion vector merge mode is selected. Indicates that the motion vector marker (MMVD) is used by the CU. Build a merge list and use the normal merge index to indicate which candidate is used If the normal merge flag is 0, the normal merge mode is used by the CU. indicates that the mode flag is not present and is used when the mode flag constraints are met. and receiving a mode flag indicating a merge-related mode.

[0005] Both the foregoing general description and the following detailed description are intended to be illustrative of the present disclosure. It should be understood that these are merely examples and are not limiting of the present disclosure. [Brief description of the drawings]

[0006] The accompanying drawings, which are incorporated in and constitute a part of this specification, are illustrative of the present disclosure. An example is provided to illustrate the principles of the present disclosure.

[0007] [Figure 1] FIG. 2 is a block diagram of an encoder according to an example of the present disclosure. [Diagram 2] FIG. 2 is a block diagram of a decoder according to an example of the present disclosure. [Diagram 3] 1 is a flowchart illustrating a method for deriving constructed affine merge candidates according to an example of the present disclosure. [Figure 4] 1 is a flowchart illustrating a method for determining whether a flag constraint is satisfied, according to an example of the present disclosure. [Figure 5A] FIG. 13 illustrates an MMVD search point according to an example of the present disclosure. [Figure 5B] FIG. 13 illustrates an MMVD search point according to an example of the present disclosure. [Figure 6A] FIG. 2 is an illustration of a control point-based affine motion model, according to an example of the present disclosure. [Figure 6B] FIG. 2 is an illustration of a control point-based affine motion model, according to an example of the present disclosure. [Figure 7] FIG. 2 illustrates an affine motion vector field (MVF) for each sub-block according to an example of the present disclosure. [Figure 8] FIG. 13 illustrates the location of an inherited affine motion predictor, according to an example of the present disclosure. [Figure 9] FIG. 13 illustrates a control point motion vector inheritance according to an example of the present disclosure. [Figure 10] FIG. 13 illustrates locations of candidate placements, according to an example of the present disclosure. [Figure 11] FIG. 1 illustrates spatially neighboring blocks used by subblock based temporal motion vector prediction (SbTMVP) according to an example of the present disclosure. [Figure 12] FIG. 1 illustrates a sub-block-based temporal motion vector prediction (SbTMVP) process according to an example of the present disclosure. [Figure 13A] FIG. 2 illustrates a triangulation according to an example of the present disclosure. [Figure 13B] FIG. 2 illustrates a triangulation according to an example of the present disclosure. [Figure 14] FIG. 1 illustrates a computing environment connected to a user interface according to an example of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Reference will now be made to details of exemplary embodiments, examples of which are illustrated in the accompanying drawings. The following description refers to the accompanying drawings, in which like reference numbers in different drawings represent Unless otherwise noted, the same or similar elements are depicted. The implementations illustrated do not represent all implementations consistent with this disclosure. Rather, those implementations are merely illustrative of the present disclosure as set forth in the appended claims. 1 is an example of an apparatus and method consistent with related aspects.

[0009] The terminology used in this disclosure is for the purpose of describing particular embodiments only. The disclosure is not intended to be limiting. In these cases, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are used interchangeably. Likewise, the plural is intended to be included. A term refers to any or all possible combinations of one or more of the associated listed items. It should also be understood that the term is intended to represent and encompasses the above-mentioned embodiments.

[0010] The terms "first," "second," "third," etc. are used herein to describe various pieces of information. may be used, but such information should not be limited by these terms. It is further understood that these terms distinguish one category of information from another. For example, without departing from the scope of this disclosure, the first information may be used in combination with the second information. Similarly, the second information may be referred to as the first information. When used in the specification, the term "if" means "when," "when," or "when" may be understood to mean "at discretion," depending on the context.

[0011] Video Encoding System Conceptually, video coding standards are similar. For example, most of them provide a way to compress video. For this reason, they use block-based processing and share similar video coding block diagrams. There is.

[0012] In this embodiment of the present disclosure, the efficiency of the syntax signaling of the merge-related mode We propose several methods to improve the performance of the system. The methods proposed can be used independently or in combination. Please note that this may apply depending on the circumstances.

[0013] FIG 1 shows a typical encoder 100. The encoder 100 receives a video input 11. 0, Motion Compensation 112, Motion Estimation 114, Intra / Inter Mode Decision 116, Block Predictor 140, adder 128, transform 130, quantization 132, prediction related information 142, and int prediction 118, picture buffer 120, inverse quantization 134, inverse transform 136, adder 126 , memory 124, in-loop filter 122, entropy coding 138 and bitstream It has a ream 144.

[0014] In an exemplary embodiment of the encoder, a video frame is divided into blocks for processing. For a given video block, prediction can be either inter prediction or intra prediction. In inter prediction, the predictor is formed based on the previously reconstructed frame. Based on the pixels of the image, the intra prediction can be formed through motion estimation and motion compensation. In the estimation, the predictor can be formed based on the reconstructed pixels in the current frame. Through a mode decision, the best predictor is selected to predict the current block. There are cases.

[0015] The prediction residual (i.e., the difference between the current block and its predictor) is sent to the transformation module. The transform coefficients are then sent to a quantization module for entropy reduction. The quantized coefficients are then multiplied by entropy to produce a compressed video bitstream. As shown in FIG. 1, block division information, motion vectors, , reference picture index, and intra prediction mode, etc. Prediction related information from the intra prediction module is also sent through the entropy coding module. The image is then processed and stored in the bitstream.

[0016] In the encoder, a decoder-related module also performs the reconstruction of pixels for prediction purposes. First, the prediction residual is reconstructed through inverse quantization and inverse transformation. Such a reconstructed prediction residual produces the unfiltered reconstructed pixels for the current block. The block predictor is then combined with the block predictor to obtain the

[0017] To improve coding efficiency and visual quality, in-loop filters are commonly used. For example, the deblocking filter is used in AVC, HEVC and now VVC. It is possible. In HEVC, Sample Adaptive Offset (SA An additional in-loop filter, called O), is specified to further improve coding efficiency. The latest VVC uses an adaptive loop filter (ALF). Yet another in-loop filter, called the There is a high possibility that it will be stolen.

[0018] FIG. 2 shows a block diagram of an exemplary decoder 200. The decoder 200 210, entropy decoding 212, inverse quantization 214, inverse transform 21 6, adder 218, intra / inter mode selection 220, intra prediction 222, memory 230, in-loop filter 228, motion compensation 224, picture buffer 226, prediction related It has an information 234 and a video output 232 .

[0019] At the decoder, the bitstream is first decoded into the quantized coefficient levels and the prediction relations. The associated information is decoded through an entropy decoding module. The quantized coefficient levels are then dequantized and dequantized to obtain the reconstructed prediction residual. The block predictor is processed through the decoded prediction information is formed through either intra prediction or motion compensation processing based on the filter The reconstructed pixel is obtained by summing the reconstructed prediction residual and the block predictor. In the case where an in-loop filter is applied, the final reconstructed image for output is derived. To do this, a filtering operation is performed on these pixels.

[0020] FIG. 3 illustrates an exemplary method for deriving constructed affine merge candidates in accordance with the present disclosure. It shows.

[0021] In step 310, the codes encoded as merge modes and merge related modes are The decoder obtains the normal merge flags for the merging unit (CU) from the decoder.

[0022] In step 312, if the normal merge flag is 1, normal merge mode or difference Indicates that the motion vector merge mode (MMVD) is used by the CU. Construct a motion vector merge list to indicate which candidate is used, and use the normal merge list to indicate which candidate is used. Use the index.

[0023] In step 314, if the normal merge flag is 0, the normal merge mode is set by the CU. indicates that the mode is not to be used when the mode flag constraints are met. The merge mode flag indicates the associated merge related mode to be used.

[0024] FIG. 4 illustrates an exemplary method for determining whether a flag constraint is satisfied in accordance with the present disclosure. This shows how.

[0025] In step 410, the coded block is obtained from the decoder. has a width and a height.

[0026] In step 412, the decoder determines the coding block width and coding block height. Check whether each is not equal to 4.

[0027] In step 414, the decoder determines whether the coding block width is not equal to 8, or Determine whether the coding block height is not equal to 4.

[0028] In step 416, the decoder determines whether the coding block width is not equal to 4, or Determine if the coding block height is not equal to 8.

[0029] In step 418, the decoder checks whether the normal merge flag is set. Determine whether or not

[0030] Generic Video Coding (VVC) At the 10th JVET Conference (April 10-20, 2018, San Diego, USA) JVET is currently developing Generic Video Coding (VVC) and VVC Test Model 1 (Vvc Test This document defines the first draft of the VVC (VTM1) encoding method. The new coding function uses binary and ternary trees to divide the coding block structure. It was decided to include quadtrees with nested multitype trees. Reference software VVC for implementing the reading method and draft VVC decoding process TM has been expanded at the JVET conference.

[0031] The picture division structure divides the input image into coding tree units (CTUs). The CTU is a quadtree with nested multitype trees. to define regions that share the same prediction mode (e.g., intra or inter) The coding units (CUs) are divided into a set of coding units (CUs) with corresponding leaf coding units (CUs). In this paper, the term "unit" defines the area of ​​an image that covers all components, and the term "block" defines the area of ​​an image that covers all components. A "block" is used to define a region that covers a particular component (e.g., luminance). , the spatial position is different when considering chroma sampling formats such as 4:2:0 There are cases.

[0032] VVC Enhanced Merge Mode In VTM3, a merge candidate list consists of five types of candidates, in order: It is constructed as follows. 1. Spatial MVP from spatially adjacent CUs 2. Temporal MVP from collocated CUs 3. History-based MVP from FIFO table 4. Pairwise average MVP 5.Zero MV

[0033] The size of the merge list is signaled in the slice header and is the maximum allowed size of the merge list. The capacity size is 6 in VTM3. For each CU code in merge mode, the best merge The index of the candidate page is truncated unary binarization (TU). The first bin of the merge index is encoded using the context. In the following context of this disclosure, the extended The merge modes mentioned above are also similar in concept to the merge modes used in HEVC. This is also called the normal merge mode.

[0034] MVD Merge Mode (MMVD) The merge mode is where the implicitly derived motion information is directly used to generate the predicted samples for the current CU. In addition to the MM mode, a motion vector difference merge mode (MMVD) is introduced into VVC. The VD flag is used to indicate whether the MMVD mode is used for the CU. It is signaled immediately after the transmission of the flag and the merge flag.

[0035] In MMVD, after a merge candidate is selected, the merge candidate is The MVD information is used to refine the model. Another piece of information is the merge candidate flag, which explicitly indicates the magnitude of the motion. The MMV includes an index indicating the direction of motion and an index indicating the motion direction. In D-mode, one of the first two candidates in the merge list is used as the MV base. The merge candidate flag is used to indicate which one is used. Signaled.

[0036] The distance index specifies the magnitude of the movement and is the default offset from the starting point. As shown in FIG. 5 (discussed below), the offset is the horizontal component of the starting MV. or vertical component. Relationship between distance index and default offset The details are shown in Table 1. [Table 1]

[0037] The direction index represents the direction of the MVD relative to the start point. The direction index is The MVD symbol may represent four directions, as shown in Table 2. The meaning of the MVD symbol is Note that the starting MV may change depending on the current picture's If the MV is non-predicted or bi-predictive with both list points pointing to the same side (i.e. ,The POC of the two references is both greater than the POC of the current picture, or The symbols in Table 2 represent the MV offsets added to the starting MV. The starting MV indicates the two MV points to different sides of the current picture. In the case of bi-predictive MV with a POC of 1, the POC of the current picture is and the POC of the other reference is smaller than the POC of the current picture) The symbols in Table 2 indicate the symbols of the MV offsets added to the 0MV components of the starting MV list. However, the symbols for list 1MV have the opposite values. [Table 2]

[0038] FIG. 5A is a diagram illustrating MMVD search points for first list (L0) references according to the present disclosure. This shows that.

[0039] FIG. 5B is a diagram illustrating MMVD search points for second list (L1) references according to the present disclosure. This shows that.

[0040] Affine motion compensation prediction In HEVC, only the translational motion model is used for motion compensation prediction. However, in the real world, zooming in / out, rotating There are many types of motion, including perspective motion, and other irregular motions. Block-based affine transformation motion compensation prediction is applied to the image shown in FIG. 6A and FIG. 6B. As described below, the affine motion field of a block is calculated by dividing the two control point motions vector (4 parameters) or three control point motion vectors (6 parameters) It is explained by the motion information.

[0041] FIG. 6A illustrates a control point-based affine motion of a four-parameter affine model according to the present disclosure. A model is shown below.

[0042] FIG. 6B illustrates the control point-based affine motion of the six-parameter affine model according to the present disclosure. A model is shown below.

[0043] For a four-parameter affine motion model, the motion of sample positions (x, y) within a block The vector is derived as shown in equation (1) below.

number

[0044] For the 6-parameter affine motion model, the motion of the sample position (x, y) within the block The vector is derived as shown in equation (2) below.

number

[0045] In the formula, (mv 0x , mv 0y ) is the motion vector of the upper left corner control point, and ( mv 1x , mv 1y ) is the motion vector of the control point in the upper right corner, and (mv 2x , mv 2y ) is the motion vector of the bottom-left corner control point.

[0046] To simplify the motion compensation prediction, a block-based affine transformation prediction is applied. To derive the motion vectors for the 4×4 luminance sub-blocks, as shown in FIG. 7 (discussed below), Thus, the motion vector of the center sample of each subblock is calculated according to the above formula, The motion compensation interpolation filter then rounds the derived motion vectors to a precision of 1 / 16. is applied to derive the prediction of each subblock using The size is also set to 4 × 4. The MV of a 4 × 4 chroma sub-block is the four corresponding It is calculated as the average of the MVs of the 4x4 luma sub-blocks.

[0047] FIG. 7 illustrates a subblock-by-subblock affine motion vector field (MVF) according to the present disclosure. ) is shown.

[0048] Two affine motion inter prediction modes, as is done for translational motion inter prediction, are There are two modes: affine merge mode and affine AMVP mode.

[0049] Affine Merge Prediction The AF_MERGE mode may be applied for CUs where both width and height are 8 or greater. In this mode, the CPMV of the current CU is generated based on the motion information of spatially neighboring CUs. There may be up to five CPMVP candidates, and the index is the current CU The following three types of CPVM candidates are signaled to indicate the one to be used: are used to form the affine merge candidate list. 6. Inherited affine merge candidates estimated from the CPMV of adjacent CUs 7. Constructed affine merge candidate CPMV derived using translational MVs of adjacent CUs P 8.Zero MV

[0050] In VTM3, there are at most two inherited affine candidates, which are the left-neighboring CU and the top-neighboring It is derived from the affine motion model of the neighboring blocks from the CU. For a candidate block, This is shown in Figure 8 (discussed below). For the left predictor, the scan order is A0->A1, For the top predictor, the scan order is B0->B1->B2. The first inherited Only candidates that are inherited are selected. No pruning check is performed between two inherited candidates. When an adjacent affine CU is identified, its control point motion vectors are This is used to derive the CPMVP candidates in the affine merge list of If the neighboring lower-left block A is coded in affine mode, as described below, The motion vectors v2, v3, and v4 of the upper left, upper right, and lower left corners of the CU containing block A are 4 is obtained. When block A is coded with the 4-parameter affine model, the current C The two CPMVs of U are calculated according to v2 and v3. Block A has 6 parameters In the case of coding with the affine model, the three CPMVs of the current CU are v2, v3, and and v4.

[0051] FIG. 8 illustrates the location of the inherited affine motion predictor according to this disclosure.

[0052] FIG. 9 illustrates control point motion vector inheritance according to this disclosure. The constructed affine candidates are constructed by combining the adjacent translational motion information of each control point. This means that candidates are constructed. The motion information of the control points is shown in Figure 10 (see below). The CPMV is derived from the explicit spatial and temporal contiguity, as described below. k ( k = 1, 2, 3, 4) represents the kth control point. In the case of CPMV1, B2->B 3->A2 block is checked and the MV of the first available block is used. In the case of PMV2, the B1->B0 block is checked, and in the case of CPMV3, the A1->A 0 block is checked. For TMVP, use CPMV4 if available. will be done.

[0053] FIG. 10 shows the locations of candidate configurations for affine merge modes constructed in accordance with the present disclosure. do.

[0054] After the MVs of the four control points are obtained, the affine merge candidates are merged with the corresponding motion information. The following combinations of control points MV are used in order for the construction: It is used. {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4}, {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, {CPMV1, CPMV3}

[0055] The combination of three CPMVs constructs a six-parameter affine merge candidate, and two C The combination of PMVs constructs a four-parameter affine merge candidate. Motion Scaling In order to avoid processing, if the reference indexes of the control points are different, The associated combination of V is discarded.

[0056] Inherited and constructed affine merge candidates are checked. If there is still room in the list after this, zero MV is inserted at the end of the list.

[0057] Sub-block-based Temporal Motion Vector Prediction (SbTMVP)

[0058] VTM supports the Sub-block-based Temporal Motion Vector Prediction (SbTMVP) method. HEVC Temporal Motion Vector Prediction (T Similar to MVP, SbTMVP uses motion vector prediction and mapping for CUs of the current picture. TMV uses motion fields from co-located pictures to improve the page mode. The same co-located picture used by P is also used for SbTVMP. VP differs from TMVP in two major aspects. 1. TMVP predicts motion at the CU level, whereas SbTMVP predicts motion at the sub-CU level. Predict the movement. 2. TMVP takes the temporal motion vector from the same-position block of the same-position picture. (The same position block is the lower right or center block for the current CU), but Sb TMVP applies motion shifting before extracting temporal motion information from co-located pictures. In this case, the motion shift is performed by shifting the motion vector from one of the spatial neighboring blocks of the current CU. is obtained from

[0059] The SbTVMP process is shown in Figs. 11, 12A, and 12B (described below). ) SbTMVP predicts the motion vectors of sub-CUs in the current CU in two steps. In the first step, the spatial adjacencies in Figure 11 are evaluated in the order A1, B1, B0, and A0. The second picture with a motion vector is estimated using the co-located picture as its reference picture. As soon as a spatial neighbor is identified, this motion vector is applied to the motion shift If no such motion is identified from the spatial neighborhood, the motion sequence is selected. The focus is set to (0,0).

[0060] FIG. 11 shows the block diagram used by the sub-block-based temporal motion vector prediction (SbTMVP). SbTMVP also uses alternative temporal motion vector prediction (Alternative Temporal Motion Vector Prediction:ATMVP) is.

[0061] In the second step, the sub-CU is extracted from the co-located picture as shown in FIGS. 12A and 12B. In step 1, the level motion information (motion vectors and reference indexes) is obtained. The specified motion shift is applied (i.e., added to the coordinates of the current block). The examples of 12A and 12B assume that the motion shift is set to block A1 motion. Then, for each sub-CU, the motion information of its corresponding block in the co-located picture is The information (minimum motion grid covering the center sample) is used to derive the motion information for the sub-CU. After the motion information of the co-located sub-CU is identified, it is used to Convert to the motion vector and reference index of the current sub-CU in a manner similar to VP processing. In this case, the temporal motion scaling is performed by changing the reference picture of the temporal motion vector to the current are applied to align them to those of the CU.

[0062] FIG. 12A shows the motion shift from the spatial neighbors to the corresponding co-located sub-CUs. The same effect when deriving sub-CU motion fields by scaling the motion information. 1 shows VVC's SbTMVP processing for one position picture.

[0063] FIG. 12B shows the motion shift from the spatial neighbors to the corresponding co-located sub-CUs. The current situation when deriving sub-CU motion fields by scaling the motion information 13 shows VVC's SbTMVP processing for the current picture.

[0064] In VTM3, a combined SbTVMP candidate and an affine merge candidate are included. Sub-block based merge lists are now used to signal the sub-block based merge mode. The subblock merging mode is used in the following contexts: SbTVMP mode The code is determined by the Sequence Parameter Set (SPS) flag. If SbTMVP mode is enabled, SbTMVP prediction is The child is added as the first entry in the list of subblock-based merge candidates, followed by Then, the affine merge candidates are followed. The size of the subblock-based merge list is simulcast in SPS. The maximum allowed size of a subblock-based merge list is 5 in VTM3. It is.

[0065] The sub-CU size used in SbTMVP is fixed at 8 × 8, which is the affine merging The SbTMVP mode does the same thing as the TMVP mode, and the SbTMVP mode does not support both width and height >= 8. Only applicable to CU.

[0066] The encoding logic for additional SbTMVP merge candidates is the same as for other merge candidates. That is, for each CU in a P or B slice, we use the SbTMVP candidates. An additional RD check is performed to determine whether to use

[0067] Combined Inter and Intra Prediction :CIIP)

[0068] In VTM3, CUs are coded in merge mode and have at least 64 luminance subpixels. If the CU contains a sample (i.e., the CU width multiplied by the CU height is 64 or more), an additional Flag indicates whether Combined Inter / Intra Prediction (CIIP) mode applies to the current CU. The ipv4_address_request is signaled to indicate whether or not a ping is sent.

[0069] To form the CIIP prediction, first, the intra prediction mode is divided into two additional syntaxes: Up to four possible intra prediction modes: DC, PL In some cases, the horizontal, vertical and horizontal predictions are used. Then, the inter-prediction and intra-prediction are used. The measured signals are derived using conventional intra- and inter-decoding processes. Afterwards, a weighted average of the inter and intra prediction signals is performed to obtain the CIIP prediction. It will be carried out.

[0070] Intra prediction mode derivation 4, including DC, PLANAR, HORIZONTAL and VERTICAL modes Up to four intra prediction modes are used to predict the luma component in CIIP mode. If the CU shape is very wide (i.e., the width is more than twice the height), H ORIZONTAL mode is not possible. If the CU shape is very narrow (i.e., the height is In these cases, VERTICAL mode is not possible. Only three intra prediction modes are possible.

[0071] The CIIP mode has three Most Probable Modes for intra prediction: The CIIP MPM candidate list is formed as follows: - The left and top neighboring blocks are set as A and B, respectively. - Block A and Block B are denoted as intraModeA and intraModeB. The intra prediction mode of block B is derived as follows. ○Let X be either A or B ○ intraModeX is used when 1) block X is not available, or 2) block X 3) If block B is not predicted using CIIP mode or intra mode, If outside the current CTU, it is set to DC. Otherwise, intraModeX is equal to 1) the intra prediction mode of block X In the case of DC or PLANAR, 2) Intrablock X VERTICAL if prediction mode is a "vertical-like" angle mode (greater than 34) or 3) the intra prediction mode of block X is a “horizontal-like” angle mode (3 4 or less), it will be set to HORIZONTAL. -If intraModeA and intraModeB are the same, ○When intraModeA is PLANAR or DC, the three MPMs are The values ​​are set in the following order: ANAR, DC, VERTICAL. In other cases, the three MPMs are {intraModeA, PLANAR, DC} It is set in the order of -Other cases (when intraModeA and intraModeB are different) ○ The first two MPMs are in the order {intraModeA, intraModeB} It is set by ○ PLANAR, DC and VERT for the first two MPM candidate modes in order The uniqueness of the ICAL is checked, and as soon as a unique mode is discovered, the third MPM is added.

[0072] If the CU shape is very wide or very narrow as described above, the MPM flag is signaled Otherwise, the MPM flag is assumed to be 1 for the CIIP intra prediction model. Signaling is used to indicate whether the mode is one of the CIIP MPM candidate modes. It will be done.

[0073] If the MPM flag is set to 1, which of the MPM candidate modes will be used for CIIP intra prediction? The MPM index is further signaled to indicate which MPM is used. If the PM flag is 0, the intra prediction mode is a mode that is "not present" in the MPM candidate list. For example, if PLANAR mode is not in the MPM candidate list, then PLAN AR is a non-existent mode, and the intra prediction mode is set to PLANAR. Since three possible intra prediction modes are allowed in CIIP, the MPM candidate list is Contains only four intra prediction modes, one of which is a non-existent mode. It is.

[0074] For the chroma components, the DM mode is always applied without any additional signaling, i.e. That is, saturation uses the same prediction mode as luminance.

[0075] The intra prediction mode of the CIIP-coded CU is preserved and the intra prediction mode of the future neighboring CUs is Used in code encoding.

[0076] Combined inter and intra prediction signals Inter prediction signal P in CIIP mode inter applies to normal merge mode The intra-prediction signal P intra teeth , are derived using the CIIP intra prediction mode following the normal intra prediction process. Then, the intra and inter prediction signals are combined using a weighted average, where The weighting value depends on the intra prediction mode, and the samples are located in the coding block as follows: It is attached. -If the intra prediction mode is DC or planar mode, or the block width or If the height is less than 4, equal weights are applied to the intra- and inter-prediction signals. It is used. - Otherwise, the weights are calculated based on the intra prediction mode (in this case, horizontal or vertical mode) and the sample position within the block. We take the horizontal prediction mode (the weights for the vertical mode are derived similarly, but in the orthogonal direction). ) The width of the block is represented as W, and the height of the block is represented as H. The coding block is first divided into four The intra prediction reference subpixel is divided into equal parts, each with dimensions (W / 4) × H. Start with the part closest to the intra prediction reference sample and end with the part farthest from the intra prediction reference sample. The weights wt for each of the four regions are set to 6, 5, 3, and 2, respectively. The final CIIP prediction signal is derived using Equation (3) below.

number

[0077] Triangulation for Inter Prediction In VTM3, a new triangulation mode is introduced for inter prediction. Split mode only applies to CUs of 8x8 or larger, and cannot be coded in skip or merge modes. If these conditions are met and the merge flag is on, the CU The level flag is signaled to indicate whether the triangulation mode applies. do.

[0078] When this mode is used, either diagonal or off-diagonal partitioning is used to split the CU is divided equally into two triangular divisions (see Figures 13A and 13B, described below). Each triangulation in a CU is inter-predicted using its own motion, resulting in a single Only predictions of 1 are allowed for each partition, i.e., each partition can have only one motion vector and A single prediction motion constraint is the same as in conventional bi-prediction. is applied to ensure that only two motion compensation predictions are needed for each CU. do.

[0079] A CU-level flag indicates that the current CU is coded using triangulation mode. If it is, an index in the range [0, 39] is further signaled. The partition index is used to determine the orientation of the triangulation (diagonal or off-diagonal), as well as the The movement for each can be obtained through a lookup table. After each prediction, the sample values ​​along the diagonal or off-diagonal edges are mixed with adaptive weights. This is the predicted signal for all CUs, which is then transformed and quantized. The approximation process is applied to all CUs as in other prediction modes. The motion fields of CUs predicted using are stored in 4x4 units.

[0080] FIG. 13A illustrates triangulation-based inter prediction according to this disclosure.

[0081] FIG. 13B illustrates triangulation-based inter prediction according to this disclosure.

[0082] Context-Adaptive Binary Arithmetic Coding :CABAC)

[0083] Context-Adaptive Binary Arithmetic Coding (CABAC) is a standard for H.264 / MPEG-4 A Entropy used in the VC and High-Efficiency Video Coding (HEVC) standards and VVC CABAC is a video encoding format that meets the needs of video encoding standards. It is based on arithmetic coding with some innovations and modifications to accommodate it. Keeping complexity low, probabilistic modeling of the more frequently used bits of any symbol This encodes binary symbols that allow Usually, each coding mode is well correlated locally, so the probability model is based on the local context. The,parameters are adaptively selected based on the,parameters, allowing for better modeling of the probabilities. Multiplication-free range division with quantized probability ranges and probability states Use quantification-free range division.

[0084] CABAC has multiple probability modes for different contexts. Convert the binary symbols into binary values. Then, for each bin (or bit), The filter selects which probability model to use and optimizes the probability estimate using information from neighboring elements. Finally, arithmetic coding is applied to compress the data.

[0085] Context modeling provides estimates of the conditional probabilities of the coding symbols Using a suitable context model, we can encode the current symbol By switching between different probability models according to the already coded symbols in the neighborhood of Thus, the given inter-symbol redundancy can be exploited.

[0086] The encoding of a data symbol involves the following steps. Binarization: CABAC means that only binary decisions (1 or 0) are encoded It uses binary arithmetic coding to encode non-binary symbols (e.g., transform coefficients or motion vectors). ) are "binarized" or converted to a binary code before being arithmetically coded. This process is a process similar to converting data symbols into variable length codes, but for binary codes is further encoded (by an arithmetic coder) before transmission. The steps are repeated for each bin (or "bit") of the binarized symbol. Context model selection: A "context model" is one of the binary symbols. or a probability model for multiple bins. This model is based on the most recently coded data sequence. Depending on the statistics of the symbol, a selection of available models may be selected. The delta stores the probability of each bin being a "1" or a "0." Arithmetic Encoding: The arithmetic coder encodes each bin according to the selected probability model. There are only two subranges per bin (corresponding to "0" or "1"). Please note that. Probability update: The selected context model is updated based on the actual encoded values. (For example, if the bin value is "1", the count of "1"s is incremented).

[0087] FIG. 14 illustrates a computing environment 14 connected to a user interface 1460. 10. The computing environment 1410 is part of a data processing server. The computing environment 1410 may include a processor 1420, a memory 1440, and , and an I / O interface 1450.

[0088] The processor 1420 typically handles display, data acquisition, data communication, and imaging. Controlling the overall operation of the computing environment 1410, such as operations related to image processing. The processor 1420 may include instructions to perform all or some of the steps of the above-described methods. The processor 14 may include one or more processors for executing instructions. 20 includes one or more modules that facilitate interaction with the processor 1420 and other components. The processor may include a central processing unit. It can be a CPU, a microprocessor, a single chip machine, a GPU, etc. stomach.

[0089] The memory 1440 may include various components to support the operation of the computing environment 1410. The device is configured to store various types of data, such as MRI, Any application, such as a data set, image data, etc., that is operated on in the computing environment 1410 The memory 1440 includes a static random access memory (RAM) 1444 for storing instructions for ... Static Random Access Memory (SRAM), electrically erasable programmable Electrically Erasable Programmable Read-Only Memory EEPROM, erasable programmable read-only memory Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory able Read-Only Memory (PROM), Read-Only Memory (RO M), magnetic memory, flash memory, magnetic or optical disks, etc. Implemented using nonvolatile or non-volatile memory devices, or a combination of both good.

[0090] The I / O interface 1450 is connected to the processor 1420 and to a keyboard, click hole, and other Provides an interface between the device and interface modules such as wheels and buttons. Buttons include, but are not limited to, a home button, a start scan button, a stop scan button, The I / O interface 1450 is an interface for connecting an encoder and a decoder. It may be connected to a power adapter.

[0091] In one embodiment, the memory 1440 includes a A number of processes executable by the processor 1420 for implementing the methods described above. A non-transitory computer readable storage medium including the program is also provided. Computer-readable storage media include ROM, RAM, CD-ROM, magnetic tape, floppy disk It may be a disk, an optical data storage device, etc.

[0092] A non-transitory computer-readable storage medium is a medium that is stored on itself and can be used by one or more processors. a plurality of programs executed by a computing device having a processor; The programs, when executed by one or more processors, The present invention has a video signal processing device implement the above-described method for motion estimation.

[0093] In one embodiment, the computing environment 1410 is configured to perform the above-described methods. , one or more Application-Specific Integrated Circuits uit:ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Pro Programmable Logic Device (PLD), Field Programmable Field-Programmable Gate Array (FPGA), graphics Graphical Processing Unit (GPU), Controller, Microcontroller The device may be implemented with a controller, microprocessor, or other electronic components.

[0094] Methods according to the present disclosure As mentioned above, in VTM-3.0, merge modes are further divided into five categories: That is, normal merging, merge mode MVD (MMVD), sub-block merging (affine (including merge and sub-block based temporal motion vector prediction), combined inter- and The current VVC is classified into intra-prediction (CIIP), intra-prediction (CIIP), and triangulation merging. The syntax of merge mode signaling in is shown in Tables 3-1 and 3-2 below. [Table 3-1] [Table 3-2] For summary purposes, signaling to indicate the corresponding merge mode in the current VVC The syntax (and associated flags) used is shown below. [Table 4]

[0095] It is observed that more than 50% of the merge modes are normal merge modes. However, In VTM-3.0, the codeword for the normal merge mode is This is the longest of the three, and is not an efficient design from a syntax analysis point of view. In the current VVC, the skip mode has a similar syntax design to the merge mode. However, there is no CIIP mode for skipping. However, the same observations can be obtained in skip mode. It was made.

[0096] Syntax for a normal merge As mentioned above, the current VVC normal merge mode scheme is (normal merge, M Several merging functions including MVD, subblock merging, CIIP and triangle merging In one embodiment of the present disclosure, the normal merge mode is the most frequently used. A clear flag for normal merge mode is provided to signal whether the As shown in the table below, one normal flag (or called normal merge flag) is The 10G bitstream is explicitly signaled, and all associated flag signaling is done by , and amended accordingly. The normal merge flags are encoded using CABAC In some schemes, only one context is used, which is the normal merge flag. In yet another scheme, multiple context models are used to encode the is used to encode the normal merge flags, and the choice of context model is determined by the Based on the usual merge flags of the adjacent blocks, or on encoded information such as the size of the current CU. Made. [Table 5]

[0097] Currently, VVC has different constraints for enabling each merge-related mode. The signaling of merge-related mode flags is also different, as summarized below. . [Table 6]

[0098] Therefore, the normal merge flag signaling also applies to each flag signaling. For example, the block size is 4×4, 8× For 4 or 4x8, only normal merge mode and MMVD are valid. In the condition (block size is 4x4, 8x4 or 4x8), only the normal merge flag is If signaled, and the normal merge flag is equal to 1, normal merge mode is used. If the other regular merge flags are equal to 0, then the MMVD is used. Examples of syntax based on the C behavioral draft are shown in Tables 7-1, 7-2, and 7-3 below. . [Table 7-1] [Table 7-2] [Table 7-3]

[0099] In the example, we note that the normal merge flags are explicitly signaled in the bitstream. However, the normal merge flag may be signaled in any arrangement, It is not necessary to use the first arrangement as described above. In yet another scheme, the usual merge flags are The flag is signaled, but after the MMVD and sub-block merge flags. It will be processed.

[0100] Integrating related merge modes into the regular merge mode In one embodiment of the present disclosure, MMVD, CIIP, and triangle merging are performed using the conventional In this scheme, the MMVD candidate, the CIIP candidate and the triangle All merge candidates are considered regular merge candidates, and regular merge indexes are This is used to indicate which candidates are used. The size of the merge candidate list is usually In one example, a regular merge index equal to N, where N is (which may be any positive integer and is smaller than the maximum size of a typical merge candidate list) Another symbol is used to indicate which MMVD mode is selected and which MMVD candidate is used. This means that the tax is signaled / received. The same scheme is used for CIIP and It may also be applied to triangle merging mode.

[0101] In yet another example, CIIP and triangle merging are combined into a regular merging mode. In this scheme, all CIIP candidates and triangle merging candidates are regular merging candidates. and the normal merge index is used to indicate which candidate is used. The size of the regular merge candidate list needs to be increased accordingly.

[0102] Adjusting constraints As mentioned above, the constraints that enable the different merge-related modes are different. In an embodiment, a control for enabling different merge modes and signaling the associated flags is provided. The constraints are further adjusted. In one example, the constraints are modified as shown in the table below. [Table 8]

[0103] In yet another example of the present disclosure, the constraints are modified as shown in the table below.

[0104] [Table 9]

[0105] In yet another example of the present disclosure, the constraints are modified as shown in the table below. So, the CIIP flag is, even if block width=128 or block height=128, , signaled, if block width = 128 or block height = 128, intra Note that it is always constrained to 0 since prediction does not support these conditions.

[0106] [Table 10]

[0107] In yet another example of the present disclosure, the constraints are modified as shown in the table below. So, the CIIP flag is, even if block width=128 or block height=128, , signaled, if block width = 128 or block height = 128, intra Note that it is always constrained to 0 since prediction does not support these conditions.

[0108] [Table 11]

[0109] Switching the order of CIIP and triangle merge flags The triangle merge mode is observed to be used more frequently, so the CIIP flag and switching the signaling order of triangle merge flags.

Claims

1. 1. A method of encoding video, comprising the steps of: signaling, by an encoder, a regular merge flag for coding units (CUs) that are coded as merge mode and merge-related mode; if the normal merge flag is signaled indicating 1, indicating that normal merge mode or Merge mode with Motion Vector Difference (MMVD) is used by the CU, constructing a motion vector merge list for the CU by the encoder and using a normal merge index to indicate which candidate is used; signaling, by the encoder, a mode flag indicating an associated merge-related mode to be used when the normal merge flag indicating 0 is signaled, and indicating that the normal merge mode is not used by the CU, and further indicating an associated merge-related mode to be used when a mode flag constraint is satisfied; The method further includes, if the normal merge flag indicating one is signaled, determining whether to signal an MMVD merge flag based on a value of an MMVD flag. method.

2. The method of claim 1 , further comprising: signaling, by the encoder, the MMVD merge flag if the MMVD flag is equal to one.

3. encoding a sub-block flag; The constraint condition of the sub-block flag is obtaining, by the encoder, an encoded block having a width and a height; indicating by the encoder whether a maximum number of subblock-based merging MVP candidates (MaxNumSubblockMergeCand) is greater than 0; indicating by the encoder whether the coding block width is 8 or more; indicating by the encoder whether the encoded block height is 8 or greater; The method of claim 1.

4. The mode flag is a Combined Inter and Intra Prediction (CIIP) flag, The constraints of the CIIP flag are: obtaining, by the encoder, an encoded block having a width and a height; Indicating whether sps_mh_intra_enabled_flag is set by the encoder; indicating whether cu_skip_flag is equal to 0 by the encoder; indicating by the encoder whether the coding block width times the coding block height is greater than or equal to 64; indicating by the encoder whether a coding block width is less than 128; indicating, by the encoder, whether an encoded block height is less than 128; The method of claim 1.

5. The method of claim 1 , further comprising signaling a regular merge flag by the encoder prior to signaling the mode flag by the encoder when a constraint of the mode flag is satisfied.

6. 6. The method of claim 5, further comprising signaling a normal merge flag by the encoder prior to signaling the differential motion vector merge mode (MMVD) flag by the encoder when a constraint of the MMVD flag is satisfied.

7. 6. The method of claim 5, further comprising signaling a regular merge flag by the encoder prior to signaling the CIIP flag by the encoder when a combined inter- and intra-prediction (CIIP) flag constraint is satisfied.

8. The method of claim 1 , further comprising signaling a regular merge flag by the encoder after signaling the mode flag by the encoder when a mode flag constraint is satisfied.

9. 10. The method of claim 8, further comprising signaling a normal merging flag by the encoder after signaling the sub-block merging mode flag by the encoder when a constraint on a sub-block merging mode flag is satisfied.

10. Context-Adaptive Binary Arithmetic Coding (CABAC) Using Multiple Context Models and signaling, by the encoder, the normal merge flag using the selection of the context model is based on the coded information. The method of claim 1.

11. one or more processors; a memory coupled to the one or more processors; a plurality of programs stored in the memory, The plurality of programs, when executed by the one or more processors, cause the electronic device to perform a method according to any one of claims 1 to 10. electronic equipment.

12. A computer program in a bitstream causing a computer to carry out the steps of the method according to any one of claims 1 to 10.

Images